Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 658266 Details for
Bug 883921
Floating disks created although template creation failed
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm.log
vdsm.log (text/x-log), 15.32 MB, created by
Vered Volansky
on 2012-12-05 15:24:14 UTC
(
hide
)
Description:
vdsm.log
Filename:
MIME Type:
Creator:
Vered Volansky
Created:
2012-12-05 15:24:14 UTC
Size:
15.32 MB
patch
obsolete
>Thread-25586::DEBUG::2012-12-02 14:01:01,279::task::568::TaskManager.Task::(_updateState) Task=`69b70f4c-f42b-4083-b011-63f90951f29a`::moving from state init -> state preparing >Thread-25586::INFO::2012-12-02 14:01:01,280::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25586::INFO::2012-12-02 14:01:01,280::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00623798370361', 'lastCheck': 1354449653.5339711, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00221419334412', 'lastCheck': 1354449658.2730811, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00117182731628', 'lastCheck': 1354449652.0384669, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00725817680359', 'lastCheck': 1354449658.2823119, 'code': 0, 'valid': True}} >Thread-25586::DEBUG::2012-12-02 14:01:01,280::task::1151::TaskManager.Task::(prepare) Task=`69b70f4c-f42b-4083-b011-63f90951f29a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00623798370361', 'lastCheck': 1354449653.5339711, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00221419334412', 'lastCheck': 1354449658.2730811, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00117182731628', 'lastCheck': 1354449652.0384669, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00725817680359', 'lastCheck': 1354449658.2823119, 'code': 0, 'valid': True}} >Thread-25586::DEBUG::2012-12-02 14:01:01,280::task::568::TaskManager.Task::(_updateState) Task=`69b70f4c-f42b-4083-b011-63f90951f29a`::moving from state preparing -> state finished >Thread-25586::DEBUG::2012-12-02 14:01:01,280::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25586::DEBUG::2012-12-02 14:01:01,280::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25586::DEBUG::2012-12-02 14:01:01,280::task::957::TaskManager.Task::(_decref) Task=`69b70f4c-f42b-4083-b011-63f90951f29a`::ref 0 aborting False >Thread-25588::DEBUG::2012-12-02 14:01:02,173::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25588::DEBUG::2012-12-02 14:01:02,173::task::568::TaskManager.Task::(_updateState) Task=`64f19dff-6fb7-444d-9598-cc7aed338132`::moving from state init -> state preparing >Thread-25588::INFO::2012-12-02 14:01:02,173::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25588::INFO::2012-12-02 14:01:02,173::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25588::DEBUG::2012-12-02 14:01:02,174::task::1151::TaskManager.Task::(prepare) Task=`64f19dff-6fb7-444d-9598-cc7aed338132`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25588::DEBUG::2012-12-02 14:01:02,174::task::568::TaskManager.Task::(_updateState) Task=`64f19dff-6fb7-444d-9598-cc7aed338132`::moving from state preparing -> state finished >Thread-25588::DEBUG::2012-12-02 14:01:02,174::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25588::DEBUG::2012-12-02 14:01:02,174::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25588::DEBUG::2012-12-02 14:01:02,174::task::957::TaskManager.Task::(_decref) Task=`64f19dff-6fb7-444d-9598-cc7aed338132`::ref 0 aborting False >Thread-25589::DEBUG::2012-12-02 14:01:02,177::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25589::DEBUG::2012-12-02 14:01:02,177::task::568::TaskManager.Task::(_updateState) Task=`66765d9a-c592-41d5-bde7-68ed53fe3133`::moving from state init -> state preparing >Thread-25589::INFO::2012-12-02 14:01:02,177::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25589::DEBUG::2012-12-02 14:01:02,178::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e60fdcba-8319-4971-a0ab-ddfc8476fcc5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25589::DEBUG::2012-12-02 14:01:02,178::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25589::DEBUG::2012-12-02 14:01:02,178::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25589::DEBUG::2012-12-02 14:01:02,178::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e60fdcba-8319-4971-a0ab-ddfc8476fcc5`::Granted request >Thread-25589::DEBUG::2012-12-02 14:01:02,178::task::794::TaskManager.Task::(resourceAcquired) Task=`66765d9a-c592-41d5-bde7-68ed53fe3133`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25589::DEBUG::2012-12-02 14:01:02,178::task::957::TaskManager.Task::(_decref) Task=`66765d9a-c592-41d5-bde7-68ed53fe3133`::ref 1 aborting False >Thread-25589::INFO::2012-12-02 14:01:02,183::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25589::DEBUG::2012-12-02 14:01:02,183::task::1151::TaskManager.Task::(prepare) Task=`66765d9a-c592-41d5-bde7-68ed53fe3133`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25589::DEBUG::2012-12-02 14:01:02,183::task::568::TaskManager.Task::(_updateState) Task=`66765d9a-c592-41d5-bde7-68ed53fe3133`::moving from state preparing -> state finished >Thread-25589::DEBUG::2012-12-02 14:01:02,183::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25589::DEBUG::2012-12-02 14:01:02,183::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25589::DEBUG::2012-12-02 14:01:02,183::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25589::DEBUG::2012-12-02 14:01:02,183::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25589::DEBUG::2012-12-02 14:01:02,183::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25589::DEBUG::2012-12-02 14:01:02,184::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25589::DEBUG::2012-12-02 14:01:02,184::task::957::TaskManager.Task::(_decref) Task=`66765d9a-c592-41d5-bde7-68ed53fe3133`::ref 0 aborting False >Thread-25594::DEBUG::2012-12-02 14:01:11,375::task::568::TaskManager.Task::(_updateState) Task=`b5a2ef3d-2204-4ef4-989e-3d43321d6a43`::moving from state init -> state preparing >Thread-25594::INFO::2012-12-02 14:01:11,376::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25594::INFO::2012-12-02 14:01:11,376::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0062849521637', 'lastCheck': 1354449663.550606, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00459694862366', 'lastCheck': 1354449668.292074, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00362992286682', 'lastCheck': 1354449662.0508411, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00688314437866', 'lastCheck': 1354449668.2993391, 'code': 0, 'valid': True}} >Thread-25594::DEBUG::2012-12-02 14:01:11,376::task::1151::TaskManager.Task::(prepare) Task=`b5a2ef3d-2204-4ef4-989e-3d43321d6a43`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0062849521637', 'lastCheck': 1354449663.550606, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00459694862366', 'lastCheck': 1354449668.292074, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00362992286682', 'lastCheck': 1354449662.0508411, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00688314437866', 'lastCheck': 1354449668.2993391, 'code': 0, 'valid': True}} >Thread-25594::DEBUG::2012-12-02 14:01:11,377::task::568::TaskManager.Task::(_updateState) Task=`b5a2ef3d-2204-4ef4-989e-3d43321d6a43`::moving from state preparing -> state finished >Thread-25594::DEBUG::2012-12-02 14:01:11,377::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25594::DEBUG::2012-12-02 14:01:11,377::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25594::DEBUG::2012-12-02 14:01:11,377::task::957::TaskManager.Task::(_decref) Task=`b5a2ef3d-2204-4ef4-989e-3d43321d6a43`::ref 0 aborting False >Thread-25596::DEBUG::2012-12-02 14:01:12,359::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25596::DEBUG::2012-12-02 14:01:12,360::task::568::TaskManager.Task::(_updateState) Task=`9db9e778-203e-4a40-b80f-0c2d39a74741`::moving from state init -> state preparing >Thread-25596::INFO::2012-12-02 14:01:12,360::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25596::INFO::2012-12-02 14:01:12,360::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25596::DEBUG::2012-12-02 14:01:12,360::task::1151::TaskManager.Task::(prepare) Task=`9db9e778-203e-4a40-b80f-0c2d39a74741`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25596::DEBUG::2012-12-02 14:01:12,360::task::568::TaskManager.Task::(_updateState) Task=`9db9e778-203e-4a40-b80f-0c2d39a74741`::moving from state preparing -> state finished >Thread-25596::DEBUG::2012-12-02 14:01:12,360::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25596::DEBUG::2012-12-02 14:01:12,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25596::DEBUG::2012-12-02 14:01:12,361::task::957::TaskManager.Task::(_decref) Task=`9db9e778-203e-4a40-b80f-0c2d39a74741`::ref 0 aborting False >Thread-25597::DEBUG::2012-12-02 14:01:12,364::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25597::DEBUG::2012-12-02 14:01:12,365::task::568::TaskManager.Task::(_updateState) Task=`3b59e1b2-7960-4a6b-9883-d21a9d9656b8`::moving from state init -> state preparing >Thread-25597::INFO::2012-12-02 14:01:12,365::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25597::DEBUG::2012-12-02 14:01:12,365::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`87f57a49-aeef-4ec4-96cd-25e2ee19a6ec`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25597::DEBUG::2012-12-02 14:01:12,365::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25597::DEBUG::2012-12-02 14:01:12,365::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25597::DEBUG::2012-12-02 14:01:12,365::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`87f57a49-aeef-4ec4-96cd-25e2ee19a6ec`::Granted request >Thread-25597::DEBUG::2012-12-02 14:01:12,366::task::794::TaskManager.Task::(resourceAcquired) Task=`3b59e1b2-7960-4a6b-9883-d21a9d9656b8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25597::DEBUG::2012-12-02 14:01:12,366::task::957::TaskManager.Task::(_decref) Task=`3b59e1b2-7960-4a6b-9883-d21a9d9656b8`::ref 1 aborting False >Thread-25597::INFO::2012-12-02 14:01:12,372::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25597::DEBUG::2012-12-02 14:01:12,372::task::1151::TaskManager.Task::(prepare) Task=`3b59e1b2-7960-4a6b-9883-d21a9d9656b8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25597::DEBUG::2012-12-02 14:01:12,373::task::568::TaskManager.Task::(_updateState) Task=`3b59e1b2-7960-4a6b-9883-d21a9d9656b8`::moving from state preparing -> state finished >Thread-25597::DEBUG::2012-12-02 14:01:12,373::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25597::DEBUG::2012-12-02 14:01:12,373::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25597::DEBUG::2012-12-02 14:01:12,373::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25597::DEBUG::2012-12-02 14:01:12,373::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25597::DEBUG::2012-12-02 14:01:12,373::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25597::DEBUG::2012-12-02 14:01:12,373::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25597::DEBUG::2012-12-02 14:01:12,374::task::957::TaskManager.Task::(_decref) Task=`3b59e1b2-7960-4a6b-9883-d21a9d9656b8`::ref 0 aborting False >Thread-25602::DEBUG::2012-12-02 14:01:21,554::task::568::TaskManager.Task::(_updateState) Task=`4d69bf9f-71d4-4939-b01f-4ade8988e30d`::moving from state init -> state preparing >Thread-25602::INFO::2012-12-02 14:01:21,554::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25602::INFO::2012-12-02 14:01:21,554::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00762104988098', 'lastCheck': 1354449673.567853, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00303387641907', 'lastCheck': 1354449678.3118939, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00321507453918', 'lastCheck': 1354449672.064563, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0081090927124', 'lastCheck': 1354449678.3190269, 'code': 0, 'valid': True}} >Thread-25602::DEBUG::2012-12-02 14:01:21,554::task::1151::TaskManager.Task::(prepare) Task=`4d69bf9f-71d4-4939-b01f-4ade8988e30d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00762104988098', 'lastCheck': 1354449673.567853, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00303387641907', 'lastCheck': 1354449678.3118939, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00321507453918', 'lastCheck': 1354449672.064563, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0081090927124', 'lastCheck': 1354449678.3190269, 'code': 0, 'valid': True}} >Thread-25602::DEBUG::2012-12-02 14:01:21,554::task::568::TaskManager.Task::(_updateState) Task=`4d69bf9f-71d4-4939-b01f-4ade8988e30d`::moving from state preparing -> state finished >Thread-25602::DEBUG::2012-12-02 14:01:21,554::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25602::DEBUG::2012-12-02 14:01:21,555::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25602::DEBUG::2012-12-02 14:01:21,555::task::957::TaskManager.Task::(_decref) Task=`4d69bf9f-71d4-4939-b01f-4ade8988e30d`::ref 0 aborting False >Thread-25604::DEBUG::2012-12-02 14:01:22,550::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25604::DEBUG::2012-12-02 14:01:22,550::task::568::TaskManager.Task::(_updateState) Task=`b108126c-c0e8-45c1-b657-aff9af5e28b1`::moving from state init -> state preparing >Thread-25604::INFO::2012-12-02 14:01:22,550::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25604::INFO::2012-12-02 14:01:22,551::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25604::DEBUG::2012-12-02 14:01:22,551::task::1151::TaskManager.Task::(prepare) Task=`b108126c-c0e8-45c1-b657-aff9af5e28b1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25604::DEBUG::2012-12-02 14:01:22,551::task::568::TaskManager.Task::(_updateState) Task=`b108126c-c0e8-45c1-b657-aff9af5e28b1`::moving from state preparing -> state finished >Thread-25604::DEBUG::2012-12-02 14:01:22,551::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25604::DEBUG::2012-12-02 14:01:22,551::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25604::DEBUG::2012-12-02 14:01:22,551::task::957::TaskManager.Task::(_decref) Task=`b108126c-c0e8-45c1-b657-aff9af5e28b1`::ref 0 aborting False >Thread-25605::DEBUG::2012-12-02 14:01:22,555::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25605::DEBUG::2012-12-02 14:01:22,555::task::568::TaskManager.Task::(_updateState) Task=`23a3b6fc-7d23-468b-9728-323ef9189308`::moving from state init -> state preparing >Thread-25605::INFO::2012-12-02 14:01:22,555::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25605::DEBUG::2012-12-02 14:01:22,556::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`58e637e4-20a2-4304-a203-13c82f7fe48f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25605::DEBUG::2012-12-02 14:01:22,556::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25605::DEBUG::2012-12-02 14:01:22,556::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25605::DEBUG::2012-12-02 14:01:22,556::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`58e637e4-20a2-4304-a203-13c82f7fe48f`::Granted request >Thread-25605::DEBUG::2012-12-02 14:01:22,556::task::794::TaskManager.Task::(resourceAcquired) Task=`23a3b6fc-7d23-468b-9728-323ef9189308`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25605::DEBUG::2012-12-02 14:01:22,556::task::957::TaskManager.Task::(_decref) Task=`23a3b6fc-7d23-468b-9728-323ef9189308`::ref 1 aborting False >Thread-25605::INFO::2012-12-02 14:01:22,563::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25605::DEBUG::2012-12-02 14:01:22,563::task::1151::TaskManager.Task::(prepare) Task=`23a3b6fc-7d23-468b-9728-323ef9189308`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25605::DEBUG::2012-12-02 14:01:22,563::task::568::TaskManager.Task::(_updateState) Task=`23a3b6fc-7d23-468b-9728-323ef9189308`::moving from state preparing -> state finished >Thread-25605::DEBUG::2012-12-02 14:01:22,563::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25605::DEBUG::2012-12-02 14:01:22,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25605::DEBUG::2012-12-02 14:01:22,564::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25605::DEBUG::2012-12-02 14:01:22,564::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25605::DEBUG::2012-12-02 14:01:22,564::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25605::DEBUG::2012-12-02 14:01:22,564::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25605::DEBUG::2012-12-02 14:01:22,564::task::957::TaskManager.Task::(_decref) Task=`23a3b6fc-7d23-468b-9728-323ef9189308`::ref 0 aborting False >Thread-25610::DEBUG::2012-12-02 14:01:31,734::task::568::TaskManager.Task::(_updateState) Task=`e5687754-37d7-4ff5-b1c7-86d7e7ecf83d`::moving from state init -> state preparing >Thread-25610::INFO::2012-12-02 14:01:31,734::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25610::INFO::2012-12-02 14:01:31,735::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00665307044983', 'lastCheck': 1354449683.5836589, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00429105758667', 'lastCheck': 1354449688.326905, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00230979919434', 'lastCheck': 1354449682.0770299, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00710892677307', 'lastCheck': 1354449688.335813, 'code': 0, 'valid': True}} >Thread-25610::DEBUG::2012-12-02 14:01:31,735::task::1151::TaskManager.Task::(prepare) Task=`e5687754-37d7-4ff5-b1c7-86d7e7ecf83d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00665307044983', 'lastCheck': 1354449683.5836589, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00429105758667', 'lastCheck': 1354449688.326905, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00230979919434', 'lastCheck': 1354449682.0770299, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00710892677307', 'lastCheck': 1354449688.335813, 'code': 0, 'valid': True}} >Thread-25610::DEBUG::2012-12-02 14:01:31,735::task::568::TaskManager.Task::(_updateState) Task=`e5687754-37d7-4ff5-b1c7-86d7e7ecf83d`::moving from state preparing -> state finished >Thread-25610::DEBUG::2012-12-02 14:01:31,735::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25610::DEBUG::2012-12-02 14:01:31,735::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25610::DEBUG::2012-12-02 14:01:31,735::task::957::TaskManager.Task::(_decref) Task=`e5687754-37d7-4ff5-b1c7-86d7e7ecf83d`::ref 0 aborting False >Thread-25612::DEBUG::2012-12-02 14:01:32,710::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25612::DEBUG::2012-12-02 14:01:32,710::task::568::TaskManager.Task::(_updateState) Task=`c73e0f3a-1ea8-4750-b9bf-7f3d3cce3c19`::moving from state init -> state preparing >Thread-25612::INFO::2012-12-02 14:01:32,711::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25612::INFO::2012-12-02 14:01:32,711::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25612::DEBUG::2012-12-02 14:01:32,711::task::1151::TaskManager.Task::(prepare) Task=`c73e0f3a-1ea8-4750-b9bf-7f3d3cce3c19`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25612::DEBUG::2012-12-02 14:01:32,711::task::568::TaskManager.Task::(_updateState) Task=`c73e0f3a-1ea8-4750-b9bf-7f3d3cce3c19`::moving from state preparing -> state finished >Thread-25612::DEBUG::2012-12-02 14:01:32,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25612::DEBUG::2012-12-02 14:01:32,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25612::DEBUG::2012-12-02 14:01:32,711::task::957::TaskManager.Task::(_decref) Task=`c73e0f3a-1ea8-4750-b9bf-7f3d3cce3c19`::ref 0 aborting False >Thread-25613::DEBUG::2012-12-02 14:01:32,715::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25613::DEBUG::2012-12-02 14:01:32,715::task::568::TaskManager.Task::(_updateState) Task=`ab8733ae-f305-4470-b303-27ebc7d0aca2`::moving from state init -> state preparing >Thread-25613::INFO::2012-12-02 14:01:32,715::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25613::DEBUG::2012-12-02 14:01:32,715::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9b29f399-6c66-43ea-80cf-86a1125978bd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25613::DEBUG::2012-12-02 14:01:32,716::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25613::DEBUG::2012-12-02 14:01:32,716::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25613::DEBUG::2012-12-02 14:01:32,716::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9b29f399-6c66-43ea-80cf-86a1125978bd`::Granted request >Thread-25613::DEBUG::2012-12-02 14:01:32,716::task::794::TaskManager.Task::(resourceAcquired) Task=`ab8733ae-f305-4470-b303-27ebc7d0aca2`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25613::DEBUG::2012-12-02 14:01:32,716::task::957::TaskManager.Task::(_decref) Task=`ab8733ae-f305-4470-b303-27ebc7d0aca2`::ref 1 aborting False >Thread-25613::INFO::2012-12-02 14:01:32,723::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25613::DEBUG::2012-12-02 14:01:32,723::task::1151::TaskManager.Task::(prepare) Task=`ab8733ae-f305-4470-b303-27ebc7d0aca2`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25613::DEBUG::2012-12-02 14:01:32,723::task::568::TaskManager.Task::(_updateState) Task=`ab8733ae-f305-4470-b303-27ebc7d0aca2`::moving from state preparing -> state finished >Thread-25613::DEBUG::2012-12-02 14:01:32,723::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25613::DEBUG::2012-12-02 14:01:32,723::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25613::DEBUG::2012-12-02 14:01:32,723::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25613::DEBUG::2012-12-02 14:01:32,724::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25613::DEBUG::2012-12-02 14:01:32,724::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25613::DEBUG::2012-12-02 14:01:32,724::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25613::DEBUG::2012-12-02 14:01:32,724::task::957::TaskManager.Task::(_decref) Task=`ab8733ae-f305-4470-b303-27ebc7d0aca2`::ref 0 aborting False >Thread-25618::DEBUG::2012-12-02 14:01:41,922::task::568::TaskManager.Task::(_updateState) Task=`7494a98a-baaa-4f1c-a38d-8084e7bbe8d9`::moving from state init -> state preparing >Thread-25618::INFO::2012-12-02 14:01:41,922::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25618::INFO::2012-12-02 14:01:41,922::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.099925994873', 'lastCheck': 1354449693.6972151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00219488143921', 'lastCheck': 1354449698.344295, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00384902954102', 'lastCheck': 1354449692.090194, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00524878501892', 'lastCheck': 1354449698.3516879, 'code': 0, 'valid': True}} >Thread-25618::DEBUG::2012-12-02 14:01:41,923::task::1151::TaskManager.Task::(prepare) Task=`7494a98a-baaa-4f1c-a38d-8084e7bbe8d9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.099925994873', 'lastCheck': 1354449693.6972151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00219488143921', 'lastCheck': 1354449698.344295, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00384902954102', 'lastCheck': 1354449692.090194, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00524878501892', 'lastCheck': 1354449698.3516879, 'code': 0, 'valid': True}} >Thread-25618::DEBUG::2012-12-02 14:01:41,923::task::568::TaskManager.Task::(_updateState) Task=`7494a98a-baaa-4f1c-a38d-8084e7bbe8d9`::moving from state preparing -> state finished >Thread-25618::DEBUG::2012-12-02 14:01:41,923::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25618::DEBUG::2012-12-02 14:01:41,923::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25618::DEBUG::2012-12-02 14:01:41,923::task::957::TaskManager.Task::(_decref) Task=`7494a98a-baaa-4f1c-a38d-8084e7bbe8d9`::ref 0 aborting False >Thread-25620::DEBUG::2012-12-02 14:01:42,880::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25620::DEBUG::2012-12-02 14:01:42,881::task::568::TaskManager.Task::(_updateState) Task=`162d01d5-ac80-4feb-bd7c-1a18e7ddd914`::moving from state init -> state preparing >Thread-25620::INFO::2012-12-02 14:01:42,881::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25620::INFO::2012-12-02 14:01:42,881::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25620::DEBUG::2012-12-02 14:01:42,881::task::1151::TaskManager.Task::(prepare) Task=`162d01d5-ac80-4feb-bd7c-1a18e7ddd914`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25620::DEBUG::2012-12-02 14:01:42,881::task::568::TaskManager.Task::(_updateState) Task=`162d01d5-ac80-4feb-bd7c-1a18e7ddd914`::moving from state preparing -> state finished >Thread-25620::DEBUG::2012-12-02 14:01:42,881::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25620::DEBUG::2012-12-02 14:01:42,882::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25620::DEBUG::2012-12-02 14:01:42,882::task::957::TaskManager.Task::(_decref) Task=`162d01d5-ac80-4feb-bd7c-1a18e7ddd914`::ref 0 aborting False >Thread-25621::DEBUG::2012-12-02 14:01:42,885::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25621::DEBUG::2012-12-02 14:01:42,886::task::568::TaskManager.Task::(_updateState) Task=`bd6fdadd-6960-4c5d-8580-808007b6ef2c`::moving from state init -> state preparing >Thread-25621::INFO::2012-12-02 14:01:42,886::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25621::DEBUG::2012-12-02 14:01:42,886::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6577f68a-1196-427f-ae6c-ade8fc1b13a0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25621::DEBUG::2012-12-02 14:01:42,886::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25621::DEBUG::2012-12-02 14:01:42,886::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25621::DEBUG::2012-12-02 14:01:42,886::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6577f68a-1196-427f-ae6c-ade8fc1b13a0`::Granted request >Thread-25621::DEBUG::2012-12-02 14:01:42,887::task::794::TaskManager.Task::(resourceAcquired) Task=`bd6fdadd-6960-4c5d-8580-808007b6ef2c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25621::DEBUG::2012-12-02 14:01:42,887::task::957::TaskManager.Task::(_decref) Task=`bd6fdadd-6960-4c5d-8580-808007b6ef2c`::ref 1 aborting False >Thread-25621::INFO::2012-12-02 14:01:42,893::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25621::DEBUG::2012-12-02 14:01:42,893::task::1151::TaskManager.Task::(prepare) Task=`bd6fdadd-6960-4c5d-8580-808007b6ef2c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25621::DEBUG::2012-12-02 14:01:42,894::task::568::TaskManager.Task::(_updateState) Task=`bd6fdadd-6960-4c5d-8580-808007b6ef2c`::moving from state preparing -> state finished >Thread-25621::DEBUG::2012-12-02 14:01:42,894::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25621::DEBUG::2012-12-02 14:01:42,894::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25621::DEBUG::2012-12-02 14:01:42,894::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25621::DEBUG::2012-12-02 14:01:42,894::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25621::DEBUG::2012-12-02 14:01:42,894::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25621::DEBUG::2012-12-02 14:01:42,894::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25621::DEBUG::2012-12-02 14:01:42,895::task::957::TaskManager.Task::(_decref) Task=`bd6fdadd-6960-4c5d-8580-808007b6ef2c`::ref 0 aborting False >Thread-25626::DEBUG::2012-12-02 14:01:52,115::task::568::TaskManager.Task::(_updateState) Task=`95eead3c-1e0a-4a8d-b93e-da7cd4daf793`::moving from state init -> state preparing >Thread-25626::INFO::2012-12-02 14:01:52,115::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25626::INFO::2012-12-02 14:01:52,116::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00573396682739', 'lastCheck': 1354449703.7120309, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00416493415833', 'lastCheck': 1354449708.359163, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00285887718201', 'lastCheck': 1354449712.1146171, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00632214546204', 'lastCheck': 1354449708.3675539, 'code': 0, 'valid': True}} >Thread-25626::DEBUG::2012-12-02 14:01:52,116::task::1151::TaskManager.Task::(prepare) Task=`95eead3c-1e0a-4a8d-b93e-da7cd4daf793`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00573396682739', 'lastCheck': 1354449703.7120309, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00416493415833', 'lastCheck': 1354449708.359163, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00285887718201', 'lastCheck': 1354449712.1146171, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00632214546204', 'lastCheck': 1354449708.3675539, 'code': 0, 'valid': True}} >Thread-25626::DEBUG::2012-12-02 14:01:52,116::task::568::TaskManager.Task::(_updateState) Task=`95eead3c-1e0a-4a8d-b93e-da7cd4daf793`::moving from state preparing -> state finished >Thread-25626::DEBUG::2012-12-02 14:01:52,116::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25626::DEBUG::2012-12-02 14:01:52,116::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25626::DEBUG::2012-12-02 14:01:52,116::task::957::TaskManager.Task::(_decref) Task=`95eead3c-1e0a-4a8d-b93e-da7cd4daf793`::ref 0 aborting False >Thread-25628::DEBUG::2012-12-02 14:01:53,085::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25628::DEBUG::2012-12-02 14:01:53,085::task::568::TaskManager.Task::(_updateState) Task=`f236da8f-4d48-4328-a15a-9ea5d8aa8e73`::moving from state init -> state preparing >Thread-25628::INFO::2012-12-02 14:01:53,085::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25628::INFO::2012-12-02 14:01:53,085::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25628::DEBUG::2012-12-02 14:01:53,085::task::1151::TaskManager.Task::(prepare) Task=`f236da8f-4d48-4328-a15a-9ea5d8aa8e73`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25628::DEBUG::2012-12-02 14:01:53,086::task::568::TaskManager.Task::(_updateState) Task=`f236da8f-4d48-4328-a15a-9ea5d8aa8e73`::moving from state preparing -> state finished >Thread-25628::DEBUG::2012-12-02 14:01:53,086::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25628::DEBUG::2012-12-02 14:01:53,086::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25628::DEBUG::2012-12-02 14:01:53,086::task::957::TaskManager.Task::(_decref) Task=`f236da8f-4d48-4328-a15a-9ea5d8aa8e73`::ref 0 aborting False >Thread-25629::DEBUG::2012-12-02 14:01:53,090::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25629::DEBUG::2012-12-02 14:01:53,090::task::568::TaskManager.Task::(_updateState) Task=`2fdebc0e-4f91-4cdb-a8a6-f801617f7eb8`::moving from state init -> state preparing >Thread-25629::INFO::2012-12-02 14:01:53,090::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25629::DEBUG::2012-12-02 14:01:53,090::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b8087f42-2b0a-4e6e-a095-88beb3ade9da`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25629::DEBUG::2012-12-02 14:01:53,090::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25629::DEBUG::2012-12-02 14:01:53,091::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25629::DEBUG::2012-12-02 14:01:53,091::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b8087f42-2b0a-4e6e-a095-88beb3ade9da`::Granted request >Thread-25629::DEBUG::2012-12-02 14:01:53,091::task::794::TaskManager.Task::(resourceAcquired) Task=`2fdebc0e-4f91-4cdb-a8a6-f801617f7eb8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25629::DEBUG::2012-12-02 14:01:53,091::task::957::TaskManager.Task::(_decref) Task=`2fdebc0e-4f91-4cdb-a8a6-f801617f7eb8`::ref 1 aborting False >Thread-25629::INFO::2012-12-02 14:01:53,097::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25629::DEBUG::2012-12-02 14:01:53,097::task::1151::TaskManager.Task::(prepare) Task=`2fdebc0e-4f91-4cdb-a8a6-f801617f7eb8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25629::DEBUG::2012-12-02 14:01:53,097::task::568::TaskManager.Task::(_updateState) Task=`2fdebc0e-4f91-4cdb-a8a6-f801617f7eb8`::moving from state preparing -> state finished >Thread-25629::DEBUG::2012-12-02 14:01:53,097::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25629::DEBUG::2012-12-02 14:01:53,097::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25629::DEBUG::2012-12-02 14:01:53,097::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25629::DEBUG::2012-12-02 14:01:53,098::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25629::DEBUG::2012-12-02 14:01:53,098::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25629::DEBUG::2012-12-02 14:01:53,098::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25629::DEBUG::2012-12-02 14:01:53,098::task::957::TaskManager.Task::(_decref) Task=`2fdebc0e-4f91-4cdb-a8a6-f801617f7eb8`::ref 0 aborting False >Thread-25634::DEBUG::2012-12-02 14:02:02,221::task::568::TaskManager.Task::(_updateState) Task=`7c1116b6-56d6-4875-abd3-34d83e7c8371`::moving from state init -> state preparing >Thread-25634::INFO::2012-12-02 14:02:02,221::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25634::INFO::2012-12-02 14:02:02,221::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00730395317078', 'lastCheck': 1354449713.7278309, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00382900238037', 'lastCheck': 1354449718.3771679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238299369812', 'lastCheck': 1354449722.1269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809788703918', 'lastCheck': 1354449718.387711, 'code': 0, 'valid': True}} >Thread-25634::DEBUG::2012-12-02 14:02:02,221::task::1151::TaskManager.Task::(prepare) Task=`7c1116b6-56d6-4875-abd3-34d83e7c8371`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00730395317078', 'lastCheck': 1354449713.7278309, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00382900238037', 'lastCheck': 1354449718.3771679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238299369812', 'lastCheck': 1354449722.1269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809788703918', 'lastCheck': 1354449718.387711, 'code': 0, 'valid': True}} >Thread-25634::DEBUG::2012-12-02 14:02:02,221::task::568::TaskManager.Task::(_updateState) Task=`7c1116b6-56d6-4875-abd3-34d83e7c8371`::moving from state preparing -> state finished >Thread-25634::DEBUG::2012-12-02 14:02:02,222::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25634::DEBUG::2012-12-02 14:02:02,222::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25634::DEBUG::2012-12-02 14:02:02,222::task::957::TaskManager.Task::(_decref) Task=`7c1116b6-56d6-4875-abd3-34d83e7c8371`::ref 0 aborting False >Thread-25636::DEBUG::2012-12-02 14:02:03,248::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25636::DEBUG::2012-12-02 14:02:03,248::task::568::TaskManager.Task::(_updateState) Task=`6b2faf1f-92c8-4020-a084-89dad8edd98b`::moving from state init -> state preparing >Thread-25636::INFO::2012-12-02 14:02:03,248::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25636::INFO::2012-12-02 14:02:03,248::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25636::DEBUG::2012-12-02 14:02:03,248::task::1151::TaskManager.Task::(prepare) Task=`6b2faf1f-92c8-4020-a084-89dad8edd98b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25636::DEBUG::2012-12-02 14:02:03,249::task::568::TaskManager.Task::(_updateState) Task=`6b2faf1f-92c8-4020-a084-89dad8edd98b`::moving from state preparing -> state finished >Thread-25636::DEBUG::2012-12-02 14:02:03,249::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25636::DEBUG::2012-12-02 14:02:03,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25636::DEBUG::2012-12-02 14:02:03,249::task::957::TaskManager.Task::(_decref) Task=`6b2faf1f-92c8-4020-a084-89dad8edd98b`::ref 0 aborting False >Thread-25637::DEBUG::2012-12-02 14:02:03,251::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25637::DEBUG::2012-12-02 14:02:03,251::task::568::TaskManager.Task::(_updateState) Task=`8b3bfd4b-6cf3-4aa4-9397-d8316152a901`::moving from state init -> state preparing >Thread-25637::INFO::2012-12-02 14:02:03,252::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25637::DEBUG::2012-12-02 14:02:03,252::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f1ba1b8d-e9a6-4128-a2f6-f171098cd814`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25637::DEBUG::2012-12-02 14:02:03,252::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25637::DEBUG::2012-12-02 14:02:03,252::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25637::DEBUG::2012-12-02 14:02:03,252::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f1ba1b8d-e9a6-4128-a2f6-f171098cd814`::Granted request >Thread-25637::DEBUG::2012-12-02 14:02:03,253::task::794::TaskManager.Task::(resourceAcquired) Task=`8b3bfd4b-6cf3-4aa4-9397-d8316152a901`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25637::DEBUG::2012-12-02 14:02:03,253::task::957::TaskManager.Task::(_decref) Task=`8b3bfd4b-6cf3-4aa4-9397-d8316152a901`::ref 1 aborting False >Thread-25637::INFO::2012-12-02 14:02:03,258::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25637::DEBUG::2012-12-02 14:02:03,258::task::1151::TaskManager.Task::(prepare) Task=`8b3bfd4b-6cf3-4aa4-9397-d8316152a901`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25637::DEBUG::2012-12-02 14:02:03,258::task::568::TaskManager.Task::(_updateState) Task=`8b3bfd4b-6cf3-4aa4-9397-d8316152a901`::moving from state preparing -> state finished >Thread-25637::DEBUG::2012-12-02 14:02:03,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25637::DEBUG::2012-12-02 14:02:03,259::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25637::DEBUG::2012-12-02 14:02:03,259::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25637::DEBUG::2012-12-02 14:02:03,259::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25637::DEBUG::2012-12-02 14:02:03,259::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25637::DEBUG::2012-12-02 14:02:03,259::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25637::DEBUG::2012-12-02 14:02:03,259::task::957::TaskManager.Task::(_decref) Task=`8b3bfd4b-6cf3-4aa4-9397-d8316152a901`::ref 0 aborting False >Thread-25642::DEBUG::2012-12-02 14:02:12,358::task::568::TaskManager.Task::(_updateState) Task=`865ce0c6-13a4-41b1-a084-90e0f9c2ae3f`::moving from state init -> state preparing >Thread-25642::INFO::2012-12-02 14:02:12,358::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25642::INFO::2012-12-02 14:02:12,358::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00614595413208', 'lastCheck': 1354449723.7428961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00237393379211', 'lastCheck': 1354449728.394228, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00295996665955', 'lastCheck': 1354449732.1404991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00625085830688', 'lastCheck': 1354449728.40271, 'code': 0, 'valid': True}} >Thread-25642::DEBUG::2012-12-02 14:02:12,358::task::1151::TaskManager.Task::(prepare) Task=`865ce0c6-13a4-41b1-a084-90e0f9c2ae3f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00614595413208', 'lastCheck': 1354449723.7428961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00237393379211', 'lastCheck': 1354449728.394228, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00295996665955', 'lastCheck': 1354449732.1404991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00625085830688', 'lastCheck': 1354449728.40271, 'code': 0, 'valid': True}} >Thread-25642::DEBUG::2012-12-02 14:02:12,359::task::568::TaskManager.Task::(_updateState) Task=`865ce0c6-13a4-41b1-a084-90e0f9c2ae3f`::moving from state preparing -> state finished >Thread-25642::DEBUG::2012-12-02 14:02:12,359::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25642::DEBUG::2012-12-02 14:02:12,359::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25642::DEBUG::2012-12-02 14:02:12,359::task::957::TaskManager.Task::(_decref) Task=`865ce0c6-13a4-41b1-a084-90e0f9c2ae3f`::ref 0 aborting False >Thread-25644::DEBUG::2012-12-02 14:02:13,425::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25644::DEBUG::2012-12-02 14:02:13,426::task::568::TaskManager.Task::(_updateState) Task=`76af8301-b113-4c4e-a8aa-7767d600a236`::moving from state init -> state preparing >Thread-25644::INFO::2012-12-02 14:02:13,426::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25644::INFO::2012-12-02 14:02:13,426::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25644::DEBUG::2012-12-02 14:02:13,426::task::1151::TaskManager.Task::(prepare) Task=`76af8301-b113-4c4e-a8aa-7767d600a236`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25644::DEBUG::2012-12-02 14:02:13,426::task::568::TaskManager.Task::(_updateState) Task=`76af8301-b113-4c4e-a8aa-7767d600a236`::moving from state preparing -> state finished >Thread-25644::DEBUG::2012-12-02 14:02:13,426::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25644::DEBUG::2012-12-02 14:02:13,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25644::DEBUG::2012-12-02 14:02:13,427::task::957::TaskManager.Task::(_decref) Task=`76af8301-b113-4c4e-a8aa-7767d600a236`::ref 0 aborting False >Thread-25645::DEBUG::2012-12-02 14:02:13,429::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25645::DEBUG::2012-12-02 14:02:13,430::task::568::TaskManager.Task::(_updateState) Task=`3b41e608-f5a0-4417-a6ae-8c2c9cb8650c`::moving from state init -> state preparing >Thread-25645::INFO::2012-12-02 14:02:13,430::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25645::DEBUG::2012-12-02 14:02:13,430::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`90723b67-0869-4772-864c-4ace03ed1e55`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25645::DEBUG::2012-12-02 14:02:13,430::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25645::DEBUG::2012-12-02 14:02:13,430::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25645::DEBUG::2012-12-02 14:02:13,430::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`90723b67-0869-4772-864c-4ace03ed1e55`::Granted request >Thread-25645::DEBUG::2012-12-02 14:02:13,431::task::794::TaskManager.Task::(resourceAcquired) Task=`3b41e608-f5a0-4417-a6ae-8c2c9cb8650c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25645::DEBUG::2012-12-02 14:02:13,431::task::957::TaskManager.Task::(_decref) Task=`3b41e608-f5a0-4417-a6ae-8c2c9cb8650c`::ref 1 aborting False >Thread-25645::INFO::2012-12-02 14:02:13,437::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25645::DEBUG::2012-12-02 14:02:13,437::task::1151::TaskManager.Task::(prepare) Task=`3b41e608-f5a0-4417-a6ae-8c2c9cb8650c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25645::DEBUG::2012-12-02 14:02:13,437::task::568::TaskManager.Task::(_updateState) Task=`3b41e608-f5a0-4417-a6ae-8c2c9cb8650c`::moving from state preparing -> state finished >Thread-25645::DEBUG::2012-12-02 14:02:13,437::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25645::DEBUG::2012-12-02 14:02:13,437::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25645::DEBUG::2012-12-02 14:02:13,437::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25645::DEBUG::2012-12-02 14:02:13,437::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25645::DEBUG::2012-12-02 14:02:13,438::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25645::DEBUG::2012-12-02 14:02:13,438::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25645::DEBUG::2012-12-02 14:02:13,438::task::957::TaskManager.Task::(_decref) Task=`3b41e608-f5a0-4417-a6ae-8c2c9cb8650c`::ref 0 aborting False >Thread-25650::DEBUG::2012-12-02 14:02:22,432::task::568::TaskManager.Task::(_updateState) Task=`42eb18bf-282b-4ccb-9651-bb296fabe00a`::moving from state init -> state preparing >Thread-25650::INFO::2012-12-02 14:02:22,432::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25650::INFO::2012-12-02 14:02:22,433::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00623798370361', 'lastCheck': 1354449733.7571959, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00218892097473', 'lastCheck': 1354449738.4149041, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00185298919678', 'lastCheck': 1354449742.1520679, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00840902328491', 'lastCheck': 1354449738.422384, 'code': 0, 'valid': True}} >Thread-25650::DEBUG::2012-12-02 14:02:22,433::task::1151::TaskManager.Task::(prepare) Task=`42eb18bf-282b-4ccb-9651-bb296fabe00a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00623798370361', 'lastCheck': 1354449733.7571959, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00218892097473', 'lastCheck': 1354449738.4149041, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00185298919678', 'lastCheck': 1354449742.1520679, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00840902328491', 'lastCheck': 1354449738.422384, 'code': 0, 'valid': True}} >Thread-25650::DEBUG::2012-12-02 14:02:22,433::task::568::TaskManager.Task::(_updateState) Task=`42eb18bf-282b-4ccb-9651-bb296fabe00a`::moving from state preparing -> state finished >Thread-25650::DEBUG::2012-12-02 14:02:22,433::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25650::DEBUG::2012-12-02 14:02:22,433::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25650::DEBUG::2012-12-02 14:02:22,433::task::957::TaskManager.Task::(_decref) Task=`42eb18bf-282b-4ccb-9651-bb296fabe00a`::ref 0 aborting False >Thread-25652::DEBUG::2012-12-02 14:02:23,563::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25652::DEBUG::2012-12-02 14:02:23,563::task::568::TaskManager.Task::(_updateState) Task=`0707c3f9-9ded-4913-8d66-e5b68c5121c7`::moving from state init -> state preparing >Thread-25652::INFO::2012-12-02 14:02:23,563::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25652::INFO::2012-12-02 14:02:23,564::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25652::DEBUG::2012-12-02 14:02:23,564::task::1151::TaskManager.Task::(prepare) Task=`0707c3f9-9ded-4913-8d66-e5b68c5121c7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25652::DEBUG::2012-12-02 14:02:23,564::task::568::TaskManager.Task::(_updateState) Task=`0707c3f9-9ded-4913-8d66-e5b68c5121c7`::moving from state preparing -> state finished >Thread-25652::DEBUG::2012-12-02 14:02:23,564::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25652::DEBUG::2012-12-02 14:02:23,564::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25652::DEBUG::2012-12-02 14:02:23,564::task::957::TaskManager.Task::(_decref) Task=`0707c3f9-9ded-4913-8d66-e5b68c5121c7`::ref 0 aborting False >Thread-25653::DEBUG::2012-12-02 14:02:23,567::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25653::DEBUG::2012-12-02 14:02:23,567::task::568::TaskManager.Task::(_updateState) Task=`4c1c8065-a41f-4ec2-8036-cf96c5a2569d`::moving from state init -> state preparing >Thread-25653::INFO::2012-12-02 14:02:23,567::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25653::DEBUG::2012-12-02 14:02:23,567::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`58f25534-aa75-42a7-b08c-c88416516af5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25653::DEBUG::2012-12-02 14:02:23,568::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25653::DEBUG::2012-12-02 14:02:23,568::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25653::DEBUG::2012-12-02 14:02:23,568::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`58f25534-aa75-42a7-b08c-c88416516af5`::Granted request >Thread-25653::DEBUG::2012-12-02 14:02:23,568::task::794::TaskManager.Task::(resourceAcquired) Task=`4c1c8065-a41f-4ec2-8036-cf96c5a2569d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25653::DEBUG::2012-12-02 14:02:23,568::task::957::TaskManager.Task::(_decref) Task=`4c1c8065-a41f-4ec2-8036-cf96c5a2569d`::ref 1 aborting False >Thread-25653::INFO::2012-12-02 14:02:23,575::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25653::DEBUG::2012-12-02 14:02:23,575::task::1151::TaskManager.Task::(prepare) Task=`4c1c8065-a41f-4ec2-8036-cf96c5a2569d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25653::DEBUG::2012-12-02 14:02:23,575::task::568::TaskManager.Task::(_updateState) Task=`4c1c8065-a41f-4ec2-8036-cf96c5a2569d`::moving from state preparing -> state finished >Thread-25653::DEBUG::2012-12-02 14:02:23,575::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25653::DEBUG::2012-12-02 14:02:23,575::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25653::DEBUG::2012-12-02 14:02:23,575::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25653::DEBUG::2012-12-02 14:02:23,575::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25653::DEBUG::2012-12-02 14:02:23,576::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25653::DEBUG::2012-12-02 14:02:23,576::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25653::DEBUG::2012-12-02 14:02:23,576::task::957::TaskManager.Task::(_decref) Task=`4c1c8065-a41f-4ec2-8036-cf96c5a2569d`::ref 0 aborting False >Thread-25658::DEBUG::2012-12-02 14:02:32,497::task::568::TaskManager.Task::(_updateState) Task=`d3a4ce7c-4ca0-4cee-91db-fd9f0d9f970c`::moving from state init -> state preparing >Thread-25658::INFO::2012-12-02 14:02:32,498::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25658::INFO::2012-12-02 14:02:32,498::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00912594795227', 'lastCheck': 1354449743.775598, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00422811508179', 'lastCheck': 1354449748.430213, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00234317779541', 'lastCheck': 1354449752.1660011, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00638389587402', 'lastCheck': 1354449748.438226, 'code': 0, 'valid': True}} >Thread-25658::DEBUG::2012-12-02 14:02:32,498::task::1151::TaskManager.Task::(prepare) Task=`d3a4ce7c-4ca0-4cee-91db-fd9f0d9f970c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00912594795227', 'lastCheck': 1354449743.775598, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00422811508179', 'lastCheck': 1354449748.430213, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00234317779541', 'lastCheck': 1354449752.1660011, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00638389587402', 'lastCheck': 1354449748.438226, 'code': 0, 'valid': True}} >Thread-25658::DEBUG::2012-12-02 14:02:32,498::task::568::TaskManager.Task::(_updateState) Task=`d3a4ce7c-4ca0-4cee-91db-fd9f0d9f970c`::moving from state preparing -> state finished >Thread-25658::DEBUG::2012-12-02 14:02:32,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25658::DEBUG::2012-12-02 14:02:32,498::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25658::DEBUG::2012-12-02 14:02:32,498::task::957::TaskManager.Task::(_decref) Task=`d3a4ce7c-4ca0-4cee-91db-fd9f0d9f970c`::ref 0 aborting False >Thread-25660::DEBUG::2012-12-02 14:02:33,692::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25660::DEBUG::2012-12-02 14:02:33,692::task::568::TaskManager.Task::(_updateState) Task=`a5621ff7-d9b0-4e31-9d1a-942e1a7c4148`::moving from state init -> state preparing >Thread-25660::INFO::2012-12-02 14:02:33,692::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25660::INFO::2012-12-02 14:02:33,692::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25660::DEBUG::2012-12-02 14:02:33,693::task::1151::TaskManager.Task::(prepare) Task=`a5621ff7-d9b0-4e31-9d1a-942e1a7c4148`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25660::DEBUG::2012-12-02 14:02:33,693::task::568::TaskManager.Task::(_updateState) Task=`a5621ff7-d9b0-4e31-9d1a-942e1a7c4148`::moving from state preparing -> state finished >Thread-25660::DEBUG::2012-12-02 14:02:33,693::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25660::DEBUG::2012-12-02 14:02:33,693::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25660::DEBUG::2012-12-02 14:02:33,693::task::957::TaskManager.Task::(_decref) Task=`a5621ff7-d9b0-4e31-9d1a-942e1a7c4148`::ref 0 aborting False >Thread-25661::DEBUG::2012-12-02 14:02:33,695::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25661::DEBUG::2012-12-02 14:02:33,696::task::568::TaskManager.Task::(_updateState) Task=`978f7889-6009-4a74-8dcd-a65c401cd758`::moving from state init -> state preparing >Thread-25661::INFO::2012-12-02 14:02:33,696::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25661::DEBUG::2012-12-02 14:02:33,696::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`151c79ed-0d32-4b70-8d04-de5f0c75bca9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25661::DEBUG::2012-12-02 14:02:33,696::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25661::DEBUG::2012-12-02 14:02:33,696::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25661::DEBUG::2012-12-02 14:02:33,697::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`151c79ed-0d32-4b70-8d04-de5f0c75bca9`::Granted request >Thread-25661::DEBUG::2012-12-02 14:02:33,697::task::794::TaskManager.Task::(resourceAcquired) Task=`978f7889-6009-4a74-8dcd-a65c401cd758`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25661::DEBUG::2012-12-02 14:02:33,697::task::957::TaskManager.Task::(_decref) Task=`978f7889-6009-4a74-8dcd-a65c401cd758`::ref 1 aborting False >Thread-25661::INFO::2012-12-02 14:02:33,701::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25661::DEBUG::2012-12-02 14:02:33,701::task::1151::TaskManager.Task::(prepare) Task=`978f7889-6009-4a74-8dcd-a65c401cd758`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25661::DEBUG::2012-12-02 14:02:33,701::task::568::TaskManager.Task::(_updateState) Task=`978f7889-6009-4a74-8dcd-a65c401cd758`::moving from state preparing -> state finished >Thread-25661::DEBUG::2012-12-02 14:02:33,701::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25661::DEBUG::2012-12-02 14:02:33,702::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25661::DEBUG::2012-12-02 14:02:33,702::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25661::DEBUG::2012-12-02 14:02:33,702::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25661::DEBUG::2012-12-02 14:02:33,702::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25661::DEBUG::2012-12-02 14:02:33,702::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25661::DEBUG::2012-12-02 14:02:33,702::task::957::TaskManager.Task::(_decref) Task=`978f7889-6009-4a74-8dcd-a65c401cd758`::ref 0 aborting False >Thread-25666::DEBUG::2012-12-02 14:02:42,608::task::568::TaskManager.Task::(_updateState) Task=`dcde8903-e92c-4ed0-83ab-da7a816e95f6`::moving from state init -> state preparing >Thread-25666::INFO::2012-12-02 14:02:42,608::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25666::INFO::2012-12-02 14:02:42,608::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794720649719', 'lastCheck': 1354449753.795337, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00501394271851', 'lastCheck': 1354449758.4471121, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00245714187622', 'lastCheck': 1354449762.1780951, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00757598876953', 'lastCheck': 1354449758.4549911, 'code': 0, 'valid': True}} >Thread-25666::DEBUG::2012-12-02 14:02:42,608::task::1151::TaskManager.Task::(prepare) Task=`dcde8903-e92c-4ed0-83ab-da7a816e95f6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794720649719', 'lastCheck': 1354449753.795337, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00501394271851', 'lastCheck': 1354449758.4471121, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00245714187622', 'lastCheck': 1354449762.1780951, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00757598876953', 'lastCheck': 1354449758.4549911, 'code': 0, 'valid': True}} >Thread-25666::DEBUG::2012-12-02 14:02:42,609::task::568::TaskManager.Task::(_updateState) Task=`dcde8903-e92c-4ed0-83ab-da7a816e95f6`::moving from state preparing -> state finished >Thread-25666::DEBUG::2012-12-02 14:02:42,609::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25666::DEBUG::2012-12-02 14:02:42,609::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25666::DEBUG::2012-12-02 14:02:42,609::task::957::TaskManager.Task::(_decref) Task=`dcde8903-e92c-4ed0-83ab-da7a816e95f6`::ref 0 aborting False >Thread-25668::DEBUG::2012-12-02 14:02:43,807::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25668::DEBUG::2012-12-02 14:02:43,807::task::568::TaskManager.Task::(_updateState) Task=`b8895547-1896-477c-9e98-ec0c6bcfbc90`::moving from state init -> state preparing >Thread-25668::INFO::2012-12-02 14:02:43,808::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25668::INFO::2012-12-02 14:02:43,808::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25668::DEBUG::2012-12-02 14:02:43,808::task::1151::TaskManager.Task::(prepare) Task=`b8895547-1896-477c-9e98-ec0c6bcfbc90`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25668::DEBUG::2012-12-02 14:02:43,808::task::568::TaskManager.Task::(_updateState) Task=`b8895547-1896-477c-9e98-ec0c6bcfbc90`::moving from state preparing -> state finished >Thread-25668::DEBUG::2012-12-02 14:02:43,808::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25668::DEBUG::2012-12-02 14:02:43,808::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25668::DEBUG::2012-12-02 14:02:43,808::task::957::TaskManager.Task::(_decref) Task=`b8895547-1896-477c-9e98-ec0c6bcfbc90`::ref 0 aborting False >Thread-25669::DEBUG::2012-12-02 14:02:43,812::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25669::DEBUG::2012-12-02 14:02:43,812::task::568::TaskManager.Task::(_updateState) Task=`c0cb3131-ae6d-4279-9ddc-ba0b7f98619b`::moving from state init -> state preparing >Thread-25669::INFO::2012-12-02 14:02:43,812::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25669::DEBUG::2012-12-02 14:02:43,812::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`db4fa472-a044-43eb-a1cc-b23518e09d33`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25669::DEBUG::2012-12-02 14:02:43,813::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25669::DEBUG::2012-12-02 14:02:43,813::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25669::DEBUG::2012-12-02 14:02:43,813::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`db4fa472-a044-43eb-a1cc-b23518e09d33`::Granted request >Thread-25669::DEBUG::2012-12-02 14:02:43,813::task::794::TaskManager.Task::(resourceAcquired) Task=`c0cb3131-ae6d-4279-9ddc-ba0b7f98619b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25669::DEBUG::2012-12-02 14:02:43,813::task::957::TaskManager.Task::(_decref) Task=`c0cb3131-ae6d-4279-9ddc-ba0b7f98619b`::ref 1 aborting False >Thread-25669::INFO::2012-12-02 14:02:43,819::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25669::DEBUG::2012-12-02 14:02:43,819::task::1151::TaskManager.Task::(prepare) Task=`c0cb3131-ae6d-4279-9ddc-ba0b7f98619b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25669::DEBUG::2012-12-02 14:02:43,819::task::568::TaskManager.Task::(_updateState) Task=`c0cb3131-ae6d-4279-9ddc-ba0b7f98619b`::moving from state preparing -> state finished >Thread-25669::DEBUG::2012-12-02 14:02:43,819::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25669::DEBUG::2012-12-02 14:02:43,819::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25669::DEBUG::2012-12-02 14:02:43,820::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25669::DEBUG::2012-12-02 14:02:43,820::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25669::DEBUG::2012-12-02 14:02:43,820::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25669::DEBUG::2012-12-02 14:02:43,820::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25669::DEBUG::2012-12-02 14:02:43,820::task::957::TaskManager.Task::(_decref) Task=`c0cb3131-ae6d-4279-9ddc-ba0b7f98619b`::ref 0 aborting False >Thread-25674::DEBUG::2012-12-02 14:02:52,762::task::568::TaskManager.Task::(_updateState) Task=`c38ac53a-1767-4879-9c6f-1c51bdf4b08e`::moving from state init -> state preparing >Thread-25674::INFO::2012-12-02 14:02:52,763::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25674::INFO::2012-12-02 14:02:52,763::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0123651027679', 'lastCheck': 1354449763.8195801, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00253701210022', 'lastCheck': 1354449768.462028, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00219202041626', 'lastCheck': 1354449772.1896601, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00730109214783', 'lastCheck': 1354449768.471786, 'code': 0, 'valid': True}} >Thread-25674::DEBUG::2012-12-02 14:02:52,763::task::1151::TaskManager.Task::(prepare) Task=`c38ac53a-1767-4879-9c6f-1c51bdf4b08e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0123651027679', 'lastCheck': 1354449763.8195801, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00253701210022', 'lastCheck': 1354449768.462028, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00219202041626', 'lastCheck': 1354449772.1896601, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00730109214783', 'lastCheck': 1354449768.471786, 'code': 0, 'valid': True}} >Thread-25674::DEBUG::2012-12-02 14:02:52,763::task::568::TaskManager.Task::(_updateState) Task=`c38ac53a-1767-4879-9c6f-1c51bdf4b08e`::moving from state preparing -> state finished >Thread-25674::DEBUG::2012-12-02 14:02:52,763::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25674::DEBUG::2012-12-02 14:02:52,763::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25674::DEBUG::2012-12-02 14:02:52,764::task::957::TaskManager.Task::(_decref) Task=`c38ac53a-1767-4879-9c6f-1c51bdf4b08e`::ref 0 aborting False >Thread-25676::DEBUG::2012-12-02 14:02:54,004::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25676::DEBUG::2012-12-02 14:02:54,005::task::568::TaskManager.Task::(_updateState) Task=`45952778-dbea-4dc7-a96d-d77fe2a19ac8`::moving from state init -> state preparing >Thread-25676::INFO::2012-12-02 14:02:54,005::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25676::INFO::2012-12-02 14:02:54,005::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25676::DEBUG::2012-12-02 14:02:54,005::task::1151::TaskManager.Task::(prepare) Task=`45952778-dbea-4dc7-a96d-d77fe2a19ac8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25676::DEBUG::2012-12-02 14:02:54,005::task::568::TaskManager.Task::(_updateState) Task=`45952778-dbea-4dc7-a96d-d77fe2a19ac8`::moving from state preparing -> state finished >Thread-25676::DEBUG::2012-12-02 14:02:54,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25676::DEBUG::2012-12-02 14:02:54,005::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25676::DEBUG::2012-12-02 14:02:54,006::task::957::TaskManager.Task::(_decref) Task=`45952778-dbea-4dc7-a96d-d77fe2a19ac8`::ref 0 aborting False >Thread-25677::DEBUG::2012-12-02 14:02:54,009::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25677::DEBUG::2012-12-02 14:02:54,009::task::568::TaskManager.Task::(_updateState) Task=`ec19073a-0d36-449c-9c6d-e16f5e8f2aa7`::moving from state init -> state preparing >Thread-25677::INFO::2012-12-02 14:02:54,009::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25677::DEBUG::2012-12-02 14:02:54,009::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d217d69c-6a3a-44aa-a03c-463a3937fc60`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25677::DEBUG::2012-12-02 14:02:54,010::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25677::DEBUG::2012-12-02 14:02:54,010::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25677::DEBUG::2012-12-02 14:02:54,010::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d217d69c-6a3a-44aa-a03c-463a3937fc60`::Granted request >Thread-25677::DEBUG::2012-12-02 14:02:54,010::task::794::TaskManager.Task::(resourceAcquired) Task=`ec19073a-0d36-449c-9c6d-e16f5e8f2aa7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25677::DEBUG::2012-12-02 14:02:54,010::task::957::TaskManager.Task::(_decref) Task=`ec19073a-0d36-449c-9c6d-e16f5e8f2aa7`::ref 1 aborting False >Thread-25677::INFO::2012-12-02 14:02:54,014::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25677::DEBUG::2012-12-02 14:02:54,014::task::1151::TaskManager.Task::(prepare) Task=`ec19073a-0d36-449c-9c6d-e16f5e8f2aa7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25677::DEBUG::2012-12-02 14:02:54,015::task::568::TaskManager.Task::(_updateState) Task=`ec19073a-0d36-449c-9c6d-e16f5e8f2aa7`::moving from state preparing -> state finished >Thread-25677::DEBUG::2012-12-02 14:02:54,015::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25677::DEBUG::2012-12-02 14:02:54,015::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25677::DEBUG::2012-12-02 14:02:54,015::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25677::DEBUG::2012-12-02 14:02:54,015::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25677::DEBUG::2012-12-02 14:02:54,015::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25677::DEBUG::2012-12-02 14:02:54,015::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25677::DEBUG::2012-12-02 14:02:54,015::task::957::TaskManager.Task::(_decref) Task=`ec19073a-0d36-449c-9c6d-e16f5e8f2aa7`::ref 0 aborting False >Thread-25682::DEBUG::2012-12-02 14:03:02,862::task::568::TaskManager.Task::(_updateState) Task=`0d32c5d5-d67a-4f07-8de8-96e89f4f7823`::moving from state init -> state preparing >Thread-25682::INFO::2012-12-02 14:03:02,862::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25682::INFO::2012-12-02 14:03:02,862::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00961112976074', 'lastCheck': 1354449773.838479, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.001797914505', 'lastCheck': 1354449778.4771211, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00197982788086', 'lastCheck': 1354449782.200794, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778603553772', 'lastCheck': 1354449778.48879, 'code': 0, 'valid': True}} >Thread-25682::DEBUG::2012-12-02 14:03:02,862::task::1151::TaskManager.Task::(prepare) Task=`0d32c5d5-d67a-4f07-8de8-96e89f4f7823`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00961112976074', 'lastCheck': 1354449773.838479, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.001797914505', 'lastCheck': 1354449778.4771211, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00197982788086', 'lastCheck': 1354449782.200794, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778603553772', 'lastCheck': 1354449778.48879, 'code': 0, 'valid': True}} >Thread-25682::DEBUG::2012-12-02 14:03:02,862::task::568::TaskManager.Task::(_updateState) Task=`0d32c5d5-d67a-4f07-8de8-96e89f4f7823`::moving from state preparing -> state finished >Thread-25682::DEBUG::2012-12-02 14:03:02,863::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25682::DEBUG::2012-12-02 14:03:02,863::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25682::DEBUG::2012-12-02 14:03:02,863::task::957::TaskManager.Task::(_decref) Task=`0d32c5d5-d67a-4f07-8de8-96e89f4f7823`::ref 0 aborting False >Thread-25684::DEBUG::2012-12-02 14:03:04,148::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25684::DEBUG::2012-12-02 14:03:04,149::task::568::TaskManager.Task::(_updateState) Task=`acafcb87-fa49-4082-9478-57cea5cf3795`::moving from state init -> state preparing >Thread-25684::INFO::2012-12-02 14:03:04,149::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25684::INFO::2012-12-02 14:03:04,149::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25684::DEBUG::2012-12-02 14:03:04,149::task::1151::TaskManager.Task::(prepare) Task=`acafcb87-fa49-4082-9478-57cea5cf3795`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25684::DEBUG::2012-12-02 14:03:04,149::task::568::TaskManager.Task::(_updateState) Task=`acafcb87-fa49-4082-9478-57cea5cf3795`::moving from state preparing -> state finished >Thread-25684::DEBUG::2012-12-02 14:03:04,150::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25684::DEBUG::2012-12-02 14:03:04,150::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25684::DEBUG::2012-12-02 14:03:04,150::task::957::TaskManager.Task::(_decref) Task=`acafcb87-fa49-4082-9478-57cea5cf3795`::ref 0 aborting False >Thread-25685::DEBUG::2012-12-02 14:03:04,152::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25685::DEBUG::2012-12-02 14:03:04,153::task::568::TaskManager.Task::(_updateState) Task=`639a1a39-1fa3-4d3c-8775-0ac9aa3eca1a`::moving from state init -> state preparing >Thread-25685::INFO::2012-12-02 14:03:04,153::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25685::DEBUG::2012-12-02 14:03:04,153::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3c06b5ce-0980-4059-ba33-bd5b356121fa`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25685::DEBUG::2012-12-02 14:03:04,153::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25685::DEBUG::2012-12-02 14:03:04,153::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25685::DEBUG::2012-12-02 14:03:04,154::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3c06b5ce-0980-4059-ba33-bd5b356121fa`::Granted request >Thread-25685::DEBUG::2012-12-02 14:03:04,154::task::794::TaskManager.Task::(resourceAcquired) Task=`639a1a39-1fa3-4d3c-8775-0ac9aa3eca1a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25685::DEBUG::2012-12-02 14:03:04,154::task::957::TaskManager.Task::(_decref) Task=`639a1a39-1fa3-4d3c-8775-0ac9aa3eca1a`::ref 1 aborting False >Thread-25685::INFO::2012-12-02 14:03:04,159::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25685::DEBUG::2012-12-02 14:03:04,159::task::1151::TaskManager.Task::(prepare) Task=`639a1a39-1fa3-4d3c-8775-0ac9aa3eca1a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25685::DEBUG::2012-12-02 14:03:04,159::task::568::TaskManager.Task::(_updateState) Task=`639a1a39-1fa3-4d3c-8775-0ac9aa3eca1a`::moving from state preparing -> state finished >Thread-25685::DEBUG::2012-12-02 14:03:04,159::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25685::DEBUG::2012-12-02 14:03:04,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25685::DEBUG::2012-12-02 14:03:04,159::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25685::DEBUG::2012-12-02 14:03:04,160::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25685::DEBUG::2012-12-02 14:03:04,160::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25685::DEBUG::2012-12-02 14:03:04,160::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25685::DEBUG::2012-12-02 14:03:04,160::task::957::TaskManager.Task::(_decref) Task=`639a1a39-1fa3-4d3c-8775-0ac9aa3eca1a`::ref 0 aborting False >Thread-25690::DEBUG::2012-12-02 14:03:12,922::task::568::TaskManager.Task::(_updateState) Task=`17675628-6c70-4cb6-b2db-6041f711cdb6`::moving from state init -> state preparing >Thread-25690::INFO::2012-12-02 14:03:12,922::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25690::INFO::2012-12-02 14:03:12,923::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00797414779663', 'lastCheck': 1354449783.8561161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00273108482361', 'lastCheck': 1354449788.496314, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00309109687805', 'lastCheck': 1354449792.2148869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00659108161926', 'lastCheck': 1354449788.5051551, 'code': 0, 'valid': True}} >Thread-25690::DEBUG::2012-12-02 14:03:12,923::task::1151::TaskManager.Task::(prepare) Task=`17675628-6c70-4cb6-b2db-6041f711cdb6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00797414779663', 'lastCheck': 1354449783.8561161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00273108482361', 'lastCheck': 1354449788.496314, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00309109687805', 'lastCheck': 1354449792.2148869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00659108161926', 'lastCheck': 1354449788.5051551, 'code': 0, 'valid': True}} >Thread-25690::DEBUG::2012-12-02 14:03:12,923::task::568::TaskManager.Task::(_updateState) Task=`17675628-6c70-4cb6-b2db-6041f711cdb6`::moving from state preparing -> state finished >Thread-25690::DEBUG::2012-12-02 14:03:12,923::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25690::DEBUG::2012-12-02 14:03:12,923::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25690::DEBUG::2012-12-02 14:03:12,923::task::957::TaskManager.Task::(_decref) Task=`17675628-6c70-4cb6-b2db-6041f711cdb6`::ref 0 aborting False >Thread-25692::DEBUG::2012-12-02 14:03:14,352::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25692::DEBUG::2012-12-02 14:03:14,352::task::568::TaskManager.Task::(_updateState) Task=`697a9711-8c2d-45bd-b997-4972f138588a`::moving from state init -> state preparing >Thread-25692::INFO::2012-12-02 14:03:14,352::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25692::INFO::2012-12-02 14:03:14,353::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25692::DEBUG::2012-12-02 14:03:14,353::task::1151::TaskManager.Task::(prepare) Task=`697a9711-8c2d-45bd-b997-4972f138588a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25692::DEBUG::2012-12-02 14:03:14,353::task::568::TaskManager.Task::(_updateState) Task=`697a9711-8c2d-45bd-b997-4972f138588a`::moving from state preparing -> state finished >Thread-25692::DEBUG::2012-12-02 14:03:14,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25692::DEBUG::2012-12-02 14:03:14,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25692::DEBUG::2012-12-02 14:03:14,353::task::957::TaskManager.Task::(_decref) Task=`697a9711-8c2d-45bd-b997-4972f138588a`::ref 0 aborting False >Thread-25693::DEBUG::2012-12-02 14:03:14,357::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25693::DEBUG::2012-12-02 14:03:14,358::task::568::TaskManager.Task::(_updateState) Task=`d56f173f-ff2f-46aa-ab1f-135f5dae2a20`::moving from state init -> state preparing >Thread-25693::INFO::2012-12-02 14:03:14,358::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25693::DEBUG::2012-12-02 14:03:14,358::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5b975e72-d77f-470f-8b2a-cd9ea7ac3ec4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25693::DEBUG::2012-12-02 14:03:14,358::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25693::DEBUG::2012-12-02 14:03:14,358::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25693::DEBUG::2012-12-02 14:03:14,358::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5b975e72-d77f-470f-8b2a-cd9ea7ac3ec4`::Granted request >Thread-25693::DEBUG::2012-12-02 14:03:14,359::task::794::TaskManager.Task::(resourceAcquired) Task=`d56f173f-ff2f-46aa-ab1f-135f5dae2a20`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25693::DEBUG::2012-12-02 14:03:14,359::task::957::TaskManager.Task::(_decref) Task=`d56f173f-ff2f-46aa-ab1f-135f5dae2a20`::ref 1 aborting False >Thread-25693::INFO::2012-12-02 14:03:14,365::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25693::DEBUG::2012-12-02 14:03:14,365::task::1151::TaskManager.Task::(prepare) Task=`d56f173f-ff2f-46aa-ab1f-135f5dae2a20`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25693::DEBUG::2012-12-02 14:03:14,365::task::568::TaskManager.Task::(_updateState) Task=`d56f173f-ff2f-46aa-ab1f-135f5dae2a20`::moving from state preparing -> state finished >Thread-25693::DEBUG::2012-12-02 14:03:14,366::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25693::DEBUG::2012-12-02 14:03:14,366::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25693::DEBUG::2012-12-02 14:03:14,366::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25693::DEBUG::2012-12-02 14:03:14,366::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25693::DEBUG::2012-12-02 14:03:14,366::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25693::DEBUG::2012-12-02 14:03:14,366::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25693::DEBUG::2012-12-02 14:03:14,366::task::957::TaskManager.Task::(_decref) Task=`d56f173f-ff2f-46aa-ab1f-135f5dae2a20`::ref 0 aborting False >Thread-25698::DEBUG::2012-12-02 14:03:22,981::task::568::TaskManager.Task::(_updateState) Task=`cef46e20-df53-4881-9cb6-0fd66e258cb1`::moving from state init -> state preparing >Thread-25698::INFO::2012-12-02 14:03:22,981::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25698::INFO::2012-12-02 14:03:22,981::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00760388374329', 'lastCheck': 1354449793.8730841, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00275301933289', 'lastCheck': 1354449798.514015, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0024631023407', 'lastCheck': 1354449802.2268839, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00405597686768', 'lastCheck': 1354449798.5173631, 'code': 0, 'valid': True}} >Thread-25698::DEBUG::2012-12-02 14:03:22,982::task::1151::TaskManager.Task::(prepare) Task=`cef46e20-df53-4881-9cb6-0fd66e258cb1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00760388374329', 'lastCheck': 1354449793.8730841, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00275301933289', 'lastCheck': 1354449798.514015, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0024631023407', 'lastCheck': 1354449802.2268839, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00405597686768', 'lastCheck': 1354449798.5173631, 'code': 0, 'valid': True}} >Thread-25698::DEBUG::2012-12-02 14:03:22,982::task::568::TaskManager.Task::(_updateState) Task=`cef46e20-df53-4881-9cb6-0fd66e258cb1`::moving from state preparing -> state finished >Thread-25698::DEBUG::2012-12-02 14:03:22,982::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25698::DEBUG::2012-12-02 14:03:22,982::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25698::DEBUG::2012-12-02 14:03:22,982::task::957::TaskManager.Task::(_decref) Task=`cef46e20-df53-4881-9cb6-0fd66e258cb1`::ref 0 aborting False >Thread-25700::DEBUG::2012-12-02 14:03:24,472::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25700::DEBUG::2012-12-02 14:03:24,473::task::568::TaskManager.Task::(_updateState) Task=`37304b19-cf59-4eec-9cdf-b38a94dc63cc`::moving from state init -> state preparing >Thread-25700::INFO::2012-12-02 14:03:24,473::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25700::INFO::2012-12-02 14:03:24,473::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25700::DEBUG::2012-12-02 14:03:24,473::task::1151::TaskManager.Task::(prepare) Task=`37304b19-cf59-4eec-9cdf-b38a94dc63cc`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25700::DEBUG::2012-12-02 14:03:24,473::task::568::TaskManager.Task::(_updateState) Task=`37304b19-cf59-4eec-9cdf-b38a94dc63cc`::moving from state preparing -> state finished >Thread-25700::DEBUG::2012-12-02 14:03:24,474::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25700::DEBUG::2012-12-02 14:03:24,474::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25700::DEBUG::2012-12-02 14:03:24,474::task::957::TaskManager.Task::(_decref) Task=`37304b19-cf59-4eec-9cdf-b38a94dc63cc`::ref 0 aborting False >Thread-25701::DEBUG::2012-12-02 14:03:24,476::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25701::DEBUG::2012-12-02 14:03:24,477::task::568::TaskManager.Task::(_updateState) Task=`304355d7-c56d-4a14-ae37-1fda1926e19a`::moving from state init -> state preparing >Thread-25701::INFO::2012-12-02 14:03:24,477::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25701::DEBUG::2012-12-02 14:03:24,477::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0b97a50f-c8b8-4f91-b2f9-0a718a3b137b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25701::DEBUG::2012-12-02 14:03:24,477::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25701::DEBUG::2012-12-02 14:03:24,477::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25701::DEBUG::2012-12-02 14:03:24,478::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0b97a50f-c8b8-4f91-b2f9-0a718a3b137b`::Granted request >Thread-25701::DEBUG::2012-12-02 14:03:24,478::task::794::TaskManager.Task::(resourceAcquired) Task=`304355d7-c56d-4a14-ae37-1fda1926e19a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25701::DEBUG::2012-12-02 14:03:24,478::task::957::TaskManager.Task::(_decref) Task=`304355d7-c56d-4a14-ae37-1fda1926e19a`::ref 1 aborting False >Thread-25701::INFO::2012-12-02 14:03:24,484::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25701::DEBUG::2012-12-02 14:03:24,485::task::1151::TaskManager.Task::(prepare) Task=`304355d7-c56d-4a14-ae37-1fda1926e19a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25701::DEBUG::2012-12-02 14:03:24,485::task::568::TaskManager.Task::(_updateState) Task=`304355d7-c56d-4a14-ae37-1fda1926e19a`::moving from state preparing -> state finished >Thread-25701::DEBUG::2012-12-02 14:03:24,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25701::DEBUG::2012-12-02 14:03:24,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25701::DEBUG::2012-12-02 14:03:24,485::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25701::DEBUG::2012-12-02 14:03:24,485::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25701::DEBUG::2012-12-02 14:03:24,485::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25701::DEBUG::2012-12-02 14:03:24,485::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25701::DEBUG::2012-12-02 14:03:24,486::task::957::TaskManager.Task::(_decref) Task=`304355d7-c56d-4a14-ae37-1fda1926e19a`::ref 0 aborting False >Thread-25706::DEBUG::2012-12-02 14:03:33,053::task::568::TaskManager.Task::(_updateState) Task=`33862185-b6eb-4555-a926-f6858f49ce07`::moving from state init -> state preparing >Thread-25706::INFO::2012-12-02 14:03:33,053::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25706::INFO::2012-12-02 14:03:33,053::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00333905220032', 'lastCheck': 1354449803.8835831, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00360989570618', 'lastCheck': 1354449808.5287271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00336909294128', 'lastCheck': 1354449812.2418511, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00783801078796', 'lastCheck': 1354449808.53705, 'code': 0, 'valid': True}} >Thread-25706::DEBUG::2012-12-02 14:03:33,053::task::1151::TaskManager.Task::(prepare) Task=`33862185-b6eb-4555-a926-f6858f49ce07`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00333905220032', 'lastCheck': 1354449803.8835831, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00360989570618', 'lastCheck': 1354449808.5287271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00336909294128', 'lastCheck': 1354449812.2418511, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00783801078796', 'lastCheck': 1354449808.53705, 'code': 0, 'valid': True}} >Thread-25706::DEBUG::2012-12-02 14:03:33,053::task::568::TaskManager.Task::(_updateState) Task=`33862185-b6eb-4555-a926-f6858f49ce07`::moving from state preparing -> state finished >Thread-25706::DEBUG::2012-12-02 14:03:33,054::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25706::DEBUG::2012-12-02 14:03:33,054::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25706::DEBUG::2012-12-02 14:03:33,054::task::957::TaskManager.Task::(_decref) Task=`33862185-b6eb-4555-a926-f6858f49ce07`::ref 0 aborting False >Thread-25708::DEBUG::2012-12-02 14:03:34,626::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25708::DEBUG::2012-12-02 14:03:34,627::task::568::TaskManager.Task::(_updateState) Task=`223efa9e-c348-4a6e-966f-ee4b8d13e327`::moving from state init -> state preparing >Thread-25708::INFO::2012-12-02 14:03:34,627::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25708::INFO::2012-12-02 14:03:34,627::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25708::DEBUG::2012-12-02 14:03:34,627::task::1151::TaskManager.Task::(prepare) Task=`223efa9e-c348-4a6e-966f-ee4b8d13e327`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25708::DEBUG::2012-12-02 14:03:34,627::task::568::TaskManager.Task::(_updateState) Task=`223efa9e-c348-4a6e-966f-ee4b8d13e327`::moving from state preparing -> state finished >Thread-25708::DEBUG::2012-12-02 14:03:34,627::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25708::DEBUG::2012-12-02 14:03:34,627::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25708::DEBUG::2012-12-02 14:03:34,628::task::957::TaskManager.Task::(_decref) Task=`223efa9e-c348-4a6e-966f-ee4b8d13e327`::ref 0 aborting False >Thread-25709::DEBUG::2012-12-02 14:03:34,630::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25709::DEBUG::2012-12-02 14:03:34,630::task::568::TaskManager.Task::(_updateState) Task=`29d9bc3a-0aa2-417e-86b1-633d26d86ced`::moving from state init -> state preparing >Thread-25709::INFO::2012-12-02 14:03:34,630::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25709::DEBUG::2012-12-02 14:03:34,631::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7aa51cdf-a365-40c6-b1a1-05bc2305c571`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25709::DEBUG::2012-12-02 14:03:34,631::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25709::DEBUG::2012-12-02 14:03:34,631::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25709::DEBUG::2012-12-02 14:03:34,631::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7aa51cdf-a365-40c6-b1a1-05bc2305c571`::Granted request >Thread-25709::DEBUG::2012-12-02 14:03:34,631::task::794::TaskManager.Task::(resourceAcquired) Task=`29d9bc3a-0aa2-417e-86b1-633d26d86ced`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25709::DEBUG::2012-12-02 14:03:34,632::task::957::TaskManager.Task::(_decref) Task=`29d9bc3a-0aa2-417e-86b1-633d26d86ced`::ref 1 aborting False >Thread-25709::INFO::2012-12-02 14:03:34,637::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25709::DEBUG::2012-12-02 14:03:34,637::task::1151::TaskManager.Task::(prepare) Task=`29d9bc3a-0aa2-417e-86b1-633d26d86ced`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25709::DEBUG::2012-12-02 14:03:34,637::task::568::TaskManager.Task::(_updateState) Task=`29d9bc3a-0aa2-417e-86b1-633d26d86ced`::moving from state preparing -> state finished >Thread-25709::DEBUG::2012-12-02 14:03:34,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25709::DEBUG::2012-12-02 14:03:34,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25709::DEBUG::2012-12-02 14:03:34,638::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25709::DEBUG::2012-12-02 14:03:34,638::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25709::DEBUG::2012-12-02 14:03:34,638::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25709::DEBUG::2012-12-02 14:03:34,638::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25709::DEBUG::2012-12-02 14:03:34,638::task::957::TaskManager.Task::(_decref) Task=`29d9bc3a-0aa2-417e-86b1-633d26d86ced`::ref 0 aborting False >Thread-25714::DEBUG::2012-12-02 14:03:43,163::task::568::TaskManager.Task::(_updateState) Task=`d1d20d7e-21e5-4c53-954f-5fa2db3b2810`::moving from state init -> state preparing >Thread-25714::INFO::2012-12-02 14:03:43,163::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25714::INFO::2012-12-02 14:03:43,164::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0085129737854', 'lastCheck': 1354449813.901988, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00396203994751', 'lastCheck': 1354449818.5475559, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213003158569', 'lastCheck': 1354449822.253011, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00790095329285', 'lastCheck': 1354449818.5583069, 'code': 0, 'valid': True}} >Thread-25714::DEBUG::2012-12-02 14:03:43,164::task::1151::TaskManager.Task::(prepare) Task=`d1d20d7e-21e5-4c53-954f-5fa2db3b2810`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0085129737854', 'lastCheck': 1354449813.901988, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00396203994751', 'lastCheck': 1354449818.5475559, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213003158569', 'lastCheck': 1354449822.253011, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00790095329285', 'lastCheck': 1354449818.5583069, 'code': 0, 'valid': True}} >Thread-25714::DEBUG::2012-12-02 14:03:43,164::task::568::TaskManager.Task::(_updateState) Task=`d1d20d7e-21e5-4c53-954f-5fa2db3b2810`::moving from state preparing -> state finished >Thread-25714::DEBUG::2012-12-02 14:03:43,164::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25714::DEBUG::2012-12-02 14:03:43,164::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25714::DEBUG::2012-12-02 14:03:43,164::task::957::TaskManager.Task::(_decref) Task=`d1d20d7e-21e5-4c53-954f-5fa2db3b2810`::ref 0 aborting False >Thread-25716::DEBUG::2012-12-02 14:03:44,772::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25716::DEBUG::2012-12-02 14:03:44,772::task::568::TaskManager.Task::(_updateState) Task=`a5fe8426-c4b6-48d0-adbb-a8f2a4e9af93`::moving from state init -> state preparing >Thread-25716::INFO::2012-12-02 14:03:44,772::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25716::INFO::2012-12-02 14:03:44,773::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25716::DEBUG::2012-12-02 14:03:44,773::task::1151::TaskManager.Task::(prepare) Task=`a5fe8426-c4b6-48d0-adbb-a8f2a4e9af93`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25716::DEBUG::2012-12-02 14:03:44,773::task::568::TaskManager.Task::(_updateState) Task=`a5fe8426-c4b6-48d0-adbb-a8f2a4e9af93`::moving from state preparing -> state finished >Thread-25716::DEBUG::2012-12-02 14:03:44,773::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25716::DEBUG::2012-12-02 14:03:44,773::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25716::DEBUG::2012-12-02 14:03:44,773::task::957::TaskManager.Task::(_decref) Task=`a5fe8426-c4b6-48d0-adbb-a8f2a4e9af93`::ref 0 aborting False >Thread-25717::DEBUG::2012-12-02 14:03:44,776::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25717::DEBUG::2012-12-02 14:03:44,776::task::568::TaskManager.Task::(_updateState) Task=`8a9d2c4b-0836-49de-af02-2cba2726a6dd`::moving from state init -> state preparing >Thread-25717::INFO::2012-12-02 14:03:44,776::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25717::DEBUG::2012-12-02 14:03:44,776::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3c26bf62-1e75-4dce-9947-1e7ab8bae5ac`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25717::DEBUG::2012-12-02 14:03:44,776::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25717::DEBUG::2012-12-02 14:03:44,777::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25717::DEBUG::2012-12-02 14:03:44,777::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3c26bf62-1e75-4dce-9947-1e7ab8bae5ac`::Granted request >Thread-25717::DEBUG::2012-12-02 14:03:44,778::task::794::TaskManager.Task::(resourceAcquired) Task=`8a9d2c4b-0836-49de-af02-2cba2726a6dd`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25717::DEBUG::2012-12-02 14:03:44,778::task::957::TaskManager.Task::(_decref) Task=`8a9d2c4b-0836-49de-af02-2cba2726a6dd`::ref 1 aborting False >Thread-25717::INFO::2012-12-02 14:03:44,784::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25717::DEBUG::2012-12-02 14:03:44,784::task::1151::TaskManager.Task::(prepare) Task=`8a9d2c4b-0836-49de-af02-2cba2726a6dd`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25717::DEBUG::2012-12-02 14:03:44,785::task::568::TaskManager.Task::(_updateState) Task=`8a9d2c4b-0836-49de-af02-2cba2726a6dd`::moving from state preparing -> state finished >Thread-25717::DEBUG::2012-12-02 14:03:44,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25717::DEBUG::2012-12-02 14:03:44,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25717::DEBUG::2012-12-02 14:03:44,785::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25717::DEBUG::2012-12-02 14:03:44,785::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25717::DEBUG::2012-12-02 14:03:44,785::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25717::DEBUG::2012-12-02 14:03:44,785::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25717::DEBUG::2012-12-02 14:03:44,785::task::957::TaskManager.Task::(_decref) Task=`8a9d2c4b-0836-49de-af02-2cba2726a6dd`::ref 0 aborting False >Thread-25722::DEBUG::2012-12-02 14:03:53,226::task::568::TaskManager.Task::(_updateState) Task=`eda1ff98-8ab8-4dac-a26a-bfc45238a328`::moving from state init -> state preparing >Thread-25722::INFO::2012-12-02 14:03:53,227::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25722::INFO::2012-12-02 14:03:53,227::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00805020332336', 'lastCheck': 1354449823.9184151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00217914581299', 'lastCheck': 1354449828.565809, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00168991088867', 'lastCheck': 1354449832.263221, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00716710090637', 'lastCheck': 1354449828.5756891, 'code': 0, 'valid': True}} >Thread-25722::DEBUG::2012-12-02 14:03:53,227::task::1151::TaskManager.Task::(prepare) Task=`eda1ff98-8ab8-4dac-a26a-bfc45238a328`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00805020332336', 'lastCheck': 1354449823.9184151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00217914581299', 'lastCheck': 1354449828.565809, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00168991088867', 'lastCheck': 1354449832.263221, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00716710090637', 'lastCheck': 1354449828.5756891, 'code': 0, 'valid': True}} >Thread-25722::DEBUG::2012-12-02 14:03:53,227::task::568::TaskManager.Task::(_updateState) Task=`eda1ff98-8ab8-4dac-a26a-bfc45238a328`::moving from state preparing -> state finished >Thread-25722::DEBUG::2012-12-02 14:03:53,227::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25722::DEBUG::2012-12-02 14:03:53,227::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25722::DEBUG::2012-12-02 14:03:53,227::task::957::TaskManager.Task::(_decref) Task=`eda1ff98-8ab8-4dac-a26a-bfc45238a328`::ref 0 aborting False >Thread-25724::DEBUG::2012-12-02 14:03:54,908::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25724::DEBUG::2012-12-02 14:03:54,909::task::568::TaskManager.Task::(_updateState) Task=`56265a81-9009-4bfa-9b00-98972f778ee8`::moving from state init -> state preparing >Thread-25724::INFO::2012-12-02 14:03:54,909::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25724::INFO::2012-12-02 14:03:54,909::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25724::DEBUG::2012-12-02 14:03:54,909::task::1151::TaskManager.Task::(prepare) Task=`56265a81-9009-4bfa-9b00-98972f778ee8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25724::DEBUG::2012-12-02 14:03:54,909::task::568::TaskManager.Task::(_updateState) Task=`56265a81-9009-4bfa-9b00-98972f778ee8`::moving from state preparing -> state finished >Thread-25724::DEBUG::2012-12-02 14:03:54,910::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25724::DEBUG::2012-12-02 14:03:54,910::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25724::DEBUG::2012-12-02 14:03:54,910::task::957::TaskManager.Task::(_decref) Task=`56265a81-9009-4bfa-9b00-98972f778ee8`::ref 0 aborting False >Thread-25725::DEBUG::2012-12-02 14:03:54,912::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25725::DEBUG::2012-12-02 14:03:54,913::task::568::TaskManager.Task::(_updateState) Task=`fbb05b1a-2ed7-4c7a-83a1-eafc9fb21956`::moving from state init -> state preparing >Thread-25725::INFO::2012-12-02 14:03:54,913::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25725::DEBUG::2012-12-02 14:03:54,913::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f8a474f2-9df9-492a-99fb-d5824ec69659`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25725::DEBUG::2012-12-02 14:03:54,913::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25725::DEBUG::2012-12-02 14:03:54,913::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25725::DEBUG::2012-12-02 14:03:54,914::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f8a474f2-9df9-492a-99fb-d5824ec69659`::Granted request >Thread-25725::DEBUG::2012-12-02 14:03:54,914::task::794::TaskManager.Task::(resourceAcquired) Task=`fbb05b1a-2ed7-4c7a-83a1-eafc9fb21956`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25725::DEBUG::2012-12-02 14:03:54,914::task::957::TaskManager.Task::(_decref) Task=`fbb05b1a-2ed7-4c7a-83a1-eafc9fb21956`::ref 1 aborting False >Thread-25725::INFO::2012-12-02 14:03:54,919::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25725::DEBUG::2012-12-02 14:03:54,919::task::1151::TaskManager.Task::(prepare) Task=`fbb05b1a-2ed7-4c7a-83a1-eafc9fb21956`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25725::DEBUG::2012-12-02 14:03:54,919::task::568::TaskManager.Task::(_updateState) Task=`fbb05b1a-2ed7-4c7a-83a1-eafc9fb21956`::moving from state preparing -> state finished >Thread-25725::DEBUG::2012-12-02 14:03:54,919::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25725::DEBUG::2012-12-02 14:03:54,919::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25725::DEBUG::2012-12-02 14:03:54,919::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25725::DEBUG::2012-12-02 14:03:54,919::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25725::DEBUG::2012-12-02 14:03:54,919::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25725::DEBUG::2012-12-02 14:03:54,919::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25725::DEBUG::2012-12-02 14:03:54,920::task::957::TaskManager.Task::(_decref) Task=`fbb05b1a-2ed7-4c7a-83a1-eafc9fb21956`::ref 0 aborting False >Thread-25730::DEBUG::2012-12-02 14:04:03,287::task::568::TaskManager.Task::(_updateState) Task=`caa37334-54be-4031-a1b5-5c52d1e36aea`::moving from state init -> state preparing >Thread-25730::INFO::2012-12-02 14:04:03,287::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25730::INFO::2012-12-02 14:04:03,287::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00817489624023', 'lastCheck': 1354449833.9352269, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00222301483154', 'lastCheck': 1354449838.5833631, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00187802314758', 'lastCheck': 1354449842.2730119, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812411308289', 'lastCheck': 1354449838.5938921, 'code': 0, 'valid': True}} >Thread-25730::DEBUG::2012-12-02 14:04:03,287::task::1151::TaskManager.Task::(prepare) Task=`caa37334-54be-4031-a1b5-5c52d1e36aea`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00817489624023', 'lastCheck': 1354449833.9352269, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00222301483154', 'lastCheck': 1354449838.5833631, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00187802314758', 'lastCheck': 1354449842.2730119, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812411308289', 'lastCheck': 1354449838.5938921, 'code': 0, 'valid': True}} >Thread-25730::DEBUG::2012-12-02 14:04:03,288::task::568::TaskManager.Task::(_updateState) Task=`caa37334-54be-4031-a1b5-5c52d1e36aea`::moving from state preparing -> state finished >Thread-25730::DEBUG::2012-12-02 14:04:03,288::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25730::DEBUG::2012-12-02 14:04:03,288::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25730::DEBUG::2012-12-02 14:04:03,288::task::957::TaskManager.Task::(_decref) Task=`caa37334-54be-4031-a1b5-5c52d1e36aea`::ref 0 aborting False >Thread-25732::DEBUG::2012-12-02 14:04:05,120::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25732::DEBUG::2012-12-02 14:04:05,121::task::568::TaskManager.Task::(_updateState) Task=`33d4ae73-8402-4dfd-b4a0-bd5d01ea4a4f`::moving from state init -> state preparing >Thread-25732::INFO::2012-12-02 14:04:05,121::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25732::INFO::2012-12-02 14:04:05,121::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25732::DEBUG::2012-12-02 14:04:05,121::task::1151::TaskManager.Task::(prepare) Task=`33d4ae73-8402-4dfd-b4a0-bd5d01ea4a4f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25732::DEBUG::2012-12-02 14:04:05,121::task::568::TaskManager.Task::(_updateState) Task=`33d4ae73-8402-4dfd-b4a0-bd5d01ea4a4f`::moving from state preparing -> state finished >Thread-25732::DEBUG::2012-12-02 14:04:05,122::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25732::DEBUG::2012-12-02 14:04:05,122::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25732::DEBUG::2012-12-02 14:04:05,122::task::957::TaskManager.Task::(_decref) Task=`33d4ae73-8402-4dfd-b4a0-bd5d01ea4a4f`::ref 0 aborting False >Thread-25733::DEBUG::2012-12-02 14:04:05,124::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25733::DEBUG::2012-12-02 14:04:05,125::task::568::TaskManager.Task::(_updateState) Task=`5e703a48-857c-4a51-8f87-eca46a7c2973`::moving from state init -> state preparing >Thread-25733::INFO::2012-12-02 14:04:05,125::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25733::DEBUG::2012-12-02 14:04:05,125::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`67760fc2-daa5-4656-a08b-678b455d56c4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25733::DEBUG::2012-12-02 14:04:05,125::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25733::DEBUG::2012-12-02 14:04:05,125::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25733::DEBUG::2012-12-02 14:04:05,125::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`67760fc2-daa5-4656-a08b-678b455d56c4`::Granted request >Thread-25733::DEBUG::2012-12-02 14:04:05,126::task::794::TaskManager.Task::(resourceAcquired) Task=`5e703a48-857c-4a51-8f87-eca46a7c2973`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25733::DEBUG::2012-12-02 14:04:05,126::task::957::TaskManager.Task::(_decref) Task=`5e703a48-857c-4a51-8f87-eca46a7c2973`::ref 1 aborting False >Thread-25733::INFO::2012-12-02 14:04:05,130::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25733::DEBUG::2012-12-02 14:04:05,130::task::1151::TaskManager.Task::(prepare) Task=`5e703a48-857c-4a51-8f87-eca46a7c2973`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25733::DEBUG::2012-12-02 14:04:05,130::task::568::TaskManager.Task::(_updateState) Task=`5e703a48-857c-4a51-8f87-eca46a7c2973`::moving from state preparing -> state finished >Thread-25733::DEBUG::2012-12-02 14:04:05,130::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25733::DEBUG::2012-12-02 14:04:05,130::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25733::DEBUG::2012-12-02 14:04:05,131::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25733::DEBUG::2012-12-02 14:04:05,131::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25733::DEBUG::2012-12-02 14:04:05,131::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25733::DEBUG::2012-12-02 14:04:05,131::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25733::DEBUG::2012-12-02 14:04:05,131::task::957::TaskManager.Task::(_decref) Task=`5e703a48-857c-4a51-8f87-eca46a7c2973`::ref 0 aborting False >Thread-25738::DEBUG::2012-12-02 14:04:13,407::task::568::TaskManager.Task::(_updateState) Task=`35aa8d04-e903-4308-84ea-e5ceb32515ba`::moving from state init -> state preparing >Thread-25738::INFO::2012-12-02 14:04:13,407::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25738::INFO::2012-12-02 14:04:13,407::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00747394561768', 'lastCheck': 1354449843.9519639, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0026228427887', 'lastCheck': 1354449848.6003389, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00326299667358', 'lastCheck': 1354449852.2862301, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00705695152283', 'lastCheck': 1354449848.6096561, 'code': 0, 'valid': True}} >Thread-25738::DEBUG::2012-12-02 14:04:13,408::task::1151::TaskManager.Task::(prepare) Task=`35aa8d04-e903-4308-84ea-e5ceb32515ba`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00747394561768', 'lastCheck': 1354449843.9519639, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0026228427887', 'lastCheck': 1354449848.6003389, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00326299667358', 'lastCheck': 1354449852.2862301, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00705695152283', 'lastCheck': 1354449848.6096561, 'code': 0, 'valid': True}} >Thread-25738::DEBUG::2012-12-02 14:04:13,408::task::568::TaskManager.Task::(_updateState) Task=`35aa8d04-e903-4308-84ea-e5ceb32515ba`::moving from state preparing -> state finished >Thread-25738::DEBUG::2012-12-02 14:04:13,408::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25738::DEBUG::2012-12-02 14:04:13,408::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25738::DEBUG::2012-12-02 14:04:13,408::task::957::TaskManager.Task::(_decref) Task=`35aa8d04-e903-4308-84ea-e5ceb32515ba`::ref 0 aborting False >Thread-25740::DEBUG::2012-12-02 14:04:15,291::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25740::DEBUG::2012-12-02 14:04:15,291::task::568::TaskManager.Task::(_updateState) Task=`3e468ba7-df5d-4968-8435-157305cc78bc`::moving from state init -> state preparing >Thread-25740::INFO::2012-12-02 14:04:15,292::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25740::INFO::2012-12-02 14:04:15,292::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25740::DEBUG::2012-12-02 14:04:15,292::task::1151::TaskManager.Task::(prepare) Task=`3e468ba7-df5d-4968-8435-157305cc78bc`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25740::DEBUG::2012-12-02 14:04:15,292::task::568::TaskManager.Task::(_updateState) Task=`3e468ba7-df5d-4968-8435-157305cc78bc`::moving from state preparing -> state finished >Thread-25740::DEBUG::2012-12-02 14:04:15,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25740::DEBUG::2012-12-02 14:04:15,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25740::DEBUG::2012-12-02 14:04:15,292::task::957::TaskManager.Task::(_decref) Task=`3e468ba7-df5d-4968-8435-157305cc78bc`::ref 0 aborting False >Thread-25741::DEBUG::2012-12-02 14:04:15,295::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25741::DEBUG::2012-12-02 14:04:15,295::task::568::TaskManager.Task::(_updateState) Task=`837b11da-030d-4e80-9f4f-dac1edaa0fe1`::moving from state init -> state preparing >Thread-25741::INFO::2012-12-02 14:04:15,295::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25741::DEBUG::2012-12-02 14:04:15,296::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`004449ab-6f11-4d09-b55e-70facb05e29a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25741::DEBUG::2012-12-02 14:04:15,296::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25741::DEBUG::2012-12-02 14:04:15,296::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25741::DEBUG::2012-12-02 14:04:15,296::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`004449ab-6f11-4d09-b55e-70facb05e29a`::Granted request >Thread-25741::DEBUG::2012-12-02 14:04:15,296::task::794::TaskManager.Task::(resourceAcquired) Task=`837b11da-030d-4e80-9f4f-dac1edaa0fe1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25741::DEBUG::2012-12-02 14:04:15,296::task::957::TaskManager.Task::(_decref) Task=`837b11da-030d-4e80-9f4f-dac1edaa0fe1`::ref 1 aborting False >Thread-25741::INFO::2012-12-02 14:04:15,301::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25741::DEBUG::2012-12-02 14:04:15,301::task::1151::TaskManager.Task::(prepare) Task=`837b11da-030d-4e80-9f4f-dac1edaa0fe1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25741::DEBUG::2012-12-02 14:04:15,301::task::568::TaskManager.Task::(_updateState) Task=`837b11da-030d-4e80-9f4f-dac1edaa0fe1`::moving from state preparing -> state finished >Thread-25741::DEBUG::2012-12-02 14:04:15,301::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25741::DEBUG::2012-12-02 14:04:15,301::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25741::DEBUG::2012-12-02 14:04:15,301::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25741::DEBUG::2012-12-02 14:04:15,301::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25741::DEBUG::2012-12-02 14:04:15,301::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25741::DEBUG::2012-12-02 14:04:15,301::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25741::DEBUG::2012-12-02 14:04:15,302::task::957::TaskManager.Task::(_decref) Task=`837b11da-030d-4e80-9f4f-dac1edaa0fe1`::ref 0 aborting False >Thread-25746::DEBUG::2012-12-02 14:04:23,497::task::568::TaskManager.Task::(_updateState) Task=`bb9b20ff-28c1-4c8c-96d9-598189670731`::moving from state init -> state preparing >Thread-25746::INFO::2012-12-02 14:04:23,498::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25746::INFO::2012-12-02 14:04:23,498::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00787091255188', 'lastCheck': 1354449853.968163, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00128197669983', 'lastCheck': 1354449858.672811, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00193309783936', 'lastCheck': 1354449862.298018, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00344896316528', 'lastCheck': 1354449858.6723149, 'code': 0, 'valid': True}} >Thread-25746::DEBUG::2012-12-02 14:04:23,498::task::1151::TaskManager.Task::(prepare) Task=`bb9b20ff-28c1-4c8c-96d9-598189670731`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00787091255188', 'lastCheck': 1354449853.968163, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00128197669983', 'lastCheck': 1354449858.672811, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00193309783936', 'lastCheck': 1354449862.298018, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00344896316528', 'lastCheck': 1354449858.6723149, 'code': 0, 'valid': True}} >Thread-25746::DEBUG::2012-12-02 14:04:23,498::task::568::TaskManager.Task::(_updateState) Task=`bb9b20ff-28c1-4c8c-96d9-598189670731`::moving from state preparing -> state finished >Thread-25746::DEBUG::2012-12-02 14:04:23,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25746::DEBUG::2012-12-02 14:04:23,499::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25746::DEBUG::2012-12-02 14:04:23,499::task::957::TaskManager.Task::(_decref) Task=`bb9b20ff-28c1-4c8c-96d9-598189670731`::ref 0 aborting False >Thread-25748::DEBUG::2012-12-02 14:04:25,437::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25748::DEBUG::2012-12-02 14:04:25,437::task::568::TaskManager.Task::(_updateState) Task=`5804b7b3-28e8-4f70-9c9c-3ce5194b1c34`::moving from state init -> state preparing >Thread-25748::INFO::2012-12-02 14:04:25,438::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25748::INFO::2012-12-02 14:04:25,438::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25748::DEBUG::2012-12-02 14:04:25,438::task::1151::TaskManager.Task::(prepare) Task=`5804b7b3-28e8-4f70-9c9c-3ce5194b1c34`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25748::DEBUG::2012-12-02 14:04:25,438::task::568::TaskManager.Task::(_updateState) Task=`5804b7b3-28e8-4f70-9c9c-3ce5194b1c34`::moving from state preparing -> state finished >Thread-25748::DEBUG::2012-12-02 14:04:25,438::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25748::DEBUG::2012-12-02 14:04:25,438::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25748::DEBUG::2012-12-02 14:04:25,438::task::957::TaskManager.Task::(_decref) Task=`5804b7b3-28e8-4f70-9c9c-3ce5194b1c34`::ref 0 aborting False >Thread-25749::DEBUG::2012-12-02 14:04:25,441::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25749::DEBUG::2012-12-02 14:04:25,441::task::568::TaskManager.Task::(_updateState) Task=`4a0cebb1-9542-4486-baf0-c3c4fe0cc7ca`::moving from state init -> state preparing >Thread-25749::INFO::2012-12-02 14:04:25,441::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25749::DEBUG::2012-12-02 14:04:25,442::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3f3df94c-f12c-4f70-a8af-101ecb489df1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25749::DEBUG::2012-12-02 14:04:25,442::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25749::DEBUG::2012-12-02 14:04:25,442::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25749::DEBUG::2012-12-02 14:04:25,442::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3f3df94c-f12c-4f70-a8af-101ecb489df1`::Granted request >Thread-25749::DEBUG::2012-12-02 14:04:25,442::task::794::TaskManager.Task::(resourceAcquired) Task=`4a0cebb1-9542-4486-baf0-c3c4fe0cc7ca`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25749::DEBUG::2012-12-02 14:04:25,442::task::957::TaskManager.Task::(_decref) Task=`4a0cebb1-9542-4486-baf0-c3c4fe0cc7ca`::ref 1 aborting False >Thread-25749::INFO::2012-12-02 14:04:25,448::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25749::DEBUG::2012-12-02 14:04:25,448::task::1151::TaskManager.Task::(prepare) Task=`4a0cebb1-9542-4486-baf0-c3c4fe0cc7ca`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25749::DEBUG::2012-12-02 14:04:25,448::task::568::TaskManager.Task::(_updateState) Task=`4a0cebb1-9542-4486-baf0-c3c4fe0cc7ca`::moving from state preparing -> state finished >Thread-25749::DEBUG::2012-12-02 14:04:25,448::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25749::DEBUG::2012-12-02 14:04:25,448::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25749::DEBUG::2012-12-02 14:04:25,448::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25749::DEBUG::2012-12-02 14:04:25,449::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25749::DEBUG::2012-12-02 14:04:25,449::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25749::DEBUG::2012-12-02 14:04:25,449::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25749::DEBUG::2012-12-02 14:04:25,449::task::957::TaskManager.Task::(_decref) Task=`4a0cebb1-9542-4486-baf0-c3c4fe0cc7ca`::ref 0 aborting False >Thread-25754::DEBUG::2012-12-02 14:04:33,558::task::568::TaskManager.Task::(_updateState) Task=`8fc64e31-c5d0-4174-99cf-f019e38b23f6`::moving from state init -> state preparing >Thread-25754::INFO::2012-12-02 14:04:33,558::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25754::INFO::2012-12-02 14:04:33,558::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00822496414185', 'lastCheck': 1354449863.985698, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00400876998901', 'lastCheck': 1354449868.693882, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00346422195435', 'lastCheck': 1354449872.3125429, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00892496109009', 'lastCheck': 1354449868.6973579, 'code': 0, 'valid': True}} >Thread-25754::DEBUG::2012-12-02 14:04:33,558::task::1151::TaskManager.Task::(prepare) Task=`8fc64e31-c5d0-4174-99cf-f019e38b23f6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00822496414185', 'lastCheck': 1354449863.985698, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00400876998901', 'lastCheck': 1354449868.693882, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00346422195435', 'lastCheck': 1354449872.3125429, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00892496109009', 'lastCheck': 1354449868.6973579, 'code': 0, 'valid': True}} >Thread-25754::DEBUG::2012-12-02 14:04:33,558::task::568::TaskManager.Task::(_updateState) Task=`8fc64e31-c5d0-4174-99cf-f019e38b23f6`::moving from state preparing -> state finished >Thread-25754::DEBUG::2012-12-02 14:04:33,559::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25754::DEBUG::2012-12-02 14:04:33,559::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25754::DEBUG::2012-12-02 14:04:33,559::task::957::TaskManager.Task::(_decref) Task=`8fc64e31-c5d0-4174-99cf-f019e38b23f6`::ref 0 aborting False >Thread-25756::DEBUG::2012-12-02 14:04:35,582::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25756::DEBUG::2012-12-02 14:04:35,582::task::568::TaskManager.Task::(_updateState) Task=`b51ac81e-4e3c-4905-8b76-06be8595ef56`::moving from state init -> state preparing >Thread-25756::INFO::2012-12-02 14:04:35,582::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25756::INFO::2012-12-02 14:04:35,582::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25756::DEBUG::2012-12-02 14:04:35,583::task::1151::TaskManager.Task::(prepare) Task=`b51ac81e-4e3c-4905-8b76-06be8595ef56`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25756::DEBUG::2012-12-02 14:04:35,583::task::568::TaskManager.Task::(_updateState) Task=`b51ac81e-4e3c-4905-8b76-06be8595ef56`::moving from state preparing -> state finished >Thread-25756::DEBUG::2012-12-02 14:04:35,583::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25756::DEBUG::2012-12-02 14:04:35,583::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25756::DEBUG::2012-12-02 14:04:35,583::task::957::TaskManager.Task::(_decref) Task=`b51ac81e-4e3c-4905-8b76-06be8595ef56`::ref 0 aborting False >Thread-25757::DEBUG::2012-12-02 14:04:35,586::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25757::DEBUG::2012-12-02 14:04:35,586::task::568::TaskManager.Task::(_updateState) Task=`c394147b-8256-4df8-922e-3eef95d215bc`::moving from state init -> state preparing >Thread-25757::INFO::2012-12-02 14:04:35,586::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25757::DEBUG::2012-12-02 14:04:35,586::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e1d136a1-d16a-47c5-abed-1530db5e8763`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25757::DEBUG::2012-12-02 14:04:35,587::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25757::DEBUG::2012-12-02 14:04:35,587::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25757::DEBUG::2012-12-02 14:04:35,587::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e1d136a1-d16a-47c5-abed-1530db5e8763`::Granted request >Thread-25757::DEBUG::2012-12-02 14:04:35,587::task::794::TaskManager.Task::(resourceAcquired) Task=`c394147b-8256-4df8-922e-3eef95d215bc`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25757::DEBUG::2012-12-02 14:04:35,587::task::957::TaskManager.Task::(_decref) Task=`c394147b-8256-4df8-922e-3eef95d215bc`::ref 1 aborting False >Thread-25757::INFO::2012-12-02 14:04:35,594::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25757::DEBUG::2012-12-02 14:04:35,594::task::1151::TaskManager.Task::(prepare) Task=`c394147b-8256-4df8-922e-3eef95d215bc`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25757::DEBUG::2012-12-02 14:04:35,594::task::568::TaskManager.Task::(_updateState) Task=`c394147b-8256-4df8-922e-3eef95d215bc`::moving from state preparing -> state finished >Thread-25757::DEBUG::2012-12-02 14:04:35,594::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25757::DEBUG::2012-12-02 14:04:35,594::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25757::DEBUG::2012-12-02 14:04:35,594::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25757::DEBUG::2012-12-02 14:04:35,595::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25757::DEBUG::2012-12-02 14:04:35,595::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25757::DEBUG::2012-12-02 14:04:35,595::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25757::DEBUG::2012-12-02 14:04:35,595::task::957::TaskManager.Task::(_decref) Task=`c394147b-8256-4df8-922e-3eef95d215bc`::ref 0 aborting False >Thread-25762::DEBUG::2012-12-02 14:04:43,724::task::568::TaskManager.Task::(_updateState) Task=`34bdfc87-b88a-470b-ad0e-a092225c06fb`::moving from state init -> state preparing >Thread-25762::INFO::2012-12-02 14:04:43,724::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25762::INFO::2012-12-02 14:04:43,724::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00398111343384', 'lastCheck': 1354449873.9993529, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00253891944885', 'lastCheck': 1354449878.711937, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0021550655365', 'lastCheck': 1354449882.3226459, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00666403770447', 'lastCheck': 1354449878.7177939, 'code': 0, 'valid': True}} >Thread-25762::DEBUG::2012-12-02 14:04:43,724::task::1151::TaskManager.Task::(prepare) Task=`34bdfc87-b88a-470b-ad0e-a092225c06fb`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00398111343384', 'lastCheck': 1354449873.9993529, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00253891944885', 'lastCheck': 1354449878.711937, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0021550655365', 'lastCheck': 1354449882.3226459, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00666403770447', 'lastCheck': 1354449878.7177939, 'code': 0, 'valid': True}} >Thread-25762::DEBUG::2012-12-02 14:04:43,724::task::568::TaskManager.Task::(_updateState) Task=`34bdfc87-b88a-470b-ad0e-a092225c06fb`::moving from state preparing -> state finished >Thread-25762::DEBUG::2012-12-02 14:04:43,724::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25762::DEBUG::2012-12-02 14:04:43,725::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25762::DEBUG::2012-12-02 14:04:43,725::task::957::TaskManager.Task::(_decref) Task=`34bdfc87-b88a-470b-ad0e-a092225c06fb`::ref 0 aborting False >Thread-25764::DEBUG::2012-12-02 14:04:45,750::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25764::DEBUG::2012-12-02 14:04:45,751::task::568::TaskManager.Task::(_updateState) Task=`e043d835-82a1-4a85-9279-9f1199a04090`::moving from state init -> state preparing >Thread-25764::INFO::2012-12-02 14:04:45,751::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25764::INFO::2012-12-02 14:04:45,751::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25764::DEBUG::2012-12-02 14:04:45,751::task::1151::TaskManager.Task::(prepare) Task=`e043d835-82a1-4a85-9279-9f1199a04090`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25764::DEBUG::2012-12-02 14:04:45,751::task::568::TaskManager.Task::(_updateState) Task=`e043d835-82a1-4a85-9279-9f1199a04090`::moving from state preparing -> state finished >Thread-25764::DEBUG::2012-12-02 14:04:45,751::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25764::DEBUG::2012-12-02 14:04:45,752::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25764::DEBUG::2012-12-02 14:04:45,752::task::957::TaskManager.Task::(_decref) Task=`e043d835-82a1-4a85-9279-9f1199a04090`::ref 0 aborting False >Thread-25765::DEBUG::2012-12-02 14:04:45,754::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25765::DEBUG::2012-12-02 14:04:45,754::task::568::TaskManager.Task::(_updateState) Task=`4b8d1950-8a8b-426d-a5ca-da6d593a1ae6`::moving from state init -> state preparing >Thread-25765::INFO::2012-12-02 14:04:45,755::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25765::DEBUG::2012-12-02 14:04:45,755::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e800b17d-b66e-43a5-9c5b-8a992d9f5870`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25765::DEBUG::2012-12-02 14:04:45,755::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25765::DEBUG::2012-12-02 14:04:45,755::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25765::DEBUG::2012-12-02 14:04:45,755::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e800b17d-b66e-43a5-9c5b-8a992d9f5870`::Granted request >Thread-25765::DEBUG::2012-12-02 14:04:45,756::task::794::TaskManager.Task::(resourceAcquired) Task=`4b8d1950-8a8b-426d-a5ca-da6d593a1ae6`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25765::DEBUG::2012-12-02 14:04:45,756::task::957::TaskManager.Task::(_decref) Task=`4b8d1950-8a8b-426d-a5ca-da6d593a1ae6`::ref 1 aborting False >Thread-25765::INFO::2012-12-02 14:04:45,762::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25765::DEBUG::2012-12-02 14:04:45,762::task::1151::TaskManager.Task::(prepare) Task=`4b8d1950-8a8b-426d-a5ca-da6d593a1ae6`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25765::DEBUG::2012-12-02 14:04:45,762::task::568::TaskManager.Task::(_updateState) Task=`4b8d1950-8a8b-426d-a5ca-da6d593a1ae6`::moving from state preparing -> state finished >Thread-25765::DEBUG::2012-12-02 14:04:45,763::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25765::DEBUG::2012-12-02 14:04:45,763::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25765::DEBUG::2012-12-02 14:04:45,763::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25765::DEBUG::2012-12-02 14:04:45,763::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25765::DEBUG::2012-12-02 14:04:45,763::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25765::DEBUG::2012-12-02 14:04:45,763::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25765::DEBUG::2012-12-02 14:04:45,763::task::957::TaskManager.Task::(_decref) Task=`4b8d1950-8a8b-426d-a5ca-da6d593a1ae6`::ref 0 aborting False >Thread-25770::DEBUG::2012-12-02 14:04:53,802::task::568::TaskManager.Task::(_updateState) Task=`5b4101f1-a138-4f0d-9796-c2a241710005`::moving from state init -> state preparing >Thread-25770::INFO::2012-12-02 14:04:53,802::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25770::INFO::2012-12-02 14:04:53,802::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.01451587677', 'lastCheck': 1354449884.024698, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00238513946533', 'lastCheck': 1354449888.7277989, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00198006629944', 'lastCheck': 1354449892.3326001, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00613403320312', 'lastCheck': 1354449888.7355559, 'code': 0, 'valid': True}} >Thread-25770::DEBUG::2012-12-02 14:04:53,802::task::1151::TaskManager.Task::(prepare) Task=`5b4101f1-a138-4f0d-9796-c2a241710005`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.01451587677', 'lastCheck': 1354449884.024698, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00238513946533', 'lastCheck': 1354449888.7277989, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00198006629944', 'lastCheck': 1354449892.3326001, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00613403320312', 'lastCheck': 1354449888.7355559, 'code': 0, 'valid': True}} >Thread-25770::DEBUG::2012-12-02 14:04:53,803::task::568::TaskManager.Task::(_updateState) Task=`5b4101f1-a138-4f0d-9796-c2a241710005`::moving from state preparing -> state finished >Thread-25770::DEBUG::2012-12-02 14:04:53,803::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25770::DEBUG::2012-12-02 14:04:53,803::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25770::DEBUG::2012-12-02 14:04:53,803::task::957::TaskManager.Task::(_decref) Task=`5b4101f1-a138-4f0d-9796-c2a241710005`::ref 0 aborting False >Thread-25772::DEBUG::2012-12-02 14:04:55,863::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25772::DEBUG::2012-12-02 14:04:55,863::task::568::TaskManager.Task::(_updateState) Task=`7db7200e-c52e-4a62-a93b-12817ffca7bd`::moving from state init -> state preparing >Thread-25772::INFO::2012-12-02 14:04:55,864::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25772::INFO::2012-12-02 14:04:55,864::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25772::DEBUG::2012-12-02 14:04:55,864::task::1151::TaskManager.Task::(prepare) Task=`7db7200e-c52e-4a62-a93b-12817ffca7bd`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25772::DEBUG::2012-12-02 14:04:55,864::task::568::TaskManager.Task::(_updateState) Task=`7db7200e-c52e-4a62-a93b-12817ffca7bd`::moving from state preparing -> state finished >Thread-25772::DEBUG::2012-12-02 14:04:55,864::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25772::DEBUG::2012-12-02 14:04:55,864::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25772::DEBUG::2012-12-02 14:04:55,864::task::957::TaskManager.Task::(_decref) Task=`7db7200e-c52e-4a62-a93b-12817ffca7bd`::ref 0 aborting False >Thread-25773::DEBUG::2012-12-02 14:04:55,867::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25773::DEBUG::2012-12-02 14:04:55,867::task::568::TaskManager.Task::(_updateState) Task=`872c750b-8a9b-4100-a811-9dd461298eee`::moving from state init -> state preparing >Thread-25773::INFO::2012-12-02 14:04:55,868::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25773::DEBUG::2012-12-02 14:04:55,868::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a385cb6f-c52b-49ab-a8f9-b77a4c82591e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25773::DEBUG::2012-12-02 14:04:55,868::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25773::DEBUG::2012-12-02 14:04:55,868::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25773::DEBUG::2012-12-02 14:04:55,868::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a385cb6f-c52b-49ab-a8f9-b77a4c82591e`::Granted request >Thread-25773::DEBUG::2012-12-02 14:04:55,868::task::794::TaskManager.Task::(resourceAcquired) Task=`872c750b-8a9b-4100-a811-9dd461298eee`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25773::DEBUG::2012-12-02 14:04:55,868::task::957::TaskManager.Task::(_decref) Task=`872c750b-8a9b-4100-a811-9dd461298eee`::ref 1 aborting False >Thread-25773::INFO::2012-12-02 14:04:55,873::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25773::DEBUG::2012-12-02 14:04:55,873::task::1151::TaskManager.Task::(prepare) Task=`872c750b-8a9b-4100-a811-9dd461298eee`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25773::DEBUG::2012-12-02 14:04:55,874::task::568::TaskManager.Task::(_updateState) Task=`872c750b-8a9b-4100-a811-9dd461298eee`::moving from state preparing -> state finished >Thread-25773::DEBUG::2012-12-02 14:04:55,874::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25773::DEBUG::2012-12-02 14:04:55,875::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25773::DEBUG::2012-12-02 14:04:55,875::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25773::DEBUG::2012-12-02 14:04:55,875::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25773::DEBUG::2012-12-02 14:04:55,875::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25773::DEBUG::2012-12-02 14:04:55,875::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25773::DEBUG::2012-12-02 14:04:55,876::task::957::TaskManager.Task::(_decref) Task=`872c750b-8a9b-4100-a811-9dd461298eee`::ref 0 aborting False >Thread-25778::DEBUG::2012-12-02 14:05:03,905::task::568::TaskManager.Task::(_updateState) Task=`4b994afc-16b5-424e-94e6-a655dd7651dd`::moving from state init -> state preparing >Thread-25778::INFO::2012-12-02 14:05:03,905::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25778::INFO::2012-12-02 14:05:03,905::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0129098892212', 'lastCheck': 1354449894.047524, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00237417221069', 'lastCheck': 1354449898.742842, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00190997123718', 'lastCheck': 1354449902.3423989, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0117659568787', 'lastCheck': 1354449898.7568879, 'code': 0, 'valid': True}} >Thread-25778::DEBUG::2012-12-02 14:05:03,905::task::1151::TaskManager.Task::(prepare) Task=`4b994afc-16b5-424e-94e6-a655dd7651dd`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0129098892212', 'lastCheck': 1354449894.047524, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00237417221069', 'lastCheck': 1354449898.742842, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00190997123718', 'lastCheck': 1354449902.3423989, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0117659568787', 'lastCheck': 1354449898.7568879, 'code': 0, 'valid': True}} >Thread-25778::DEBUG::2012-12-02 14:05:03,906::task::568::TaskManager.Task::(_updateState) Task=`4b994afc-16b5-424e-94e6-a655dd7651dd`::moving from state preparing -> state finished >Thread-25778::DEBUG::2012-12-02 14:05:03,906::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25778::DEBUG::2012-12-02 14:05:03,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25778::DEBUG::2012-12-02 14:05:03,906::task::957::TaskManager.Task::(_decref) Task=`4b994afc-16b5-424e-94e6-a655dd7651dd`::ref 0 aborting False >Thread-25781::DEBUG::2012-12-02 14:05:06,108::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25781::DEBUG::2012-12-02 14:05:06,108::task::568::TaskManager.Task::(_updateState) Task=`fbf43f8e-0218-4a56-b074-dcea96ca7d61`::moving from state init -> state preparing >Thread-25781::INFO::2012-12-02 14:05:06,108::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25781::INFO::2012-12-02 14:05:06,108::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25781::DEBUG::2012-12-02 14:05:06,108::task::1151::TaskManager.Task::(prepare) Task=`fbf43f8e-0218-4a56-b074-dcea96ca7d61`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25781::DEBUG::2012-12-02 14:05:06,109::task::568::TaskManager.Task::(_updateState) Task=`fbf43f8e-0218-4a56-b074-dcea96ca7d61`::moving from state preparing -> state finished >Thread-25781::DEBUG::2012-12-02 14:05:06,109::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25781::DEBUG::2012-12-02 14:05:06,109::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25781::DEBUG::2012-12-02 14:05:06,109::task::957::TaskManager.Task::(_decref) Task=`fbf43f8e-0218-4a56-b074-dcea96ca7d61`::ref 0 aborting False >Thread-25782::DEBUG::2012-12-02 14:05:06,112::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25782::DEBUG::2012-12-02 14:05:06,112::task::568::TaskManager.Task::(_updateState) Task=`ff866dc5-73ed-4ac9-9b63-64ea12431f8c`::moving from state init -> state preparing >Thread-25782::INFO::2012-12-02 14:05:06,112::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25782::DEBUG::2012-12-02 14:05:06,112::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3a573e01-1635-4ee4-affe-faff5dc68268`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25782::DEBUG::2012-12-02 14:05:06,113::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25782::DEBUG::2012-12-02 14:05:06,113::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25782::DEBUG::2012-12-02 14:05:06,113::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3a573e01-1635-4ee4-affe-faff5dc68268`::Granted request >Thread-25782::DEBUG::2012-12-02 14:05:06,113::task::794::TaskManager.Task::(resourceAcquired) Task=`ff866dc5-73ed-4ac9-9b63-64ea12431f8c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25782::DEBUG::2012-12-02 14:05:06,113::task::957::TaskManager.Task::(_decref) Task=`ff866dc5-73ed-4ac9-9b63-64ea12431f8c`::ref 1 aborting False >Thread-25782::INFO::2012-12-02 14:05:06,118::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25782::DEBUG::2012-12-02 14:05:06,118::task::1151::TaskManager.Task::(prepare) Task=`ff866dc5-73ed-4ac9-9b63-64ea12431f8c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25782::DEBUG::2012-12-02 14:05:06,118::task::568::TaskManager.Task::(_updateState) Task=`ff866dc5-73ed-4ac9-9b63-64ea12431f8c`::moving from state preparing -> state finished >Thread-25782::DEBUG::2012-12-02 14:05:06,119::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25782::DEBUG::2012-12-02 14:05:06,119::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25782::DEBUG::2012-12-02 14:05:06,119::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25782::DEBUG::2012-12-02 14:05:06,119::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25782::DEBUG::2012-12-02 14:05:06,119::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25782::DEBUG::2012-12-02 14:05:06,119::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25782::DEBUG::2012-12-02 14:05:06,119::task::957::TaskManager.Task::(_decref) Task=`ff866dc5-73ed-4ac9-9b63-64ea12431f8c`::ref 0 aborting False >Thread-25786::DEBUG::2012-12-02 14:05:13,962::task::568::TaskManager.Task::(_updateState) Task=`5281eb0b-b928-480b-80b4-96f80bab9ba3`::moving from state init -> state preparing >Thread-25786::INFO::2012-12-02 14:05:13,963::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25786::INFO::2012-12-02 14:05:13,963::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00343012809753', 'lastCheck': 1354449904.0594471, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00306010246277', 'lastCheck': 1354449908.7559221, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00429606437683', 'lastCheck': 1354449912.355639, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00665807723999', 'lastCheck': 1354449908.7727211, 'code': 0, 'valid': True}} >Thread-25786::DEBUG::2012-12-02 14:05:13,963::task::1151::TaskManager.Task::(prepare) Task=`5281eb0b-b928-480b-80b4-96f80bab9ba3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00343012809753', 'lastCheck': 1354449904.0594471, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00306010246277', 'lastCheck': 1354449908.7559221, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00429606437683', 'lastCheck': 1354449912.355639, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00665807723999', 'lastCheck': 1354449908.7727211, 'code': 0, 'valid': True}} >Thread-25786::DEBUG::2012-12-02 14:05:13,963::task::568::TaskManager.Task::(_updateState) Task=`5281eb0b-b928-480b-80b4-96f80bab9ba3`::moving from state preparing -> state finished >Thread-25786::DEBUG::2012-12-02 14:05:13,963::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25786::DEBUG::2012-12-02 14:05:13,963::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25786::DEBUG::2012-12-02 14:05:13,963::task::957::TaskManager.Task::(_decref) Task=`5281eb0b-b928-480b-80b4-96f80bab9ba3`::ref 0 aborting False >Thread-25789::DEBUG::2012-12-02 14:05:16,210::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25789::DEBUG::2012-12-02 14:05:16,211::task::568::TaskManager.Task::(_updateState) Task=`c54bbb4a-c4a3-4617-85b2-5f60aae8088e`::moving from state init -> state preparing >Thread-25789::INFO::2012-12-02 14:05:16,211::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25789::INFO::2012-12-02 14:05:16,211::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25789::DEBUG::2012-12-02 14:05:16,211::task::1151::TaskManager.Task::(prepare) Task=`c54bbb4a-c4a3-4617-85b2-5f60aae8088e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25789::DEBUG::2012-12-02 14:05:16,211::task::568::TaskManager.Task::(_updateState) Task=`c54bbb4a-c4a3-4617-85b2-5f60aae8088e`::moving from state preparing -> state finished >Thread-25789::DEBUG::2012-12-02 14:05:16,211::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25789::DEBUG::2012-12-02 14:05:16,212::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25789::DEBUG::2012-12-02 14:05:16,212::task::957::TaskManager.Task::(_decref) Task=`c54bbb4a-c4a3-4617-85b2-5f60aae8088e`::ref 0 aborting False >Thread-25790::DEBUG::2012-12-02 14:05:16,214::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25790::DEBUG::2012-12-02 14:05:16,214::task::568::TaskManager.Task::(_updateState) Task=`525e2a44-6769-4289-9a28-f1c4b821b746`::moving from state init -> state preparing >Thread-25790::INFO::2012-12-02 14:05:16,214::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25790::DEBUG::2012-12-02 14:05:16,215::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f0984103-738a-4977-8f70-b17475baf6ef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25790::DEBUG::2012-12-02 14:05:16,215::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25790::DEBUG::2012-12-02 14:05:16,215::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25790::DEBUG::2012-12-02 14:05:16,215::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f0984103-738a-4977-8f70-b17475baf6ef`::Granted request >Thread-25790::DEBUG::2012-12-02 14:05:16,215::task::794::TaskManager.Task::(resourceAcquired) Task=`525e2a44-6769-4289-9a28-f1c4b821b746`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25790::DEBUG::2012-12-02 14:05:16,216::task::957::TaskManager.Task::(_decref) Task=`525e2a44-6769-4289-9a28-f1c4b821b746`::ref 1 aborting False >Thread-25790::INFO::2012-12-02 14:05:16,220::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25790::DEBUG::2012-12-02 14:05:16,220::task::1151::TaskManager.Task::(prepare) Task=`525e2a44-6769-4289-9a28-f1c4b821b746`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25790::DEBUG::2012-12-02 14:05:16,220::task::568::TaskManager.Task::(_updateState) Task=`525e2a44-6769-4289-9a28-f1c4b821b746`::moving from state preparing -> state finished >Thread-25790::DEBUG::2012-12-02 14:05:16,220::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25790::DEBUG::2012-12-02 14:05:16,220::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25790::DEBUG::2012-12-02 14:05:16,220::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25790::DEBUG::2012-12-02 14:05:16,220::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25790::DEBUG::2012-12-02 14:05:16,221::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25790::DEBUG::2012-12-02 14:05:16,221::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25790::DEBUG::2012-12-02 14:05:16,221::task::957::TaskManager.Task::(_decref) Task=`525e2a44-6769-4289-9a28-f1c4b821b746`::ref 0 aborting False >Thread-25794::DEBUG::2012-12-02 14:05:24,025::task::568::TaskManager.Task::(_updateState) Task=`4a17e445-3f68-4fb8-8f06-4ed55ea26589`::moving from state init -> state preparing >Thread-25794::INFO::2012-12-02 14:05:24,025::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25794::INFO::2012-12-02 14:05:24,025::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00994396209717', 'lastCheck': 1354449914.0796349, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00215911865234', 'lastCheck': 1354449918.7703431, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00204205513', 'lastCheck': 1354449922.3673329, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00677108764648', 'lastCheck': 1354449918.7875071, 'code': 0, 'valid': True}} >Thread-25794::DEBUG::2012-12-02 14:05:24,025::task::1151::TaskManager.Task::(prepare) Task=`4a17e445-3f68-4fb8-8f06-4ed55ea26589`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00994396209717', 'lastCheck': 1354449914.0796349, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00215911865234', 'lastCheck': 1354449918.7703431, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00204205513', 'lastCheck': 1354449922.3673329, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00677108764648', 'lastCheck': 1354449918.7875071, 'code': 0, 'valid': True}} >Thread-25794::DEBUG::2012-12-02 14:05:24,025::task::568::TaskManager.Task::(_updateState) Task=`4a17e445-3f68-4fb8-8f06-4ed55ea26589`::moving from state preparing -> state finished >Thread-25794::DEBUG::2012-12-02 14:05:24,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25794::DEBUG::2012-12-02 14:05:24,026::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25794::DEBUG::2012-12-02 14:05:24,026::task::957::TaskManager.Task::(_decref) Task=`4a17e445-3f68-4fb8-8f06-4ed55ea26589`::ref 0 aborting False >Thread-25797::DEBUG::2012-12-02 14:05:26,313::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25797::DEBUG::2012-12-02 14:05:26,313::task::568::TaskManager.Task::(_updateState) Task=`70751a23-5bc9-48b9-b70f-6961ca71cd4f`::moving from state init -> state preparing >Thread-25797::INFO::2012-12-02 14:05:26,314::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25797::INFO::2012-12-02 14:05:26,314::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25797::DEBUG::2012-12-02 14:05:26,314::task::1151::TaskManager.Task::(prepare) Task=`70751a23-5bc9-48b9-b70f-6961ca71cd4f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25797::DEBUG::2012-12-02 14:05:26,314::task::568::TaskManager.Task::(_updateState) Task=`70751a23-5bc9-48b9-b70f-6961ca71cd4f`::moving from state preparing -> state finished >Thread-25797::DEBUG::2012-12-02 14:05:26,314::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25797::DEBUG::2012-12-02 14:05:26,314::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25797::DEBUG::2012-12-02 14:05:26,314::task::957::TaskManager.Task::(_decref) Task=`70751a23-5bc9-48b9-b70f-6961ca71cd4f`::ref 0 aborting False >Thread-25798::DEBUG::2012-12-02 14:05:26,317::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25798::DEBUG::2012-12-02 14:05:26,317::task::568::TaskManager.Task::(_updateState) Task=`1e4f7f1f-72a8-488a-a2a5-c87b79c2e5c1`::moving from state init -> state preparing >Thread-25798::INFO::2012-12-02 14:05:26,317::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25798::DEBUG::2012-12-02 14:05:26,318::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ffa05b46-9a88-41e2-9702-abd9e76f1be8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25798::DEBUG::2012-12-02 14:05:26,318::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25798::DEBUG::2012-12-02 14:05:26,318::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25798::DEBUG::2012-12-02 14:05:26,318::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ffa05b46-9a88-41e2-9702-abd9e76f1be8`::Granted request >Thread-25798::DEBUG::2012-12-02 14:05:26,318::task::794::TaskManager.Task::(resourceAcquired) Task=`1e4f7f1f-72a8-488a-a2a5-c87b79c2e5c1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25798::DEBUG::2012-12-02 14:05:26,318::task::957::TaskManager.Task::(_decref) Task=`1e4f7f1f-72a8-488a-a2a5-c87b79c2e5c1`::ref 1 aborting False >Thread-25798::INFO::2012-12-02 14:05:26,325::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25798::DEBUG::2012-12-02 14:05:26,325::task::1151::TaskManager.Task::(prepare) Task=`1e4f7f1f-72a8-488a-a2a5-c87b79c2e5c1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25798::DEBUG::2012-12-02 14:05:26,325::task::568::TaskManager.Task::(_updateState) Task=`1e4f7f1f-72a8-488a-a2a5-c87b79c2e5c1`::moving from state preparing -> state finished >Thread-25798::DEBUG::2012-12-02 14:05:26,325::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25798::DEBUG::2012-12-02 14:05:26,325::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25798::DEBUG::2012-12-02 14:05:26,325::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25798::DEBUG::2012-12-02 14:05:26,326::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25798::DEBUG::2012-12-02 14:05:26,326::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25798::DEBUG::2012-12-02 14:05:26,326::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25798::DEBUG::2012-12-02 14:05:26,326::task::957::TaskManager.Task::(_decref) Task=`1e4f7f1f-72a8-488a-a2a5-c87b79c2e5c1`::ref 0 aborting False >Thread-25802::DEBUG::2012-12-02 14:05:34,128::task::568::TaskManager.Task::(_updateState) Task=`818af826-ced1-48dd-957c-f55f191a4c44`::moving from state init -> state preparing >Thread-25802::INFO::2012-12-02 14:05:34,128::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25802::INFO::2012-12-02 14:05:34,128::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00879907608032', 'lastCheck': 1354449934.1188431, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00342702865601', 'lastCheck': 1354449928.783628, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0031521320343', 'lastCheck': 1354449932.3813939, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00740790367126', 'lastCheck': 1354449928.803838, 'code': 0, 'valid': True}} >Thread-25802::DEBUG::2012-12-02 14:05:34,128::task::1151::TaskManager.Task::(prepare) Task=`818af826-ced1-48dd-957c-f55f191a4c44`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00879907608032', 'lastCheck': 1354449934.1188431, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00342702865601', 'lastCheck': 1354449928.783628, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0031521320343', 'lastCheck': 1354449932.3813939, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00740790367126', 'lastCheck': 1354449928.803838, 'code': 0, 'valid': True}} >Thread-25802::DEBUG::2012-12-02 14:05:34,128::task::568::TaskManager.Task::(_updateState) Task=`818af826-ced1-48dd-957c-f55f191a4c44`::moving from state preparing -> state finished >Thread-25802::DEBUG::2012-12-02 14:05:34,129::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25802::DEBUG::2012-12-02 14:05:34,129::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25802::DEBUG::2012-12-02 14:05:34,129::task::957::TaskManager.Task::(_decref) Task=`818af826-ced1-48dd-957c-f55f191a4c44`::ref 0 aborting False >Thread-25805::DEBUG::2012-12-02 14:05:36,872::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25805::DEBUG::2012-12-02 14:05:36,873::task::568::TaskManager.Task::(_updateState) Task=`bffd3349-4043-4320-821d-3172640cf447`::moving from state init -> state preparing >Thread-25805::INFO::2012-12-02 14:05:36,873::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25805::INFO::2012-12-02 14:05:36,873::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25805::DEBUG::2012-12-02 14:05:36,873::task::1151::TaskManager.Task::(prepare) Task=`bffd3349-4043-4320-821d-3172640cf447`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25805::DEBUG::2012-12-02 14:05:36,873::task::568::TaskManager.Task::(_updateState) Task=`bffd3349-4043-4320-821d-3172640cf447`::moving from state preparing -> state finished >Thread-25805::DEBUG::2012-12-02 14:05:36,873::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25805::DEBUG::2012-12-02 14:05:36,874::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25805::DEBUG::2012-12-02 14:05:36,874::task::957::TaskManager.Task::(_decref) Task=`bffd3349-4043-4320-821d-3172640cf447`::ref 0 aborting False >Thread-25806::DEBUG::2012-12-02 14:05:36,876::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25806::DEBUG::2012-12-02 14:05:36,876::task::568::TaskManager.Task::(_updateState) Task=`d75136cf-c2fb-4272-95a5-b8f6bb72a22e`::moving from state init -> state preparing >Thread-25806::INFO::2012-12-02 14:05:36,877::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25806::DEBUG::2012-12-02 14:05:36,877::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`fff9ffed-b2cb-4bc4-9c61-28e249cbd3ad`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25806::DEBUG::2012-12-02 14:05:36,877::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25806::DEBUG::2012-12-02 14:05:36,877::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25806::DEBUG::2012-12-02 14:05:36,877::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`fff9ffed-b2cb-4bc4-9c61-28e249cbd3ad`::Granted request >Thread-25806::DEBUG::2012-12-02 14:05:36,878::task::794::TaskManager.Task::(resourceAcquired) Task=`d75136cf-c2fb-4272-95a5-b8f6bb72a22e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25806::DEBUG::2012-12-02 14:05:36,878::task::957::TaskManager.Task::(_decref) Task=`d75136cf-c2fb-4272-95a5-b8f6bb72a22e`::ref 1 aborting False >Thread-25806::INFO::2012-12-02 14:05:36,884::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25806::DEBUG::2012-12-02 14:05:36,884::task::1151::TaskManager.Task::(prepare) Task=`d75136cf-c2fb-4272-95a5-b8f6bb72a22e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25806::DEBUG::2012-12-02 14:05:36,884::task::568::TaskManager.Task::(_updateState) Task=`d75136cf-c2fb-4272-95a5-b8f6bb72a22e`::moving from state preparing -> state finished >Thread-25806::DEBUG::2012-12-02 14:05:36,885::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25806::DEBUG::2012-12-02 14:05:36,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25806::DEBUG::2012-12-02 14:05:36,885::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25806::DEBUG::2012-12-02 14:05:36,885::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25806::DEBUG::2012-12-02 14:05:36,885::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25806::DEBUG::2012-12-02 14:05:36,885::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25806::DEBUG::2012-12-02 14:05:36,885::task::957::TaskManager.Task::(_decref) Task=`d75136cf-c2fb-4272-95a5-b8f6bb72a22e`::ref 0 aborting False >Thread-25810::DEBUG::2012-12-02 14:05:44,281::task::568::TaskManager.Task::(_updateState) Task=`49c98460-8db7-429b-806f-d17a4baf79df`::moving from state init -> state preparing >Thread-25810::INFO::2012-12-02 14:05:44,281::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25810::INFO::2012-12-02 14:05:44,282::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00342607498169', 'lastCheck': 1354449944.130434, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00197196006775', 'lastCheck': 1354449938.79636, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00335288047791', 'lastCheck': 1354449942.394793, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00732493400574', 'lastCheck': 1354449938.8204739, 'code': 0, 'valid': True}} >Thread-25810::DEBUG::2012-12-02 14:05:44,282::task::1151::TaskManager.Task::(prepare) Task=`49c98460-8db7-429b-806f-d17a4baf79df`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00342607498169', 'lastCheck': 1354449944.130434, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00197196006775', 'lastCheck': 1354449938.79636, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00335288047791', 'lastCheck': 1354449942.394793, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00732493400574', 'lastCheck': 1354449938.8204739, 'code': 0, 'valid': True}} >Thread-25810::DEBUG::2012-12-02 14:05:44,282::task::568::TaskManager.Task::(_updateState) Task=`49c98460-8db7-429b-806f-d17a4baf79df`::moving from state preparing -> state finished >Thread-25810::DEBUG::2012-12-02 14:05:44,282::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25810::DEBUG::2012-12-02 14:05:44,282::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25810::DEBUG::2012-12-02 14:05:44,282::task::957::TaskManager.Task::(_decref) Task=`49c98460-8db7-429b-806f-d17a4baf79df`::ref 0 aborting False >Thread-25813::DEBUG::2012-12-02 14:05:47,000::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25813::DEBUG::2012-12-02 14:05:47,001::task::568::TaskManager.Task::(_updateState) Task=`12fca162-dbf8-47f2-93af-603e71e796a7`::moving from state init -> state preparing >Thread-25813::INFO::2012-12-02 14:05:47,001::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25813::INFO::2012-12-02 14:05:47,001::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25813::DEBUG::2012-12-02 14:05:47,001::task::1151::TaskManager.Task::(prepare) Task=`12fca162-dbf8-47f2-93af-603e71e796a7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25813::DEBUG::2012-12-02 14:05:47,001::task::568::TaskManager.Task::(_updateState) Task=`12fca162-dbf8-47f2-93af-603e71e796a7`::moving from state preparing -> state finished >Thread-25813::DEBUG::2012-12-02 14:05:47,001::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25813::DEBUG::2012-12-02 14:05:47,001::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25813::DEBUG::2012-12-02 14:05:47,002::task::957::TaskManager.Task::(_decref) Task=`12fca162-dbf8-47f2-93af-603e71e796a7`::ref 0 aborting False >Thread-25814::DEBUG::2012-12-02 14:05:47,004::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25814::DEBUG::2012-12-02 14:05:47,004::task::568::TaskManager.Task::(_updateState) Task=`830b2e0a-b2e8-424d-82e3-a3093a705409`::moving from state init -> state preparing >Thread-25814::INFO::2012-12-02 14:05:47,005::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25814::DEBUG::2012-12-02 14:05:47,005::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0b9d767f-0069-4baf-82c1-b1c843812ee0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25814::DEBUG::2012-12-02 14:05:47,005::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25814::DEBUG::2012-12-02 14:05:47,005::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25814::DEBUG::2012-12-02 14:05:47,005::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0b9d767f-0069-4baf-82c1-b1c843812ee0`::Granted request >Thread-25814::DEBUG::2012-12-02 14:05:47,006::task::794::TaskManager.Task::(resourceAcquired) Task=`830b2e0a-b2e8-424d-82e3-a3093a705409`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25814::DEBUG::2012-12-02 14:05:47,006::task::957::TaskManager.Task::(_decref) Task=`830b2e0a-b2e8-424d-82e3-a3093a705409`::ref 1 aborting False >Thread-25814::INFO::2012-12-02 14:05:47,010::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25814::DEBUG::2012-12-02 14:05:47,010::task::1151::TaskManager.Task::(prepare) Task=`830b2e0a-b2e8-424d-82e3-a3093a705409`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25814::DEBUG::2012-12-02 14:05:47,010::task::568::TaskManager.Task::(_updateState) Task=`830b2e0a-b2e8-424d-82e3-a3093a705409`::moving from state preparing -> state finished >Thread-25814::DEBUG::2012-12-02 14:05:47,010::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25814::DEBUG::2012-12-02 14:05:47,010::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25814::DEBUG::2012-12-02 14:05:47,010::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25814::DEBUG::2012-12-02 14:05:47,011::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25814::DEBUG::2012-12-02 14:05:47,011::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25814::DEBUG::2012-12-02 14:05:47,011::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25814::DEBUG::2012-12-02 14:05:47,011::task::957::TaskManager.Task::(_decref) Task=`830b2e0a-b2e8-424d-82e3-a3093a705409`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:05:52,395::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:05:52,395::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:05:52,395::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:05:52,611::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:05:52,613::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:05:52,613::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:05:52,649::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:05:52,649::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:05:52,653::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:05:52,656::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:05:52,656::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:05:52,656::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-11821::DEBUG::2012-12-02 14:05:54,130::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:05:54,130::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:05:54,131::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:05:54,334::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:05:54,336::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:05:54,336::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:05:54,371::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:05:54,371::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:05:54,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:05:54,384::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:05:54,384::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:05:54,384::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-25940::DEBUG::2012-12-02 14:05:54,407::task::568::TaskManager.Task::(_updateState) Task=`861678bd-c60c-4142-8ce1-0a853517dbd6`::moving from state init -> state preparing >Thread-25940::INFO::2012-12-02 14:05:54,407::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-25940::INFO::2012-12-02 14:05:54,407::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00622296333313', 'lastCheck': 1354449954.3997791, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00202298164368', 'lastCheck': 1354449948.8082261, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0014820098877', 'lastCheck': 1354449952.6667271, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00633406639099', 'lastCheck': 1354449948.8356669, 'code': 0, 'valid': True}} >Thread-25940::DEBUG::2012-12-02 14:05:54,407::task::1151::TaskManager.Task::(prepare) Task=`861678bd-c60c-4142-8ce1-0a853517dbd6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00622296333313', 'lastCheck': 1354449954.3997791, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00202298164368', 'lastCheck': 1354449948.8082261, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0014820098877', 'lastCheck': 1354449952.6667271, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00633406639099', 'lastCheck': 1354449948.8356669, 'code': 0, 'valid': True}} >Thread-25940::DEBUG::2012-12-02 14:05:54,408::task::568::TaskManager.Task::(_updateState) Task=`861678bd-c60c-4142-8ce1-0a853517dbd6`::moving from state preparing -> state finished >Thread-25940::DEBUG::2012-12-02 14:05:54,408::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25940::DEBUG::2012-12-02 14:05:54,408::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25940::DEBUG::2012-12-02 14:05:54,408::task::957::TaskManager.Task::(_decref) Task=`861678bd-c60c-4142-8ce1-0a853517dbd6`::ref 0 aborting False >Thread-25943::DEBUG::2012-12-02 14:05:57,104::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25943::DEBUG::2012-12-02 14:05:57,104::task::568::TaskManager.Task::(_updateState) Task=`a75cc6fc-d326-4013-9e62-a42718243ab4`::moving from state init -> state preparing >Thread-25943::INFO::2012-12-02 14:05:57,105::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25943::INFO::2012-12-02 14:05:57,105::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25943::DEBUG::2012-12-02 14:05:57,105::task::1151::TaskManager.Task::(prepare) Task=`a75cc6fc-d326-4013-9e62-a42718243ab4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-25943::DEBUG::2012-12-02 14:05:57,105::task::568::TaskManager.Task::(_updateState) Task=`a75cc6fc-d326-4013-9e62-a42718243ab4`::moving from state preparing -> state finished >Thread-25943::DEBUG::2012-12-02 14:05:57,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-25943::DEBUG::2012-12-02 14:05:57,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25943::DEBUG::2012-12-02 14:05:57,105::task::957::TaskManager.Task::(_decref) Task=`a75cc6fc-d326-4013-9e62-a42718243ab4`::ref 0 aborting False >Thread-25944::DEBUG::2012-12-02 14:05:57,108::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-25944::DEBUG::2012-12-02 14:05:57,108::task::568::TaskManager.Task::(_updateState) Task=`e1cf17c2-ca5d-457b-ba64-4474d0c95608`::moving from state init -> state preparing >Thread-25944::INFO::2012-12-02 14:05:57,108::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-25944::DEBUG::2012-12-02 14:05:57,108::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9a35c7f7-6458-46da-a8bd-65f058a85878`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-25944::DEBUG::2012-12-02 14:05:57,108::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-25944::DEBUG::2012-12-02 14:05:57,109::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-25944::DEBUG::2012-12-02 14:05:57,109::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9a35c7f7-6458-46da-a8bd-65f058a85878`::Granted request >Thread-25944::DEBUG::2012-12-02 14:05:57,109::task::794::TaskManager.Task::(resourceAcquired) Task=`e1cf17c2-ca5d-457b-ba64-4474d0c95608`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-25944::DEBUG::2012-12-02 14:05:57,109::task::957::TaskManager.Task::(_decref) Task=`e1cf17c2-ca5d-457b-ba64-4474d0c95608`::ref 1 aborting False >Thread-25944::INFO::2012-12-02 14:05:57,113::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25944::DEBUG::2012-12-02 14:05:57,113::task::1151::TaskManager.Task::(prepare) Task=`e1cf17c2-ca5d-457b-ba64-4474d0c95608`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-25944::DEBUG::2012-12-02 14:05:57,113::task::568::TaskManager.Task::(_updateState) Task=`e1cf17c2-ca5d-457b-ba64-4474d0c95608`::moving from state preparing -> state finished >Thread-25944::DEBUG::2012-12-02 14:05:57,114::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-25944::DEBUG::2012-12-02 14:05:57,114::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-25944::DEBUG::2012-12-02 14:05:57,114::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-25944::DEBUG::2012-12-02 14:05:57,114::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-25944::DEBUG::2012-12-02 14:05:57,114::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-25944::DEBUG::2012-12-02 14:05:57,114::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-25944::DEBUG::2012-12-02 14:05:57,114::task::957::TaskManager.Task::(_decref) Task=`e1cf17c2-ca5d-457b-ba64-4474d0c95608`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:05:58,808::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:05:58,808::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:05:58,809::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:05:59,013::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:05:59,015::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:05:59,015::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:05:59,050::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:05:59,050::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:05:59,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:05:59,055::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:05:59,055::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:05:59,055::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:05:59,055::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:05:59,055::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:05:59,058::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:05:59,263::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:05:59,264::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:05:59,265::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:05:59,299::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:05:59,300::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:05:59,308::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:05:59,311::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:05:59,311::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:05:59,311::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-26070::DEBUG::2012-12-02 14:06:04,473::task::568::TaskManager.Task::(_updateState) Task=`2a16b0c5-4228-4bc6-ab59-9d28d335c1e1`::moving from state init -> state preparing >Thread-26070::INFO::2012-12-02 14:06:04,474::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26070::INFO::2012-12-02 14:06:04,474::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00987887382507', 'lastCheck': 1354449964.42272, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00241994857788', 'lastCheck': 1354449959.31897, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191116333008', 'lastCheck': 1354449962.677856, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0068519115448', 'lastCheck': 1354449959.327796, 'code': 0, 'valid': True}} >Thread-26070::DEBUG::2012-12-02 14:06:04,474::task::1151::TaskManager.Task::(prepare) Task=`2a16b0c5-4228-4bc6-ab59-9d28d335c1e1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00987887382507', 'lastCheck': 1354449964.42272, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00241994857788', 'lastCheck': 1354449959.31897, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191116333008', 'lastCheck': 1354449962.677856, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0068519115448', 'lastCheck': 1354449959.327796, 'code': 0, 'valid': True}} >Thread-26070::DEBUG::2012-12-02 14:06:04,474::task::568::TaskManager.Task::(_updateState) Task=`2a16b0c5-4228-4bc6-ab59-9d28d335c1e1`::moving from state preparing -> state finished >Thread-26070::DEBUG::2012-12-02 14:06:04,474::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26070::DEBUG::2012-12-02 14:06:04,474::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26070::DEBUG::2012-12-02 14:06:04,475::task::957::TaskManager.Task::(_decref) Task=`2a16b0c5-4228-4bc6-ab59-9d28d335c1e1`::ref 0 aborting False >Thread-26073::DEBUG::2012-12-02 14:06:07,596::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26073::DEBUG::2012-12-02 14:06:07,596::task::568::TaskManager.Task::(_updateState) Task=`5c857b72-e928-43d6-a9a9-27f58bbaadcc`::moving from state init -> state preparing >Thread-26073::INFO::2012-12-02 14:06:07,597::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26073::INFO::2012-12-02 14:06:07,597::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26073::DEBUG::2012-12-02 14:06:07,597::task::1151::TaskManager.Task::(prepare) Task=`5c857b72-e928-43d6-a9a9-27f58bbaadcc`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26073::DEBUG::2012-12-02 14:06:07,597::task::568::TaskManager.Task::(_updateState) Task=`5c857b72-e928-43d6-a9a9-27f58bbaadcc`::moving from state preparing -> state finished >Thread-26073::DEBUG::2012-12-02 14:06:07,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26073::DEBUG::2012-12-02 14:06:07,597::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26073::DEBUG::2012-12-02 14:06:07,597::task::957::TaskManager.Task::(_decref) Task=`5c857b72-e928-43d6-a9a9-27f58bbaadcc`::ref 0 aborting False >Thread-26074::DEBUG::2012-12-02 14:06:07,600::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26074::DEBUG::2012-12-02 14:06:07,600::task::568::TaskManager.Task::(_updateState) Task=`c2accbe3-df11-4d5d-9d3d-8bb026bea48d`::moving from state init -> state preparing >Thread-26074::INFO::2012-12-02 14:06:07,600::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26074::DEBUG::2012-12-02 14:06:07,601::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1152d537-d5b1-4c23-8d5b-b15c5821ca97`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26074::DEBUG::2012-12-02 14:06:07,601::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26074::DEBUG::2012-12-02 14:06:07,601::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26074::DEBUG::2012-12-02 14:06:07,601::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1152d537-d5b1-4c23-8d5b-b15c5821ca97`::Granted request >Thread-26074::DEBUG::2012-12-02 14:06:07,601::task::794::TaskManager.Task::(resourceAcquired) Task=`c2accbe3-df11-4d5d-9d3d-8bb026bea48d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26074::DEBUG::2012-12-02 14:06:07,602::task::957::TaskManager.Task::(_decref) Task=`c2accbe3-df11-4d5d-9d3d-8bb026bea48d`::ref 1 aborting False >Thread-26074::INFO::2012-12-02 14:06:07,607::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26074::DEBUG::2012-12-02 14:06:07,607::task::1151::TaskManager.Task::(prepare) Task=`c2accbe3-df11-4d5d-9d3d-8bb026bea48d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26074::DEBUG::2012-12-02 14:06:07,608::task::568::TaskManager.Task::(_updateState) Task=`c2accbe3-df11-4d5d-9d3d-8bb026bea48d`::moving from state preparing -> state finished >Thread-26074::DEBUG::2012-12-02 14:06:07,608::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26074::DEBUG::2012-12-02 14:06:07,608::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26074::DEBUG::2012-12-02 14:06:07,608::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26074::DEBUG::2012-12-02 14:06:07,608::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26074::DEBUG::2012-12-02 14:06:07,608::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26074::DEBUG::2012-12-02 14:06:07,608::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26074::DEBUG::2012-12-02 14:06:07,608::task::957::TaskManager.Task::(_decref) Task=`c2accbe3-df11-4d5d-9d3d-8bb026bea48d`::ref 0 aborting False >Thread-26078::DEBUG::2012-12-02 14:06:15,534::task::568::TaskManager.Task::(_updateState) Task=`33990762-05e3-4d32-8824-ef2e3256823f`::moving from state init -> state preparing >Thread-26078::INFO::2012-12-02 14:06:15,534::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26078::INFO::2012-12-02 14:06:15,534::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00857996940613', 'lastCheck': 1354449974.4409471, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00402593612671', 'lastCheck': 1354449969.3364351, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00400710105896', 'lastCheck': 1354449972.6929309, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00765109062195', 'lastCheck': 1354449969.3461969, 'code': 0, 'valid': True}} >Thread-26078::DEBUG::2012-12-02 14:06:15,534::task::1151::TaskManager.Task::(prepare) Task=`33990762-05e3-4d32-8824-ef2e3256823f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00857996940613', 'lastCheck': 1354449974.4409471, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00402593612671', 'lastCheck': 1354449969.3364351, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00400710105896', 'lastCheck': 1354449972.6929309, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00765109062195', 'lastCheck': 1354449969.3461969, 'code': 0, 'valid': True}} >Thread-26078::DEBUG::2012-12-02 14:06:15,534::task::568::TaskManager.Task::(_updateState) Task=`33990762-05e3-4d32-8824-ef2e3256823f`::moving from state preparing -> state finished >Thread-26078::DEBUG::2012-12-02 14:06:15,535::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26078::DEBUG::2012-12-02 14:06:15,535::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26078::DEBUG::2012-12-02 14:06:15,535::task::957::TaskManager.Task::(_decref) Task=`33990762-05e3-4d32-8824-ef2e3256823f`::ref 0 aborting False >Thread-26081::DEBUG::2012-12-02 14:06:17,774::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26081::DEBUG::2012-12-02 14:06:17,775::task::568::TaskManager.Task::(_updateState) Task=`42aa1eb2-b857-4ca2-a38c-e56027e723b0`::moving from state init -> state preparing >Thread-26081::INFO::2012-12-02 14:06:17,775::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26081::INFO::2012-12-02 14:06:17,775::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26081::DEBUG::2012-12-02 14:06:17,775::task::1151::TaskManager.Task::(prepare) Task=`42aa1eb2-b857-4ca2-a38c-e56027e723b0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26081::DEBUG::2012-12-02 14:06:17,775::task::568::TaskManager.Task::(_updateState) Task=`42aa1eb2-b857-4ca2-a38c-e56027e723b0`::moving from state preparing -> state finished >Thread-26081::DEBUG::2012-12-02 14:06:17,775::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26081::DEBUG::2012-12-02 14:06:17,775::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26081::DEBUG::2012-12-02 14:06:17,776::task::957::TaskManager.Task::(_decref) Task=`42aa1eb2-b857-4ca2-a38c-e56027e723b0`::ref 0 aborting False >Thread-26082::DEBUG::2012-12-02 14:06:17,778::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26082::DEBUG::2012-12-02 14:06:17,778::task::568::TaskManager.Task::(_updateState) Task=`7b7a4b56-3c51-4b12-8ebc-894a56ef5311`::moving from state init -> state preparing >Thread-26082::INFO::2012-12-02 14:06:17,778::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26082::DEBUG::2012-12-02 14:06:17,779::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`dc46ec7a-3a5c-4f51-89b5-e4c98bd865bd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26082::DEBUG::2012-12-02 14:06:17,779::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26082::DEBUG::2012-12-02 14:06:17,779::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26082::DEBUG::2012-12-02 14:06:17,779::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`dc46ec7a-3a5c-4f51-89b5-e4c98bd865bd`::Granted request >Thread-26082::DEBUG::2012-12-02 14:06:17,779::task::794::TaskManager.Task::(resourceAcquired) Task=`7b7a4b56-3c51-4b12-8ebc-894a56ef5311`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26082::DEBUG::2012-12-02 14:06:17,780::task::957::TaskManager.Task::(_decref) Task=`7b7a4b56-3c51-4b12-8ebc-894a56ef5311`::ref 1 aborting False >Thread-26082::INFO::2012-12-02 14:06:17,786::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26082::DEBUG::2012-12-02 14:06:17,786::task::1151::TaskManager.Task::(prepare) Task=`7b7a4b56-3c51-4b12-8ebc-894a56ef5311`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26082::DEBUG::2012-12-02 14:06:17,786::task::568::TaskManager.Task::(_updateState) Task=`7b7a4b56-3c51-4b12-8ebc-894a56ef5311`::moving from state preparing -> state finished >Thread-26082::DEBUG::2012-12-02 14:06:17,786::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26082::DEBUG::2012-12-02 14:06:17,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26082::DEBUG::2012-12-02 14:06:17,786::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26082::DEBUG::2012-12-02 14:06:17,786::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26082::DEBUG::2012-12-02 14:06:17,786::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26082::DEBUG::2012-12-02 14:06:17,786::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26082::DEBUG::2012-12-02 14:06:17,787::task::957::TaskManager.Task::(_decref) Task=`7b7a4b56-3c51-4b12-8ebc-894a56ef5311`::ref 0 aborting False >Thread-26086::DEBUG::2012-12-02 14:06:25,616::task::568::TaskManager.Task::(_updateState) Task=`f5bba63e-80c2-4826-9092-a0dacca645d7`::moving from state init -> state preparing >Thread-26086::INFO::2012-12-02 14:06:25,616::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26086::INFO::2012-12-02 14:06:25,616::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00934410095215', 'lastCheck': 1354449984.460665, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00183200836182', 'lastCheck': 1354449979.356086, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00144600868225', 'lastCheck': 1354449982.702565, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00679492950439', 'lastCheck': 1354449979.3631909, 'code': 0, 'valid': True}} >Thread-26086::DEBUG::2012-12-02 14:06:25,616::task::1151::TaskManager.Task::(prepare) Task=`f5bba63e-80c2-4826-9092-a0dacca645d7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00934410095215', 'lastCheck': 1354449984.460665, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00183200836182', 'lastCheck': 1354449979.356086, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00144600868225', 'lastCheck': 1354449982.702565, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00679492950439', 'lastCheck': 1354449979.3631909, 'code': 0, 'valid': True}} >Thread-26086::DEBUG::2012-12-02 14:06:25,617::task::568::TaskManager.Task::(_updateState) Task=`f5bba63e-80c2-4826-9092-a0dacca645d7`::moving from state preparing -> state finished >Thread-26086::DEBUG::2012-12-02 14:06:25,617::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26086::DEBUG::2012-12-02 14:06:25,617::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26086::DEBUG::2012-12-02 14:06:25,617::task::957::TaskManager.Task::(_decref) Task=`f5bba63e-80c2-4826-9092-a0dacca645d7`::ref 0 aborting False >Thread-26089::DEBUG::2012-12-02 14:06:29,096::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26089::DEBUG::2012-12-02 14:06:29,096::task::568::TaskManager.Task::(_updateState) Task=`1ca427ed-b758-4cae-a6b9-d13064ac796e`::moving from state init -> state preparing >Thread-26089::INFO::2012-12-02 14:06:29,096::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26089::INFO::2012-12-02 14:06:29,096::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26089::DEBUG::2012-12-02 14:06:29,097::task::1151::TaskManager.Task::(prepare) Task=`1ca427ed-b758-4cae-a6b9-d13064ac796e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26089::DEBUG::2012-12-02 14:06:29,097::task::568::TaskManager.Task::(_updateState) Task=`1ca427ed-b758-4cae-a6b9-d13064ac796e`::moving from state preparing -> state finished >Thread-26089::DEBUG::2012-12-02 14:06:29,097::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26089::DEBUG::2012-12-02 14:06:29,097::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26089::DEBUG::2012-12-02 14:06:29,097::task::957::TaskManager.Task::(_decref) Task=`1ca427ed-b758-4cae-a6b9-d13064ac796e`::ref 0 aborting False >Thread-26090::DEBUG::2012-12-02 14:06:29,100::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26090::DEBUG::2012-12-02 14:06:29,100::task::568::TaskManager.Task::(_updateState) Task=`b76508c4-6ee0-45de-9205-a02e0dd00fa9`::moving from state init -> state preparing >Thread-26090::INFO::2012-12-02 14:06:29,100::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26090::DEBUG::2012-12-02 14:06:29,100::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`80aa966a-6f7c-4ab6-9465-e62be2063173`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26090::DEBUG::2012-12-02 14:06:29,101::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26090::DEBUG::2012-12-02 14:06:29,101::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26090::DEBUG::2012-12-02 14:06:29,101::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`80aa966a-6f7c-4ab6-9465-e62be2063173`::Granted request >Thread-26090::DEBUG::2012-12-02 14:06:29,101::task::794::TaskManager.Task::(resourceAcquired) Task=`b76508c4-6ee0-45de-9205-a02e0dd00fa9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26090::DEBUG::2012-12-02 14:06:29,101::task::957::TaskManager.Task::(_decref) Task=`b76508c4-6ee0-45de-9205-a02e0dd00fa9`::ref 1 aborting False >Thread-26090::INFO::2012-12-02 14:06:29,107::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26090::DEBUG::2012-12-02 14:06:29,107::task::1151::TaskManager.Task::(prepare) Task=`b76508c4-6ee0-45de-9205-a02e0dd00fa9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26090::DEBUG::2012-12-02 14:06:29,107::task::568::TaskManager.Task::(_updateState) Task=`b76508c4-6ee0-45de-9205-a02e0dd00fa9`::moving from state preparing -> state finished >Thread-26090::DEBUG::2012-12-02 14:06:29,107::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26090::DEBUG::2012-12-02 14:06:29,108::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26090::DEBUG::2012-12-02 14:06:29,108::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26090::DEBUG::2012-12-02 14:06:29,108::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26090::DEBUG::2012-12-02 14:06:29,108::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26090::DEBUG::2012-12-02 14:06:29,108::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26090::DEBUG::2012-12-02 14:06:29,108::task::957::TaskManager.Task::(_decref) Task=`b76508c4-6ee0-45de-9205-a02e0dd00fa9`::ref 0 aborting False >Thread-26094::DEBUG::2012-12-02 14:06:35,733::task::568::TaskManager.Task::(_updateState) Task=`6a34632d-ce3f-4635-8fec-00fd17809efc`::moving from state init -> state preparing >Thread-26094::INFO::2012-12-02 14:06:35,733::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26094::INFO::2012-12-02 14:06:35,734::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00927495956421', 'lastCheck': 1354449994.4824369, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00605797767639', 'lastCheck': 1354449989.3726971, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00285601615906', 'lastCheck': 1354449992.7168939, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00763916969299', 'lastCheck': 1354449989.37971, 'code': 0, 'valid': True}} >Thread-26094::DEBUG::2012-12-02 14:06:35,734::task::1151::TaskManager.Task::(prepare) Task=`6a34632d-ce3f-4635-8fec-00fd17809efc`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00927495956421', 'lastCheck': 1354449994.4824369, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00605797767639', 'lastCheck': 1354449989.3726971, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00285601615906', 'lastCheck': 1354449992.7168939, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00763916969299', 'lastCheck': 1354449989.37971, 'code': 0, 'valid': True}} >Thread-26094::DEBUG::2012-12-02 14:06:35,734::task::568::TaskManager.Task::(_updateState) Task=`6a34632d-ce3f-4635-8fec-00fd17809efc`::moving from state preparing -> state finished >Thread-26094::DEBUG::2012-12-02 14:06:35,734::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26094::DEBUG::2012-12-02 14:06:35,734::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26094::DEBUG::2012-12-02 14:06:35,734::task::957::TaskManager.Task::(_decref) Task=`6a34632d-ce3f-4635-8fec-00fd17809efc`::ref 0 aborting False >Thread-26097::DEBUG::2012-12-02 14:06:39,341::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26097::DEBUG::2012-12-02 14:06:39,341::task::568::TaskManager.Task::(_updateState) Task=`6b0d3479-d880-4e15-ab8e-64cb995d1390`::moving from state init -> state preparing >Thread-26097::INFO::2012-12-02 14:06:39,341::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26097::INFO::2012-12-02 14:06:39,342::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26097::DEBUG::2012-12-02 14:06:39,342::task::1151::TaskManager.Task::(prepare) Task=`6b0d3479-d880-4e15-ab8e-64cb995d1390`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26097::DEBUG::2012-12-02 14:06:39,342::task::568::TaskManager.Task::(_updateState) Task=`6b0d3479-d880-4e15-ab8e-64cb995d1390`::moving from state preparing -> state finished >Thread-26097::DEBUG::2012-12-02 14:06:39,342::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26097::DEBUG::2012-12-02 14:06:39,342::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26097::DEBUG::2012-12-02 14:06:39,342::task::957::TaskManager.Task::(_decref) Task=`6b0d3479-d880-4e15-ab8e-64cb995d1390`::ref 0 aborting False >Thread-26098::DEBUG::2012-12-02 14:06:39,345::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26098::DEBUG::2012-12-02 14:06:39,345::task::568::TaskManager.Task::(_updateState) Task=`4f5c8ffc-b7d5-4a8b-a9b8-0c5ddb739ba1`::moving from state init -> state preparing >Thread-26098::INFO::2012-12-02 14:06:39,345::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26098::DEBUG::2012-12-02 14:06:39,346::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`090b7054-ddc7-49b0-b4f4-dc5ec261066d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26098::DEBUG::2012-12-02 14:06:39,346::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26098::DEBUG::2012-12-02 14:06:39,346::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26098::DEBUG::2012-12-02 14:06:39,346::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`090b7054-ddc7-49b0-b4f4-dc5ec261066d`::Granted request >Thread-26098::DEBUG::2012-12-02 14:06:39,346::task::794::TaskManager.Task::(resourceAcquired) Task=`4f5c8ffc-b7d5-4a8b-a9b8-0c5ddb739ba1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26098::DEBUG::2012-12-02 14:06:39,346::task::957::TaskManager.Task::(_decref) Task=`4f5c8ffc-b7d5-4a8b-a9b8-0c5ddb739ba1`::ref 1 aborting False >Thread-26098::INFO::2012-12-02 14:06:39,353::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26098::DEBUG::2012-12-02 14:06:39,353::task::1151::TaskManager.Task::(prepare) Task=`4f5c8ffc-b7d5-4a8b-a9b8-0c5ddb739ba1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26098::DEBUG::2012-12-02 14:06:39,353::task::568::TaskManager.Task::(_updateState) Task=`4f5c8ffc-b7d5-4a8b-a9b8-0c5ddb739ba1`::moving from state preparing -> state finished >Thread-26098::DEBUG::2012-12-02 14:06:39,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26098::DEBUG::2012-12-02 14:06:39,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26098::DEBUG::2012-12-02 14:06:39,354::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26098::DEBUG::2012-12-02 14:06:39,354::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26098::DEBUG::2012-12-02 14:06:39,354::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26098::DEBUG::2012-12-02 14:06:39,354::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26098::DEBUG::2012-12-02 14:06:39,354::task::957::TaskManager.Task::(_decref) Task=`4f5c8ffc-b7d5-4a8b-a9b8-0c5ddb739ba1`::ref 0 aborting False >Thread-26102::DEBUG::2012-12-02 14:06:45,848::task::568::TaskManager.Task::(_updateState) Task=`b677cb52-3933-4a44-bfe0-2d15ec77f655`::moving from state init -> state preparing >Thread-26102::INFO::2012-12-02 14:06:45,848::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26102::INFO::2012-12-02 14:06:45,848::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00796604156494', 'lastCheck': 1354450004.500077, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266289710999', 'lastCheck': 1354449999.3894429, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191378593445', 'lastCheck': 1354450002.7282341, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00789093971252', 'lastCheck': 1354449999.3977849, 'code': 0, 'valid': True}} >Thread-26102::DEBUG::2012-12-02 14:06:45,849::task::1151::TaskManager.Task::(prepare) Task=`b677cb52-3933-4a44-bfe0-2d15ec77f655`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00796604156494', 'lastCheck': 1354450004.500077, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266289710999', 'lastCheck': 1354449999.3894429, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191378593445', 'lastCheck': 1354450002.7282341, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00789093971252', 'lastCheck': 1354449999.3977849, 'code': 0, 'valid': True}} >Thread-26102::DEBUG::2012-12-02 14:06:45,849::task::568::TaskManager.Task::(_updateState) Task=`b677cb52-3933-4a44-bfe0-2d15ec77f655`::moving from state preparing -> state finished >Thread-26102::DEBUG::2012-12-02 14:06:45,849::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26102::DEBUG::2012-12-02 14:06:45,849::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26102::DEBUG::2012-12-02 14:06:45,849::task::957::TaskManager.Task::(_decref) Task=`b677cb52-3933-4a44-bfe0-2d15ec77f655`::ref 0 aborting False >Thread-26105::DEBUG::2012-12-02 14:06:49,484::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26105::DEBUG::2012-12-02 14:06:49,485::task::568::TaskManager.Task::(_updateState) Task=`aa899b5b-c9a8-42c9-ad0f-3a8d4ac9f580`::moving from state init -> state preparing >Thread-26105::INFO::2012-12-02 14:06:49,485::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26105::INFO::2012-12-02 14:06:49,485::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26105::DEBUG::2012-12-02 14:06:49,485::task::1151::TaskManager.Task::(prepare) Task=`aa899b5b-c9a8-42c9-ad0f-3a8d4ac9f580`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26105::DEBUG::2012-12-02 14:06:49,485::task::568::TaskManager.Task::(_updateState) Task=`aa899b5b-c9a8-42c9-ad0f-3a8d4ac9f580`::moving from state preparing -> state finished >Thread-26105::DEBUG::2012-12-02 14:06:49,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26105::DEBUG::2012-12-02 14:06:49,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26105::DEBUG::2012-12-02 14:06:49,485::task::957::TaskManager.Task::(_decref) Task=`aa899b5b-c9a8-42c9-ad0f-3a8d4ac9f580`::ref 0 aborting False >Thread-26106::DEBUG::2012-12-02 14:06:49,488::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26106::DEBUG::2012-12-02 14:06:49,488::task::568::TaskManager.Task::(_updateState) Task=`ac08be40-8408-4bed-927d-cb1291403cd8`::moving from state init -> state preparing >Thread-26106::INFO::2012-12-02 14:06:49,488::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26106::DEBUG::2012-12-02 14:06:49,488::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7278cf13-807b-4aa7-b13c-770483fa9c60`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26106::DEBUG::2012-12-02 14:06:49,489::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26106::DEBUG::2012-12-02 14:06:49,489::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26106::DEBUG::2012-12-02 14:06:49,489::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7278cf13-807b-4aa7-b13c-770483fa9c60`::Granted request >Thread-26106::DEBUG::2012-12-02 14:06:49,489::task::794::TaskManager.Task::(resourceAcquired) Task=`ac08be40-8408-4bed-927d-cb1291403cd8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26106::DEBUG::2012-12-02 14:06:49,489::task::957::TaskManager.Task::(_decref) Task=`ac08be40-8408-4bed-927d-cb1291403cd8`::ref 1 aborting False >Thread-26106::INFO::2012-12-02 14:06:49,493::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-26106::DEBUG::2012-12-02 14:06:49,493::task::1151::TaskManager.Task::(prepare) Task=`ac08be40-8408-4bed-927d-cb1291403cd8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-26106::DEBUG::2012-12-02 14:06:49,494::task::568::TaskManager.Task::(_updateState) Task=`ac08be40-8408-4bed-927d-cb1291403cd8`::moving from state preparing -> state finished >Thread-26106::DEBUG::2012-12-02 14:06:49,494::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26106::DEBUG::2012-12-02 14:06:49,494::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26106::DEBUG::2012-12-02 14:06:49,494::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26106::DEBUG::2012-12-02 14:06:49,494::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26106::DEBUG::2012-12-02 14:06:49,494::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26106::DEBUG::2012-12-02 14:06:49,494::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26106::DEBUG::2012-12-02 14:06:49,494::task::957::TaskManager.Task::(_decref) Task=`ac08be40-8408-4bed-927d-cb1291403cd8`::ref 0 aborting False >Thread-26110::DEBUG::2012-12-02 14:06:55,927::task::568::TaskManager.Task::(_updateState) Task=`3d30dd7f-d5f9-4de5-b84d-07783e7c795f`::moving from state init -> state preparing >Thread-26110::INFO::2012-12-02 14:06:55,931::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26110::INFO::2012-12-02 14:06:55,931::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00852799415588', 'lastCheck': 1354450014.5191319, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00410389900208', 'lastCheck': 1354450009.405597, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00197815895081', 'lastCheck': 1354450012.7395501, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0102360248566', 'lastCheck': 1354450009.4185081, 'code': 0, 'valid': True}} >Thread-26110::DEBUG::2012-12-02 14:06:55,931::task::1151::TaskManager.Task::(prepare) Task=`3d30dd7f-d5f9-4de5-b84d-07783e7c795f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00852799415588', 'lastCheck': 1354450014.5191319, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00410389900208', 'lastCheck': 1354450009.405597, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00197815895081', 'lastCheck': 1354450012.7395501, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0102360248566', 'lastCheck': 1354450009.4185081, 'code': 0, 'valid': True}} >Thread-26110::DEBUG::2012-12-02 14:06:55,931::task::568::TaskManager.Task::(_updateState) Task=`3d30dd7f-d5f9-4de5-b84d-07783e7c795f`::moving from state preparing -> state finished >Thread-26110::DEBUG::2012-12-02 14:06:55,931::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26110::DEBUG::2012-12-02 14:06:55,931::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26110::DEBUG::2012-12-02 14:06:55,931::task::957::TaskManager.Task::(_decref) Task=`3d30dd7f-d5f9-4de5-b84d-07783e7c795f`::ref 0 aborting False >Thread-26113::DEBUG::2012-12-02 14:06:59,641::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26113::DEBUG::2012-12-02 14:06:59,642::task::568::TaskManager.Task::(_updateState) Task=`d40ae361-5240-4aac-ac73-0a171f0fa36f`::moving from state init -> state preparing >Thread-26113::INFO::2012-12-02 14:06:59,642::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26113::INFO::2012-12-02 14:06:59,642::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26113::DEBUG::2012-12-02 14:06:59,642::task::1151::TaskManager.Task::(prepare) Task=`d40ae361-5240-4aac-ac73-0a171f0fa36f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26113::DEBUG::2012-12-02 14:06:59,642::task::568::TaskManager.Task::(_updateState) Task=`d40ae361-5240-4aac-ac73-0a171f0fa36f`::moving from state preparing -> state finished >Thread-26113::DEBUG::2012-12-02 14:06:59,642::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26113::DEBUG::2012-12-02 14:06:59,643::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26113::DEBUG::2012-12-02 14:06:59,643::task::957::TaskManager.Task::(_decref) Task=`d40ae361-5240-4aac-ac73-0a171f0fa36f`::ref 0 aborting False >Thread-26114::DEBUG::2012-12-02 14:06:59,645::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26114::DEBUG::2012-12-02 14:06:59,645::task::568::TaskManager.Task::(_updateState) Task=`6936c69e-da6a-4f0d-abd9-9c1d1b1fef35`::moving from state init -> state preparing >Thread-26114::INFO::2012-12-02 14:06:59,646::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26114::DEBUG::2012-12-02 14:06:59,646::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ab1a7915-b2c4-439c-b978-a17599f0176b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26114::DEBUG::2012-12-02 14:06:59,646::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26114::DEBUG::2012-12-02 14:06:59,646::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26114::DEBUG::2012-12-02 14:06:59,646::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ab1a7915-b2c4-439c-b978-a17599f0176b`::Granted request >Thread-26114::DEBUG::2012-12-02 14:06:59,647::task::794::TaskManager.Task::(resourceAcquired) Task=`6936c69e-da6a-4f0d-abd9-9c1d1b1fef35`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26114::DEBUG::2012-12-02 14:06:59,647::task::957::TaskManager.Task::(_decref) Task=`6936c69e-da6a-4f0d-abd9-9c1d1b1fef35`::ref 1 aborting False >Thread-26114::INFO::2012-12-02 14:06:59,653::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-26114::DEBUG::2012-12-02 14:06:59,653::task::1151::TaskManager.Task::(prepare) Task=`6936c69e-da6a-4f0d-abd9-9c1d1b1fef35`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-26114::DEBUG::2012-12-02 14:06:59,654::task::568::TaskManager.Task::(_updateState) Task=`6936c69e-da6a-4f0d-abd9-9c1d1b1fef35`::moving from state preparing -> state finished >Thread-26114::DEBUG::2012-12-02 14:06:59,654::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26114::DEBUG::2012-12-02 14:06:59,654::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26114::DEBUG::2012-12-02 14:06:59,654::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26114::DEBUG::2012-12-02 14:06:59,654::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26114::DEBUG::2012-12-02 14:06:59,654::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26114::DEBUG::2012-12-02 14:06:59,654::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26114::DEBUG::2012-12-02 14:06:59,655::task::957::TaskManager.Task::(_decref) Task=`6936c69e-da6a-4f0d-abd9-9c1d1b1fef35`::ref 0 aborting False >Thread-26118::DEBUG::2012-12-02 14:07:06,047::task::568::TaskManager.Task::(_updateState) Task=`7a4f9df5-0976-40b2-a93c-dcde3df43afd`::moving from state init -> state preparing >Thread-26118::INFO::2012-12-02 14:07:06,047::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26118::INFO::2012-12-02 14:07:06,047::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007807970047', 'lastCheck': 1354450024.5368271, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00227689743042', 'lastCheck': 1354450019.4187, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00205516815186', 'lastCheck': 1354450022.7509761, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00793600082397', 'lastCheck': 1354450019.435348, 'code': 0, 'valid': True}} >Thread-26118::DEBUG::2012-12-02 14:07:06,048::task::1151::TaskManager.Task::(prepare) Task=`7a4f9df5-0976-40b2-a93c-dcde3df43afd`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007807970047', 'lastCheck': 1354450024.5368271, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00227689743042', 'lastCheck': 1354450019.4187, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00205516815186', 'lastCheck': 1354450022.7509761, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00793600082397', 'lastCheck': 1354450019.435348, 'code': 0, 'valid': True}} >Thread-26118::DEBUG::2012-12-02 14:07:06,048::task::568::TaskManager.Task::(_updateState) Task=`7a4f9df5-0976-40b2-a93c-dcde3df43afd`::moving from state preparing -> state finished >Thread-26118::DEBUG::2012-12-02 14:07:06,048::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26118::DEBUG::2012-12-02 14:07:06,048::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26118::DEBUG::2012-12-02 14:07:06,048::task::957::TaskManager.Task::(_decref) Task=`7a4f9df5-0976-40b2-a93c-dcde3df43afd`::ref 0 aborting False >Thread-26121::DEBUG::2012-12-02 14:07:09,809::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26121::DEBUG::2012-12-02 14:07:09,810::task::568::TaskManager.Task::(_updateState) Task=`5738f6f0-8085-4b96-936b-cff1a420ff2b`::moving from state init -> state preparing >Thread-26121::INFO::2012-12-02 14:07:09,810::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26121::INFO::2012-12-02 14:07:09,810::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26121::DEBUG::2012-12-02 14:07:09,810::task::1151::TaskManager.Task::(prepare) Task=`5738f6f0-8085-4b96-936b-cff1a420ff2b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26121::DEBUG::2012-12-02 14:07:09,810::task::568::TaskManager.Task::(_updateState) Task=`5738f6f0-8085-4b96-936b-cff1a420ff2b`::moving from state preparing -> state finished >Thread-26121::DEBUG::2012-12-02 14:07:09,810::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26121::DEBUG::2012-12-02 14:07:09,810::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26121::DEBUG::2012-12-02 14:07:09,811::task::957::TaskManager.Task::(_decref) Task=`5738f6f0-8085-4b96-936b-cff1a420ff2b`::ref 0 aborting False >Thread-26122::DEBUG::2012-12-02 14:07:09,813::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26122::DEBUG::2012-12-02 14:07:09,813::task::568::TaskManager.Task::(_updateState) Task=`a452894b-3f48-4c90-b8df-32eeb07dcb8a`::moving from state init -> state preparing >Thread-26122::INFO::2012-12-02 14:07:09,814::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26122::DEBUG::2012-12-02 14:07:09,814::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`867de869-3749-4409-b3c8-25e0c86b2653`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26122::DEBUG::2012-12-02 14:07:09,814::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26122::DEBUG::2012-12-02 14:07:09,814::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26122::DEBUG::2012-12-02 14:07:09,814::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`867de869-3749-4409-b3c8-25e0c86b2653`::Granted request >Thread-26122::DEBUG::2012-12-02 14:07:09,815::task::794::TaskManager.Task::(resourceAcquired) Task=`a452894b-3f48-4c90-b8df-32eeb07dcb8a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26122::DEBUG::2012-12-02 14:07:09,815::task::957::TaskManager.Task::(_decref) Task=`a452894b-3f48-4c90-b8df-32eeb07dcb8a`::ref 1 aborting False >Thread-26122::INFO::2012-12-02 14:07:09,819::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-26122::DEBUG::2012-12-02 14:07:09,819::task::1151::TaskManager.Task::(prepare) Task=`a452894b-3f48-4c90-b8df-32eeb07dcb8a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-26122::DEBUG::2012-12-02 14:07:09,819::task::568::TaskManager.Task::(_updateState) Task=`a452894b-3f48-4c90-b8df-32eeb07dcb8a`::moving from state preparing -> state finished >Thread-26122::DEBUG::2012-12-02 14:07:09,820::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26122::DEBUG::2012-12-02 14:07:09,820::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26122::DEBUG::2012-12-02 14:07:09,820::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26122::DEBUG::2012-12-02 14:07:09,820::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26122::DEBUG::2012-12-02 14:07:09,820::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26122::DEBUG::2012-12-02 14:07:09,820::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26122::DEBUG::2012-12-02 14:07:09,820::task::957::TaskManager.Task::(_decref) Task=`a452894b-3f48-4c90-b8df-32eeb07dcb8a`::ref 0 aborting False >Thread-26126::DEBUG::2012-12-02 14:07:16,137::task::568::TaskManager.Task::(_updateState) Task=`edf5cdf7-e571-41b2-93a5-a58e82067064`::moving from state init -> state preparing >Thread-26126::INFO::2012-12-02 14:07:16,137::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26126::INFO::2012-12-02 14:07:16,137::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00829100608826', 'lastCheck': 1354450034.5560589, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00400900840759', 'lastCheck': 1354450029.4427781, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00288319587708', 'lastCheck': 1354450032.7642829, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00697088241577', 'lastCheck': 1354450029.4527321, 'code': 0, 'valid': True}} >Thread-26126::DEBUG::2012-12-02 14:07:16,137::task::1151::TaskManager.Task::(prepare) Task=`edf5cdf7-e571-41b2-93a5-a58e82067064`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00829100608826', 'lastCheck': 1354450034.5560589, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00400900840759', 'lastCheck': 1354450029.4427781, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00288319587708', 'lastCheck': 1354450032.7642829, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00697088241577', 'lastCheck': 1354450029.4527321, 'code': 0, 'valid': True}} >Thread-26126::DEBUG::2012-12-02 14:07:16,138::task::568::TaskManager.Task::(_updateState) Task=`edf5cdf7-e571-41b2-93a5-a58e82067064`::moving from state preparing -> state finished >Thread-26126::DEBUG::2012-12-02 14:07:16,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26126::DEBUG::2012-12-02 14:07:16,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26126::DEBUG::2012-12-02 14:07:16,138::task::957::TaskManager.Task::(_decref) Task=`edf5cdf7-e571-41b2-93a5-a58e82067064`::ref 0 aborting False >Thread-26129::DEBUG::2012-12-02 14:07:19,973::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26129::DEBUG::2012-12-02 14:07:19,973::task::568::TaskManager.Task::(_updateState) Task=`e6b19bbf-d470-4018-a932-45ded4003840`::moving from state init -> state preparing >Thread-26129::INFO::2012-12-02 14:07:19,973::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26129::INFO::2012-12-02 14:07:19,973::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26129::DEBUG::2012-12-02 14:07:19,974::task::1151::TaskManager.Task::(prepare) Task=`e6b19bbf-d470-4018-a932-45ded4003840`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26129::DEBUG::2012-12-02 14:07:19,974::task::568::TaskManager.Task::(_updateState) Task=`e6b19bbf-d470-4018-a932-45ded4003840`::moving from state preparing -> state finished >Thread-26129::DEBUG::2012-12-02 14:07:19,974::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26129::DEBUG::2012-12-02 14:07:19,974::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26129::DEBUG::2012-12-02 14:07:19,974::task::957::TaskManager.Task::(_decref) Task=`e6b19bbf-d470-4018-a932-45ded4003840`::ref 0 aborting False >Thread-26130::DEBUG::2012-12-02 14:07:19,978::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26130::DEBUG::2012-12-02 14:07:19,978::task::568::TaskManager.Task::(_updateState) Task=`1caadbe7-39af-4d9c-a890-be114c4ac2cc`::moving from state init -> state preparing >Thread-26130::INFO::2012-12-02 14:07:19,978::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26130::DEBUG::2012-12-02 14:07:19,979::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`936eef0f-1dff-4726-a4d2-9144c0041097`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26130::DEBUG::2012-12-02 14:07:19,979::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26130::DEBUG::2012-12-02 14:07:19,979::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26130::DEBUG::2012-12-02 14:07:19,979::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`936eef0f-1dff-4726-a4d2-9144c0041097`::Granted request >Thread-26130::DEBUG::2012-12-02 14:07:19,979::task::794::TaskManager.Task::(resourceAcquired) Task=`1caadbe7-39af-4d9c-a890-be114c4ac2cc`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26130::DEBUG::2012-12-02 14:07:19,979::task::957::TaskManager.Task::(_decref) Task=`1caadbe7-39af-4d9c-a890-be114c4ac2cc`::ref 1 aborting False >Thread-26130::INFO::2012-12-02 14:07:19,986::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26130::DEBUG::2012-12-02 14:07:19,986::task::1151::TaskManager.Task::(prepare) Task=`1caadbe7-39af-4d9c-a890-be114c4ac2cc`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26130::DEBUG::2012-12-02 14:07:19,986::task::568::TaskManager.Task::(_updateState) Task=`1caadbe7-39af-4d9c-a890-be114c4ac2cc`::moving from state preparing -> state finished >Thread-26130::DEBUG::2012-12-02 14:07:19,986::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26130::DEBUG::2012-12-02 14:07:19,986::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26130::DEBUG::2012-12-02 14:07:19,986::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26130::DEBUG::2012-12-02 14:07:19,986::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26130::DEBUG::2012-12-02 14:07:19,986::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26130::DEBUG::2012-12-02 14:07:19,986::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26130::DEBUG::2012-12-02 14:07:19,987::task::957::TaskManager.Task::(_decref) Task=`1caadbe7-39af-4d9c-a890-be114c4ac2cc`::ref 0 aborting False >Thread-26134::DEBUG::2012-12-02 14:07:26,278::task::568::TaskManager.Task::(_updateState) Task=`4ddacf40-bb29-439c-be8d-cd5f910bb168`::moving from state init -> state preparing >Thread-26134::INFO::2012-12-02 14:07:26,278::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26134::INFO::2012-12-02 14:07:26,278::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00335907936096', 'lastCheck': 1354450044.5676479, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00176596641541', 'lastCheck': 1354450039.4649479, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018789768219', 'lastCheck': 1354450042.774204, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00690293312073', 'lastCheck': 1354450039.471132, 'code': 0, 'valid': True}} >Thread-26134::DEBUG::2012-12-02 14:07:26,278::task::1151::TaskManager.Task::(prepare) Task=`4ddacf40-bb29-439c-be8d-cd5f910bb168`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00335907936096', 'lastCheck': 1354450044.5676479, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00176596641541', 'lastCheck': 1354450039.4649479, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018789768219', 'lastCheck': 1354450042.774204, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00690293312073', 'lastCheck': 1354450039.471132, 'code': 0, 'valid': True}} >Thread-26134::DEBUG::2012-12-02 14:07:26,278::task::568::TaskManager.Task::(_updateState) Task=`4ddacf40-bb29-439c-be8d-cd5f910bb168`::moving from state preparing -> state finished >Thread-26134::DEBUG::2012-12-02 14:07:26,279::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26134::DEBUG::2012-12-02 14:07:26,279::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26134::DEBUG::2012-12-02 14:07:26,279::task::957::TaskManager.Task::(_decref) Task=`4ddacf40-bb29-439c-be8d-cd5f910bb168`::ref 0 aborting False >Thread-26137::DEBUG::2012-12-02 14:07:30,144::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26137::DEBUG::2012-12-02 14:07:30,144::task::568::TaskManager.Task::(_updateState) Task=`7ffbec81-1c0d-410e-ae10-6369d658a17a`::moving from state init -> state preparing >Thread-26137::INFO::2012-12-02 14:07:30,144::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26137::INFO::2012-12-02 14:07:30,144::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26137::DEBUG::2012-12-02 14:07:30,145::task::1151::TaskManager.Task::(prepare) Task=`7ffbec81-1c0d-410e-ae10-6369d658a17a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26137::DEBUG::2012-12-02 14:07:30,145::task::568::TaskManager.Task::(_updateState) Task=`7ffbec81-1c0d-410e-ae10-6369d658a17a`::moving from state preparing -> state finished >Thread-26137::DEBUG::2012-12-02 14:07:30,145::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26137::DEBUG::2012-12-02 14:07:30,145::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26137::DEBUG::2012-12-02 14:07:30,145::task::957::TaskManager.Task::(_decref) Task=`7ffbec81-1c0d-410e-ae10-6369d658a17a`::ref 0 aborting False >Thread-26138::DEBUG::2012-12-02 14:07:30,148::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26138::DEBUG::2012-12-02 14:07:30,148::task::568::TaskManager.Task::(_updateState) Task=`058d24bc-e776-48fb-98e8-71fafbe80058`::moving from state init -> state preparing >Thread-26138::INFO::2012-12-02 14:07:30,148::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26138::DEBUG::2012-12-02 14:07:30,148::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cad4ffbf-a956-4f4e-93a4-ec7a6a78c4ad`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26138::DEBUG::2012-12-02 14:07:30,148::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26138::DEBUG::2012-12-02 14:07:30,149::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26138::DEBUG::2012-12-02 14:07:30,149::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cad4ffbf-a956-4f4e-93a4-ec7a6a78c4ad`::Granted request >Thread-26138::DEBUG::2012-12-02 14:07:30,149::task::794::TaskManager.Task::(resourceAcquired) Task=`058d24bc-e776-48fb-98e8-71fafbe80058`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26138::DEBUG::2012-12-02 14:07:30,149::task::957::TaskManager.Task::(_decref) Task=`058d24bc-e776-48fb-98e8-71fafbe80058`::ref 1 aborting False >Thread-26138::INFO::2012-12-02 14:07:30,155::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26138::DEBUG::2012-12-02 14:07:30,155::task::1151::TaskManager.Task::(prepare) Task=`058d24bc-e776-48fb-98e8-71fafbe80058`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26138::DEBUG::2012-12-02 14:07:30,156::task::568::TaskManager.Task::(_updateState) Task=`058d24bc-e776-48fb-98e8-71fafbe80058`::moving from state preparing -> state finished >Thread-26138::DEBUG::2012-12-02 14:07:30,156::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26138::DEBUG::2012-12-02 14:07:30,156::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26138::DEBUG::2012-12-02 14:07:30,156::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26138::DEBUG::2012-12-02 14:07:30,156::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26138::DEBUG::2012-12-02 14:07:30,156::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26138::DEBUG::2012-12-02 14:07:30,156::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26138::DEBUG::2012-12-02 14:07:30,156::task::957::TaskManager.Task::(_decref) Task=`058d24bc-e776-48fb-98e8-71fafbe80058`::ref 0 aborting False >Thread-26142::DEBUG::2012-12-02 14:07:36,421::task::568::TaskManager.Task::(_updateState) Task=`74b897bf-a54c-4da8-8959-756fd6e0c517`::moving from state init -> state preparing >Thread-26142::INFO::2012-12-02 14:07:36,421::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26142::INFO::2012-12-02 14:07:36,421::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00878190994263', 'lastCheck': 1354450054.588913, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0045530796051', 'lastCheck': 1354450049.4813559, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00284004211426', 'lastCheck': 1354450052.786253, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00724792480469', 'lastCheck': 1354450049.4892509, 'code': 0, 'valid': True}} >Thread-26142::DEBUG::2012-12-02 14:07:36,421::task::1151::TaskManager.Task::(prepare) Task=`74b897bf-a54c-4da8-8959-756fd6e0c517`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00878190994263', 'lastCheck': 1354450054.588913, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0045530796051', 'lastCheck': 1354450049.4813559, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00284004211426', 'lastCheck': 1354450052.786253, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00724792480469', 'lastCheck': 1354450049.4892509, 'code': 0, 'valid': True}} >Thread-26142::DEBUG::2012-12-02 14:07:36,421::task::568::TaskManager.Task::(_updateState) Task=`74b897bf-a54c-4da8-8959-756fd6e0c517`::moving from state preparing -> state finished >Thread-26142::DEBUG::2012-12-02 14:07:36,421::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26142::DEBUG::2012-12-02 14:07:36,422::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26142::DEBUG::2012-12-02 14:07:36,422::task::957::TaskManager.Task::(_decref) Task=`74b897bf-a54c-4da8-8959-756fd6e0c517`::ref 0 aborting False >Thread-26145::DEBUG::2012-12-02 14:07:40,320::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26145::DEBUG::2012-12-02 14:07:40,320::task::568::TaskManager.Task::(_updateState) Task=`69716682-b8a8-4f61-98d5-4a53e9f2881c`::moving from state init -> state preparing >Thread-26145::INFO::2012-12-02 14:07:40,321::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26145::INFO::2012-12-02 14:07:40,321::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26145::DEBUG::2012-12-02 14:07:40,321::task::1151::TaskManager.Task::(prepare) Task=`69716682-b8a8-4f61-98d5-4a53e9f2881c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26145::DEBUG::2012-12-02 14:07:40,321::task::568::TaskManager.Task::(_updateState) Task=`69716682-b8a8-4f61-98d5-4a53e9f2881c`::moving from state preparing -> state finished >Thread-26145::DEBUG::2012-12-02 14:07:40,321::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26145::DEBUG::2012-12-02 14:07:40,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26145::DEBUG::2012-12-02 14:07:40,321::task::957::TaskManager.Task::(_decref) Task=`69716682-b8a8-4f61-98d5-4a53e9f2881c`::ref 0 aborting False >Thread-26146::DEBUG::2012-12-02 14:07:40,324::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26146::DEBUG::2012-12-02 14:07:40,324::task::568::TaskManager.Task::(_updateState) Task=`89c904b3-1274-4cfc-aeac-f07e1de8e5a2`::moving from state init -> state preparing >Thread-26146::INFO::2012-12-02 14:07:40,324::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26146::DEBUG::2012-12-02 14:07:40,325::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b8109a7e-c050-47f3-b098-374a42368e3f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26146::DEBUG::2012-12-02 14:07:40,325::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26146::DEBUG::2012-12-02 14:07:40,325::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26146::DEBUG::2012-12-02 14:07:40,325::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b8109a7e-c050-47f3-b098-374a42368e3f`::Granted request >Thread-26146::DEBUG::2012-12-02 14:07:40,325::task::794::TaskManager.Task::(resourceAcquired) Task=`89c904b3-1274-4cfc-aeac-f07e1de8e5a2`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26146::DEBUG::2012-12-02 14:07:40,325::task::957::TaskManager.Task::(_decref) Task=`89c904b3-1274-4cfc-aeac-f07e1de8e5a2`::ref 1 aborting False >Thread-26146::INFO::2012-12-02 14:07:40,330::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26146::DEBUG::2012-12-02 14:07:40,330::task::1151::TaskManager.Task::(prepare) Task=`89c904b3-1274-4cfc-aeac-f07e1de8e5a2`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26146::DEBUG::2012-12-02 14:07:40,330::task::568::TaskManager.Task::(_updateState) Task=`89c904b3-1274-4cfc-aeac-f07e1de8e5a2`::moving from state preparing -> state finished >Thread-26146::DEBUG::2012-12-02 14:07:40,330::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26146::DEBUG::2012-12-02 14:07:40,330::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26146::DEBUG::2012-12-02 14:07:40,330::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26146::DEBUG::2012-12-02 14:07:40,330::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26146::DEBUG::2012-12-02 14:07:40,330::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26146::DEBUG::2012-12-02 14:07:40,330::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26146::DEBUG::2012-12-02 14:07:40,330::task::957::TaskManager.Task::(_decref) Task=`89c904b3-1274-4cfc-aeac-f07e1de8e5a2`::ref 0 aborting False >Thread-26150::DEBUG::2012-12-02 14:07:46,514::task::568::TaskManager.Task::(_updateState) Task=`50cedc7c-826a-4993-bd40-0ab349f95b69`::moving from state init -> state preparing >Thread-26150::INFO::2012-12-02 14:07:46,514::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26150::INFO::2012-12-02 14:07:46,514::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00897288322449', 'lastCheck': 1354450064.6073151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0059769153595', 'lastCheck': 1354450059.500623, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0023889541626', 'lastCheck': 1354450062.7979219, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00851893424988', 'lastCheck': 1354450059.510607, 'code': 0, 'valid': True}} >Thread-26150::DEBUG::2012-12-02 14:07:46,515::task::1151::TaskManager.Task::(prepare) Task=`50cedc7c-826a-4993-bd40-0ab349f95b69`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00897288322449', 'lastCheck': 1354450064.6073151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0059769153595', 'lastCheck': 1354450059.500623, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0023889541626', 'lastCheck': 1354450062.7979219, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00851893424988', 'lastCheck': 1354450059.510607, 'code': 0, 'valid': True}} >Thread-26150::DEBUG::2012-12-02 14:07:46,515::task::568::TaskManager.Task::(_updateState) Task=`50cedc7c-826a-4993-bd40-0ab349f95b69`::moving from state preparing -> state finished >Thread-26150::DEBUG::2012-12-02 14:07:46,515::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26150::DEBUG::2012-12-02 14:07:46,515::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26150::DEBUG::2012-12-02 14:07:46,515::task::957::TaskManager.Task::(_decref) Task=`50cedc7c-826a-4993-bd40-0ab349f95b69`::ref 0 aborting False >Thread-26153::DEBUG::2012-12-02 14:07:50,476::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26153::DEBUG::2012-12-02 14:07:50,476::task::568::TaskManager.Task::(_updateState) Task=`c9187155-b689-4007-ba91-b6757bdeead1`::moving from state init -> state preparing >Thread-26153::INFO::2012-12-02 14:07:50,477::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26153::INFO::2012-12-02 14:07:50,477::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26153::DEBUG::2012-12-02 14:07:50,477::task::1151::TaskManager.Task::(prepare) Task=`c9187155-b689-4007-ba91-b6757bdeead1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26153::DEBUG::2012-12-02 14:07:50,477::task::568::TaskManager.Task::(_updateState) Task=`c9187155-b689-4007-ba91-b6757bdeead1`::moving from state preparing -> state finished >Thread-26153::DEBUG::2012-12-02 14:07:50,477::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26153::DEBUG::2012-12-02 14:07:50,477::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26153::DEBUG::2012-12-02 14:07:50,477::task::957::TaskManager.Task::(_decref) Task=`c9187155-b689-4007-ba91-b6757bdeead1`::ref 0 aborting False >Thread-26154::DEBUG::2012-12-02 14:07:50,480::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26154::DEBUG::2012-12-02 14:07:50,480::task::568::TaskManager.Task::(_updateState) Task=`772cacc5-fcb7-48c9-97bc-2a61e9f5145c`::moving from state init -> state preparing >Thread-26154::INFO::2012-12-02 14:07:50,480::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26154::DEBUG::2012-12-02 14:07:50,481::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`070c0388-7d09-42ca-95ea-ec7168d81d88`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26154::DEBUG::2012-12-02 14:07:50,481::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26154::DEBUG::2012-12-02 14:07:50,481::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26154::DEBUG::2012-12-02 14:07:50,481::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`070c0388-7d09-42ca-95ea-ec7168d81d88`::Granted request >Thread-26154::DEBUG::2012-12-02 14:07:50,481::task::794::TaskManager.Task::(resourceAcquired) Task=`772cacc5-fcb7-48c9-97bc-2a61e9f5145c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26154::DEBUG::2012-12-02 14:07:50,481::task::957::TaskManager.Task::(_decref) Task=`772cacc5-fcb7-48c9-97bc-2a61e9f5145c`::ref 1 aborting False >Thread-26154::INFO::2012-12-02 14:07:50,487::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26154::DEBUG::2012-12-02 14:07:50,487::task::1151::TaskManager.Task::(prepare) Task=`772cacc5-fcb7-48c9-97bc-2a61e9f5145c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26154::DEBUG::2012-12-02 14:07:50,487::task::568::TaskManager.Task::(_updateState) Task=`772cacc5-fcb7-48c9-97bc-2a61e9f5145c`::moving from state preparing -> state finished >Thread-26154::DEBUG::2012-12-02 14:07:50,488::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26154::DEBUG::2012-12-02 14:07:50,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26154::DEBUG::2012-12-02 14:07:50,488::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26154::DEBUG::2012-12-02 14:07:50,488::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26154::DEBUG::2012-12-02 14:07:50,488::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26154::DEBUG::2012-12-02 14:07:50,488::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26154::DEBUG::2012-12-02 14:07:50,488::task::957::TaskManager.Task::(_decref) Task=`772cacc5-fcb7-48c9-97bc-2a61e9f5145c`::ref 0 aborting False >Thread-26158::DEBUG::2012-12-02 14:07:56,609::task::568::TaskManager.Task::(_updateState) Task=`fdea05b2-7f1e-4c6c-8ad1-1e6161516293`::moving from state init -> state preparing >Thread-26158::INFO::2012-12-02 14:07:56,610::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26158::INFO::2012-12-02 14:07:56,610::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00831198692322', 'lastCheck': 1354450074.6254971, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00269293785095', 'lastCheck': 1354450069.5176549, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022439956665', 'lastCheck': 1354450072.8099, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00731801986694', 'lastCheck': 1354450069.5273621, 'code': 0, 'valid': True}} >Thread-26158::DEBUG::2012-12-02 14:07:56,610::task::1151::TaskManager.Task::(prepare) Task=`fdea05b2-7f1e-4c6c-8ad1-1e6161516293`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00831198692322', 'lastCheck': 1354450074.6254971, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00269293785095', 'lastCheck': 1354450069.5176549, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022439956665', 'lastCheck': 1354450072.8099, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00731801986694', 'lastCheck': 1354450069.5273621, 'code': 0, 'valid': True}} >Thread-26158::DEBUG::2012-12-02 14:07:56,610::task::568::TaskManager.Task::(_updateState) Task=`fdea05b2-7f1e-4c6c-8ad1-1e6161516293`::moving from state preparing -> state finished >Thread-26158::DEBUG::2012-12-02 14:07:56,610::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26158::DEBUG::2012-12-02 14:07:56,610::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26158::DEBUG::2012-12-02 14:07:56,611::task::957::TaskManager.Task::(_decref) Task=`fdea05b2-7f1e-4c6c-8ad1-1e6161516293`::ref 0 aborting False >Thread-26161::DEBUG::2012-12-02 14:08:00,638::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26161::DEBUG::2012-12-02 14:08:00,639::task::568::TaskManager.Task::(_updateState) Task=`12550ac5-01fa-4522-aabc-c71992d716a8`::moving from state init -> state preparing >Thread-26161::INFO::2012-12-02 14:08:00,639::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26161::INFO::2012-12-02 14:08:00,639::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26161::DEBUG::2012-12-02 14:08:00,639::task::1151::TaskManager.Task::(prepare) Task=`12550ac5-01fa-4522-aabc-c71992d716a8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26161::DEBUG::2012-12-02 14:08:00,639::task::568::TaskManager.Task::(_updateState) Task=`12550ac5-01fa-4522-aabc-c71992d716a8`::moving from state preparing -> state finished >Thread-26161::DEBUG::2012-12-02 14:08:00,639::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26161::DEBUG::2012-12-02 14:08:00,639::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26161::DEBUG::2012-12-02 14:08:00,640::task::957::TaskManager.Task::(_decref) Task=`12550ac5-01fa-4522-aabc-c71992d716a8`::ref 0 aborting False >Thread-26162::DEBUG::2012-12-02 14:08:00,642::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26162::DEBUG::2012-12-02 14:08:00,642::task::568::TaskManager.Task::(_updateState) Task=`10b0fee2-4f1b-4d31-8c34-7598ef4d9a93`::moving from state init -> state preparing >Thread-26162::INFO::2012-12-02 14:08:00,642::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26162::DEBUG::2012-12-02 14:08:00,643::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a0d0a77d-9e04-473d-8782-8cc7e56d612a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26162::DEBUG::2012-12-02 14:08:00,643::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26162::DEBUG::2012-12-02 14:08:00,643::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26162::DEBUG::2012-12-02 14:08:00,643::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a0d0a77d-9e04-473d-8782-8cc7e56d612a`::Granted request >Thread-26162::DEBUG::2012-12-02 14:08:00,643::task::794::TaskManager.Task::(resourceAcquired) Task=`10b0fee2-4f1b-4d31-8c34-7598ef4d9a93`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26162::DEBUG::2012-12-02 14:08:00,644::task::957::TaskManager.Task::(_decref) Task=`10b0fee2-4f1b-4d31-8c34-7598ef4d9a93`::ref 1 aborting False >Thread-26162::INFO::2012-12-02 14:08:00,649::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26162::DEBUG::2012-12-02 14:08:00,649::task::1151::TaskManager.Task::(prepare) Task=`10b0fee2-4f1b-4d31-8c34-7598ef4d9a93`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26162::DEBUG::2012-12-02 14:08:00,649::task::568::TaskManager.Task::(_updateState) Task=`10b0fee2-4f1b-4d31-8c34-7598ef4d9a93`::moving from state preparing -> state finished >Thread-26162::DEBUG::2012-12-02 14:08:00,649::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26162::DEBUG::2012-12-02 14:08:00,649::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26162::DEBUG::2012-12-02 14:08:00,649::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26162::DEBUG::2012-12-02 14:08:00,649::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26162::DEBUG::2012-12-02 14:08:00,649::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26162::DEBUG::2012-12-02 14:08:00,649::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26162::DEBUG::2012-12-02 14:08:00,650::task::957::TaskManager.Task::(_decref) Task=`10b0fee2-4f1b-4d31-8c34-7598ef4d9a93`::ref 0 aborting False >Thread-26166::DEBUG::2012-12-02 14:08:06,726::task::568::TaskManager.Task::(_updateState) Task=`f552cab1-1194-4c89-a111-7a0363d03128`::moving from state init -> state preparing >Thread-26166::INFO::2012-12-02 14:08:06,726::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26166::INFO::2012-12-02 14:08:06,726::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00801587104797', 'lastCheck': 1354450084.643301, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00286293029785', 'lastCheck': 1354450079.53526, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00240302085876', 'lastCheck': 1354450082.821146, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00674486160278', 'lastCheck': 1354450079.544286, 'code': 0, 'valid': True}} >Thread-26166::DEBUG::2012-12-02 14:08:06,726::task::1151::TaskManager.Task::(prepare) Task=`f552cab1-1194-4c89-a111-7a0363d03128`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00801587104797', 'lastCheck': 1354450084.643301, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00286293029785', 'lastCheck': 1354450079.53526, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00240302085876', 'lastCheck': 1354450082.821146, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00674486160278', 'lastCheck': 1354450079.544286, 'code': 0, 'valid': True}} >Thread-26166::DEBUG::2012-12-02 14:08:06,727::task::568::TaskManager.Task::(_updateState) Task=`f552cab1-1194-4c89-a111-7a0363d03128`::moving from state preparing -> state finished >Thread-26166::DEBUG::2012-12-02 14:08:06,727::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26166::DEBUG::2012-12-02 14:08:06,727::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26166::DEBUG::2012-12-02 14:08:06,727::task::957::TaskManager.Task::(_decref) Task=`f552cab1-1194-4c89-a111-7a0363d03128`::ref 0 aborting False >Thread-26169::DEBUG::2012-12-02 14:08:10,801::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26169::DEBUG::2012-12-02 14:08:10,801::task::568::TaskManager.Task::(_updateState) Task=`5406e567-c4aa-48d1-8c97-3d56fc239786`::moving from state init -> state preparing >Thread-26169::INFO::2012-12-02 14:08:10,801::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26169::INFO::2012-12-02 14:08:10,801::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26169::DEBUG::2012-12-02 14:08:10,802::task::1151::TaskManager.Task::(prepare) Task=`5406e567-c4aa-48d1-8c97-3d56fc239786`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26169::DEBUG::2012-12-02 14:08:10,802::task::568::TaskManager.Task::(_updateState) Task=`5406e567-c4aa-48d1-8c97-3d56fc239786`::moving from state preparing -> state finished >Thread-26169::DEBUG::2012-12-02 14:08:10,802::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26169::DEBUG::2012-12-02 14:08:10,802::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26169::DEBUG::2012-12-02 14:08:10,802::task::957::TaskManager.Task::(_decref) Task=`5406e567-c4aa-48d1-8c97-3d56fc239786`::ref 0 aborting False >Thread-26170::DEBUG::2012-12-02 14:08:10,805::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26170::DEBUG::2012-12-02 14:08:10,805::task::568::TaskManager.Task::(_updateState) Task=`7a93263d-02a4-45ba-8174-4153058e328f`::moving from state init -> state preparing >Thread-26170::INFO::2012-12-02 14:08:10,806::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26170::DEBUG::2012-12-02 14:08:10,806::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cda9a210-714b-448b-9742-3e5503561317`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26170::DEBUG::2012-12-02 14:08:10,807::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26170::DEBUG::2012-12-02 14:08:10,807::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26170::DEBUG::2012-12-02 14:08:10,807::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cda9a210-714b-448b-9742-3e5503561317`::Granted request >Thread-26170::DEBUG::2012-12-02 14:08:10,807::task::794::TaskManager.Task::(resourceAcquired) Task=`7a93263d-02a4-45ba-8174-4153058e328f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26170::DEBUG::2012-12-02 14:08:10,807::task::957::TaskManager.Task::(_decref) Task=`7a93263d-02a4-45ba-8174-4153058e328f`::ref 1 aborting False >Thread-26170::INFO::2012-12-02 14:08:10,814::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26170::DEBUG::2012-12-02 14:08:10,814::task::1151::TaskManager.Task::(prepare) Task=`7a93263d-02a4-45ba-8174-4153058e328f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26170::DEBUG::2012-12-02 14:08:10,814::task::568::TaskManager.Task::(_updateState) Task=`7a93263d-02a4-45ba-8174-4153058e328f`::moving from state preparing -> state finished >Thread-26170::DEBUG::2012-12-02 14:08:10,814::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26170::DEBUG::2012-12-02 14:08:10,814::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26170::DEBUG::2012-12-02 14:08:10,815::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26170::DEBUG::2012-12-02 14:08:10,815::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26170::DEBUG::2012-12-02 14:08:10,815::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26170::DEBUG::2012-12-02 14:08:10,815::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26170::DEBUG::2012-12-02 14:08:10,815::task::957::TaskManager.Task::(_decref) Task=`7a93263d-02a4-45ba-8174-4153058e328f`::ref 0 aborting False >Thread-26174::DEBUG::2012-12-02 14:08:16,822::task::568::TaskManager.Task::(_updateState) Task=`e631a3b8-e335-438c-a07e-899b83d391e5`::moving from state init -> state preparing >Thread-26174::INFO::2012-12-02 14:08:16,823::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26174::INFO::2012-12-02 14:08:16,823::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00815606117249', 'lastCheck': 1354450094.6606801, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00542998313904', 'lastCheck': 1354450089.5529981, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00300312042236', 'lastCheck': 1354450092.834378, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00690197944641', 'lastCheck': 1354450089.5607109, 'code': 0, 'valid': True}} >Thread-26174::DEBUG::2012-12-02 14:08:16,823::task::1151::TaskManager.Task::(prepare) Task=`e631a3b8-e335-438c-a07e-899b83d391e5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00815606117249', 'lastCheck': 1354450094.6606801, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00542998313904', 'lastCheck': 1354450089.5529981, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00300312042236', 'lastCheck': 1354450092.834378, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00690197944641', 'lastCheck': 1354450089.5607109, 'code': 0, 'valid': True}} >Thread-26174::DEBUG::2012-12-02 14:08:16,823::task::568::TaskManager.Task::(_updateState) Task=`e631a3b8-e335-438c-a07e-899b83d391e5`::moving from state preparing -> state finished >Thread-26174::DEBUG::2012-12-02 14:08:16,823::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26174::DEBUG::2012-12-02 14:08:16,823::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26174::DEBUG::2012-12-02 14:08:16,824::task::957::TaskManager.Task::(_decref) Task=`e631a3b8-e335-438c-a07e-899b83d391e5`::ref 0 aborting False >Thread-26178::DEBUG::2012-12-02 14:08:20,948::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26178::DEBUG::2012-12-02 14:08:20,948::task::568::TaskManager.Task::(_updateState) Task=`bf43d713-ad0e-4cd0-8ff7-b9e81a75dc9b`::moving from state init -> state preparing >Thread-26178::INFO::2012-12-02 14:08:20,949::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26178::INFO::2012-12-02 14:08:20,949::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26178::DEBUG::2012-12-02 14:08:20,949::task::1151::TaskManager.Task::(prepare) Task=`bf43d713-ad0e-4cd0-8ff7-b9e81a75dc9b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26178::DEBUG::2012-12-02 14:08:20,949::task::568::TaskManager.Task::(_updateState) Task=`bf43d713-ad0e-4cd0-8ff7-b9e81a75dc9b`::moving from state preparing -> state finished >Thread-26178::DEBUG::2012-12-02 14:08:20,949::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26178::DEBUG::2012-12-02 14:08:20,949::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26178::DEBUG::2012-12-02 14:08:20,949::task::957::TaskManager.Task::(_decref) Task=`bf43d713-ad0e-4cd0-8ff7-b9e81a75dc9b`::ref 0 aborting False >Thread-26179::DEBUG::2012-12-02 14:08:20,953::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26179::DEBUG::2012-12-02 14:08:20,953::task::568::TaskManager.Task::(_updateState) Task=`3e346b5f-20cf-4575-bb40-d99c96ad527d`::moving from state init -> state preparing >Thread-26179::INFO::2012-12-02 14:08:20,953::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26179::DEBUG::2012-12-02 14:08:20,954::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4952e558-48e5-4de0-ab96-ba10fd2134af`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26179::DEBUG::2012-12-02 14:08:20,954::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26179::DEBUG::2012-12-02 14:08:20,954::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26179::DEBUG::2012-12-02 14:08:20,955::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4952e558-48e5-4de0-ab96-ba10fd2134af`::Granted request >Thread-26179::DEBUG::2012-12-02 14:08:20,955::task::794::TaskManager.Task::(resourceAcquired) Task=`3e346b5f-20cf-4575-bb40-d99c96ad527d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26179::DEBUG::2012-12-02 14:08:20,955::task::957::TaskManager.Task::(_decref) Task=`3e346b5f-20cf-4575-bb40-d99c96ad527d`::ref 1 aborting False >Thread-26179::INFO::2012-12-02 14:08:20,959::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26179::DEBUG::2012-12-02 14:08:20,960::task::1151::TaskManager.Task::(prepare) Task=`3e346b5f-20cf-4575-bb40-d99c96ad527d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26179::DEBUG::2012-12-02 14:08:20,960::task::568::TaskManager.Task::(_updateState) Task=`3e346b5f-20cf-4575-bb40-d99c96ad527d`::moving from state preparing -> state finished >Thread-26179::DEBUG::2012-12-02 14:08:20,960::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26179::DEBUG::2012-12-02 14:08:20,960::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26179::DEBUG::2012-12-02 14:08:20,960::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26179::DEBUG::2012-12-02 14:08:20,960::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26179::DEBUG::2012-12-02 14:08:20,960::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26179::DEBUG::2012-12-02 14:08:20,960::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26179::DEBUG::2012-12-02 14:08:20,960::task::957::TaskManager.Task::(_decref) Task=`3e346b5f-20cf-4575-bb40-d99c96ad527d`::ref 0 aborting False >Thread-26182::DEBUG::2012-12-02 14:08:26,929::task::568::TaskManager.Task::(_updateState) Task=`e4da3ca8-b911-4748-bdbe-223a3df2f370`::moving from state init -> state preparing >Thread-26182::INFO::2012-12-02 14:08:26,929::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26182::INFO::2012-12-02 14:08:26,930::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00800108909607', 'lastCheck': 1354450104.6775501, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266599655151', 'lastCheck': 1354450099.5702219, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00130891799927', 'lastCheck': 1354450102.8421111, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00814199447632', 'lastCheck': 1354450099.5781269, 'code': 0, 'valid': True}} >Thread-26182::DEBUG::2012-12-02 14:08:26,930::task::1151::TaskManager.Task::(prepare) Task=`e4da3ca8-b911-4748-bdbe-223a3df2f370`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00800108909607', 'lastCheck': 1354450104.6775501, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266599655151', 'lastCheck': 1354450099.5702219, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00130891799927', 'lastCheck': 1354450102.8421111, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00814199447632', 'lastCheck': 1354450099.5781269, 'code': 0, 'valid': True}} >Thread-26182::DEBUG::2012-12-02 14:08:26,930::task::568::TaskManager.Task::(_updateState) Task=`e4da3ca8-b911-4748-bdbe-223a3df2f370`::moving from state preparing -> state finished >Thread-26182::DEBUG::2012-12-02 14:08:26,930::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26182::DEBUG::2012-12-02 14:08:26,930::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26182::DEBUG::2012-12-02 14:08:26,930::task::957::TaskManager.Task::(_decref) Task=`e4da3ca8-b911-4748-bdbe-223a3df2f370`::ref 0 aborting False >Thread-26186::DEBUG::2012-12-02 14:08:31,127::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26186::DEBUG::2012-12-02 14:08:31,127::task::568::TaskManager.Task::(_updateState) Task=`a034ef65-9006-4515-943e-ff61bdd4838e`::moving from state init -> state preparing >Thread-26186::INFO::2012-12-02 14:08:31,127::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26186::INFO::2012-12-02 14:08:31,128::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26186::DEBUG::2012-12-02 14:08:31,128::task::1151::TaskManager.Task::(prepare) Task=`a034ef65-9006-4515-943e-ff61bdd4838e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26186::DEBUG::2012-12-02 14:08:31,128::task::568::TaskManager.Task::(_updateState) Task=`a034ef65-9006-4515-943e-ff61bdd4838e`::moving from state preparing -> state finished >Thread-26186::DEBUG::2012-12-02 14:08:31,128::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26186::DEBUG::2012-12-02 14:08:31,128::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26186::DEBUG::2012-12-02 14:08:31,128::task::957::TaskManager.Task::(_decref) Task=`a034ef65-9006-4515-943e-ff61bdd4838e`::ref 0 aborting False >Thread-26187::DEBUG::2012-12-02 14:08:31,132::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26187::DEBUG::2012-12-02 14:08:31,132::task::568::TaskManager.Task::(_updateState) Task=`7c5253f1-a2b1-45b8-9ebf-7a1f79966d96`::moving from state init -> state preparing >Thread-26187::INFO::2012-12-02 14:08:31,132::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26187::DEBUG::2012-12-02 14:08:31,133::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b91379dd-3da3-425c-b351-cc4748cd3a0b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26187::DEBUG::2012-12-02 14:08:31,133::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26187::DEBUG::2012-12-02 14:08:31,133::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26187::DEBUG::2012-12-02 14:08:31,133::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b91379dd-3da3-425c-b351-cc4748cd3a0b`::Granted request >Thread-26187::DEBUG::2012-12-02 14:08:31,133::task::794::TaskManager.Task::(resourceAcquired) Task=`7c5253f1-a2b1-45b8-9ebf-7a1f79966d96`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26187::DEBUG::2012-12-02 14:08:31,133::task::957::TaskManager.Task::(_decref) Task=`7c5253f1-a2b1-45b8-9ebf-7a1f79966d96`::ref 1 aborting False >Thread-26187::INFO::2012-12-02 14:08:31,138::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26187::DEBUG::2012-12-02 14:08:31,138::task::1151::TaskManager.Task::(prepare) Task=`7c5253f1-a2b1-45b8-9ebf-7a1f79966d96`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26187::DEBUG::2012-12-02 14:08:31,139::task::568::TaskManager.Task::(_updateState) Task=`7c5253f1-a2b1-45b8-9ebf-7a1f79966d96`::moving from state preparing -> state finished >Thread-26187::DEBUG::2012-12-02 14:08:31,139::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26187::DEBUG::2012-12-02 14:08:31,139::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26187::DEBUG::2012-12-02 14:08:31,139::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26187::DEBUG::2012-12-02 14:08:31,139::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26187::DEBUG::2012-12-02 14:08:31,139::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26187::DEBUG::2012-12-02 14:08:31,139::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26187::DEBUG::2012-12-02 14:08:31,139::task::957::TaskManager.Task::(_decref) Task=`7c5253f1-a2b1-45b8-9ebf-7a1f79966d96`::ref 0 aborting False >Thread-26190::DEBUG::2012-12-02 14:08:37,052::task::568::TaskManager.Task::(_updateState) Task=`98e45ba8-220d-4a34-b5a5-313861f8c998`::moving from state init -> state preparing >Thread-26190::INFO::2012-12-02 14:08:37,052::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26190::INFO::2012-12-02 14:08:37,053::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00869917869568', 'lastCheck': 1354450114.6988521, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00440907478333', 'lastCheck': 1354450109.5859771, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00388193130493', 'lastCheck': 1354450112.85707, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072660446167', 'lastCheck': 1354450109.59517, 'code': 0, 'valid': True}} >Thread-26190::DEBUG::2012-12-02 14:08:37,053::task::1151::TaskManager.Task::(prepare) Task=`98e45ba8-220d-4a34-b5a5-313861f8c998`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00869917869568', 'lastCheck': 1354450114.6988521, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00440907478333', 'lastCheck': 1354450109.5859771, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00388193130493', 'lastCheck': 1354450112.85707, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072660446167', 'lastCheck': 1354450109.59517, 'code': 0, 'valid': True}} >Thread-26190::DEBUG::2012-12-02 14:08:37,053::task::568::TaskManager.Task::(_updateState) Task=`98e45ba8-220d-4a34-b5a5-313861f8c998`::moving from state preparing -> state finished >Thread-26190::DEBUG::2012-12-02 14:08:37,053::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26190::DEBUG::2012-12-02 14:08:37,053::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26190::DEBUG::2012-12-02 14:08:37,053::task::957::TaskManager.Task::(_decref) Task=`98e45ba8-220d-4a34-b5a5-313861f8c998`::ref 0 aborting False >Thread-26194::DEBUG::2012-12-02 14:08:41,311::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26194::DEBUG::2012-12-02 14:08:41,311::task::568::TaskManager.Task::(_updateState) Task=`a877ca8b-7d3a-4797-aab7-7d3818fda397`::moving from state init -> state preparing >Thread-26194::INFO::2012-12-02 14:08:41,312::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26194::INFO::2012-12-02 14:08:41,312::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26194::DEBUG::2012-12-02 14:08:41,312::task::1151::TaskManager.Task::(prepare) Task=`a877ca8b-7d3a-4797-aab7-7d3818fda397`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26194::DEBUG::2012-12-02 14:08:41,312::task::568::TaskManager.Task::(_updateState) Task=`a877ca8b-7d3a-4797-aab7-7d3818fda397`::moving from state preparing -> state finished >Thread-26194::DEBUG::2012-12-02 14:08:41,312::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26194::DEBUG::2012-12-02 14:08:41,312::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26194::DEBUG::2012-12-02 14:08:41,312::task::957::TaskManager.Task::(_decref) Task=`a877ca8b-7d3a-4797-aab7-7d3818fda397`::ref 0 aborting False >Thread-26195::DEBUG::2012-12-02 14:08:41,315::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26195::DEBUG::2012-12-02 14:08:41,315::task::568::TaskManager.Task::(_updateState) Task=`2b329958-be20-4f21-bd24-f95442c4c764`::moving from state init -> state preparing >Thread-26195::INFO::2012-12-02 14:08:41,315::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26195::DEBUG::2012-12-02 14:08:41,316::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`12435496-da38-4cd7-81d4-9f73ac242d24`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26195::DEBUG::2012-12-02 14:08:41,316::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26195::DEBUG::2012-12-02 14:08:41,316::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26195::DEBUG::2012-12-02 14:08:41,316::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`12435496-da38-4cd7-81d4-9f73ac242d24`::Granted request >Thread-26195::DEBUG::2012-12-02 14:08:41,316::task::794::TaskManager.Task::(resourceAcquired) Task=`2b329958-be20-4f21-bd24-f95442c4c764`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26195::DEBUG::2012-12-02 14:08:41,316::task::957::TaskManager.Task::(_decref) Task=`2b329958-be20-4f21-bd24-f95442c4c764`::ref 1 aborting False >Thread-26195::INFO::2012-12-02 14:08:41,321::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26195::DEBUG::2012-12-02 14:08:41,321::task::1151::TaskManager.Task::(prepare) Task=`2b329958-be20-4f21-bd24-f95442c4c764`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26195::DEBUG::2012-12-02 14:08:41,321::task::568::TaskManager.Task::(_updateState) Task=`2b329958-be20-4f21-bd24-f95442c4c764`::moving from state preparing -> state finished >Thread-26195::DEBUG::2012-12-02 14:08:41,321::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26195::DEBUG::2012-12-02 14:08:41,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26195::DEBUG::2012-12-02 14:08:41,321::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26195::DEBUG::2012-12-02 14:08:41,321::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26195::DEBUG::2012-12-02 14:08:41,321::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26195::DEBUG::2012-12-02 14:08:41,322::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26195::DEBUG::2012-12-02 14:08:41,322::task::957::TaskManager.Task::(_decref) Task=`2b329958-be20-4f21-bd24-f95442c4c764`::ref 0 aborting False >Thread-26198::DEBUG::2012-12-02 14:08:47,166::task::568::TaskManager.Task::(_updateState) Task=`d64aacf7-7172-4f0d-a5f5-1185c86cff29`::moving from state init -> state preparing >Thread-26198::INFO::2012-12-02 14:08:47,166::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26198::INFO::2012-12-02 14:08:47,166::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00576400756836', 'lastCheck': 1354450124.714355, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00308895111084', 'lastCheck': 1354450119.6046529, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0021390914917', 'lastCheck': 1354450122.8689229, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00733113288879', 'lastCheck': 1354450119.615989, 'code': 0, 'valid': True}} >Thread-26198::DEBUG::2012-12-02 14:08:47,166::task::1151::TaskManager.Task::(prepare) Task=`d64aacf7-7172-4f0d-a5f5-1185c86cff29`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00576400756836', 'lastCheck': 1354450124.714355, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00308895111084', 'lastCheck': 1354450119.6046529, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0021390914917', 'lastCheck': 1354450122.8689229, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00733113288879', 'lastCheck': 1354450119.615989, 'code': 0, 'valid': True}} >Thread-26198::DEBUG::2012-12-02 14:08:47,166::task::568::TaskManager.Task::(_updateState) Task=`d64aacf7-7172-4f0d-a5f5-1185c86cff29`::moving from state preparing -> state finished >Thread-26198::DEBUG::2012-12-02 14:08:47,167::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26198::DEBUG::2012-12-02 14:08:47,167::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26198::DEBUG::2012-12-02 14:08:47,167::task::957::TaskManager.Task::(_decref) Task=`d64aacf7-7172-4f0d-a5f5-1185c86cff29`::ref 0 aborting False >Thread-26202::DEBUG::2012-12-02 14:08:51,416::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26202::DEBUG::2012-12-02 14:08:51,417::task::568::TaskManager.Task::(_updateState) Task=`e507794c-62ea-4fdc-8f7a-43b98c24560b`::moving from state init -> state preparing >Thread-26202::INFO::2012-12-02 14:08:51,417::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26202::INFO::2012-12-02 14:08:51,417::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26202::DEBUG::2012-12-02 14:08:51,417::task::1151::TaskManager.Task::(prepare) Task=`e507794c-62ea-4fdc-8f7a-43b98c24560b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26202::DEBUG::2012-12-02 14:08:51,418::task::568::TaskManager.Task::(_updateState) Task=`e507794c-62ea-4fdc-8f7a-43b98c24560b`::moving from state preparing -> state finished >Thread-26202::DEBUG::2012-12-02 14:08:51,418::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26202::DEBUG::2012-12-02 14:08:51,418::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26202::DEBUG::2012-12-02 14:08:51,418::task::957::TaskManager.Task::(_decref) Task=`e507794c-62ea-4fdc-8f7a-43b98c24560b`::ref 0 aborting False >Thread-26203::DEBUG::2012-12-02 14:08:51,420::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26203::DEBUG::2012-12-02 14:08:51,421::task::568::TaskManager.Task::(_updateState) Task=`d988c2eb-7838-49d1-8f8e-24ed6b322d1b`::moving from state init -> state preparing >Thread-26203::INFO::2012-12-02 14:08:51,421::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26203::DEBUG::2012-12-02 14:08:51,421::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`50a367e1-88a4-4b82-b1a4-8584b4371d49`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26203::DEBUG::2012-12-02 14:08:51,421::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26203::DEBUG::2012-12-02 14:08:51,421::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26203::DEBUG::2012-12-02 14:08:51,422::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`50a367e1-88a4-4b82-b1a4-8584b4371d49`::Granted request >Thread-26203::DEBUG::2012-12-02 14:08:51,422::task::794::TaskManager.Task::(resourceAcquired) Task=`d988c2eb-7838-49d1-8f8e-24ed6b322d1b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26203::DEBUG::2012-12-02 14:08:51,422::task::957::TaskManager.Task::(_decref) Task=`d988c2eb-7838-49d1-8f8e-24ed6b322d1b`::ref 1 aborting False >Thread-26203::INFO::2012-12-02 14:08:51,428::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26203::DEBUG::2012-12-02 14:08:51,428::task::1151::TaskManager.Task::(prepare) Task=`d988c2eb-7838-49d1-8f8e-24ed6b322d1b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26203::DEBUG::2012-12-02 14:08:51,428::task::568::TaskManager.Task::(_updateState) Task=`d988c2eb-7838-49d1-8f8e-24ed6b322d1b`::moving from state preparing -> state finished >Thread-26203::DEBUG::2012-12-02 14:08:51,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26203::DEBUG::2012-12-02 14:08:51,428::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26203::DEBUG::2012-12-02 14:08:51,428::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26203::DEBUG::2012-12-02 14:08:51,428::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26203::DEBUG::2012-12-02 14:08:51,428::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26203::DEBUG::2012-12-02 14:08:51,428::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26203::DEBUG::2012-12-02 14:08:51,429::task::957::TaskManager.Task::(_decref) Task=`d988c2eb-7838-49d1-8f8e-24ed6b322d1b`::ref 0 aborting False >Thread-26206::DEBUG::2012-12-02 14:08:57,292::task::568::TaskManager.Task::(_updateState) Task=`84759630-7e48-40ea-93bf-4877ebd7a3db`::moving from state init -> state preparing >Thread-26206::INFO::2012-12-02 14:08:57,292::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26206::INFO::2012-12-02 14:08:57,292::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00954508781433', 'lastCheck': 1354450134.733151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00199103355408', 'lastCheck': 1354450129.6200819, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00139999389648', 'lastCheck': 1354450132.8786249, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072979927063', 'lastCheck': 1354450129.6332531, 'code': 0, 'valid': True}} >Thread-26206::DEBUG::2012-12-02 14:08:57,292::task::1151::TaskManager.Task::(prepare) Task=`84759630-7e48-40ea-93bf-4877ebd7a3db`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00954508781433', 'lastCheck': 1354450134.733151, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00199103355408', 'lastCheck': 1354450129.6200819, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00139999389648', 'lastCheck': 1354450132.8786249, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072979927063', 'lastCheck': 1354450129.6332531, 'code': 0, 'valid': True}} >Thread-26206::DEBUG::2012-12-02 14:08:57,293::task::568::TaskManager.Task::(_updateState) Task=`84759630-7e48-40ea-93bf-4877ebd7a3db`::moving from state preparing -> state finished >Thread-26206::DEBUG::2012-12-02 14:08:57,293::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26206::DEBUG::2012-12-02 14:08:57,293::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26206::DEBUG::2012-12-02 14:08:57,293::task::957::TaskManager.Task::(_decref) Task=`84759630-7e48-40ea-93bf-4877ebd7a3db`::ref 0 aborting False >Thread-26210::DEBUG::2012-12-02 14:09:01,547::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26210::DEBUG::2012-12-02 14:09:01,547::task::568::TaskManager.Task::(_updateState) Task=`e6c7cec1-2313-43c0-bf28-28a4077ddb77`::moving from state init -> state preparing >Thread-26210::INFO::2012-12-02 14:09:01,547::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26210::INFO::2012-12-02 14:09:01,548::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26210::DEBUG::2012-12-02 14:09:01,548::task::1151::TaskManager.Task::(prepare) Task=`e6c7cec1-2313-43c0-bf28-28a4077ddb77`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26210::DEBUG::2012-12-02 14:09:01,548::task::568::TaskManager.Task::(_updateState) Task=`e6c7cec1-2313-43c0-bf28-28a4077ddb77`::moving from state preparing -> state finished >Thread-26210::DEBUG::2012-12-02 14:09:01,548::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26210::DEBUG::2012-12-02 14:09:01,548::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26210::DEBUG::2012-12-02 14:09:01,548::task::957::TaskManager.Task::(_decref) Task=`e6c7cec1-2313-43c0-bf28-28a4077ddb77`::ref 0 aborting False >Thread-26211::DEBUG::2012-12-02 14:09:01,551::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26211::DEBUG::2012-12-02 14:09:01,551::task::568::TaskManager.Task::(_updateState) Task=`4b26631a-8255-425a-bcaf-2dd2d920c2bb`::moving from state init -> state preparing >Thread-26211::INFO::2012-12-02 14:09:01,551::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26211::DEBUG::2012-12-02 14:09:01,552::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bb824c29-ca83-453f-a411-40be8b5b1d40`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26211::DEBUG::2012-12-02 14:09:01,552::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26211::DEBUG::2012-12-02 14:09:01,552::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26211::DEBUG::2012-12-02 14:09:01,552::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bb824c29-ca83-453f-a411-40be8b5b1d40`::Granted request >Thread-26211::DEBUG::2012-12-02 14:09:01,552::task::794::TaskManager.Task::(resourceAcquired) Task=`4b26631a-8255-425a-bcaf-2dd2d920c2bb`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26211::DEBUG::2012-12-02 14:09:01,552::task::957::TaskManager.Task::(_decref) Task=`4b26631a-8255-425a-bcaf-2dd2d920c2bb`::ref 1 aborting False >Thread-26211::INFO::2012-12-02 14:09:01,558::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26211::DEBUG::2012-12-02 14:09:01,558::task::1151::TaskManager.Task::(prepare) Task=`4b26631a-8255-425a-bcaf-2dd2d920c2bb`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26211::DEBUG::2012-12-02 14:09:01,558::task::568::TaskManager.Task::(_updateState) Task=`4b26631a-8255-425a-bcaf-2dd2d920c2bb`::moving from state preparing -> state finished >Thread-26211::DEBUG::2012-12-02 14:09:01,558::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26211::DEBUG::2012-12-02 14:09:01,558::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26211::DEBUG::2012-12-02 14:09:01,558::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26211::DEBUG::2012-12-02 14:09:01,558::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26211::DEBUG::2012-12-02 14:09:01,559::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26211::DEBUG::2012-12-02 14:09:01,559::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26211::DEBUG::2012-12-02 14:09:01,559::task::957::TaskManager.Task::(_decref) Task=`4b26631a-8255-425a-bcaf-2dd2d920c2bb`::ref 0 aborting False >Thread-26214::DEBUG::2012-12-02 14:09:07,396::task::568::TaskManager.Task::(_updateState) Task=`1ba4fe00-bb7d-43ba-b1dd-1f3f59853dc6`::moving from state init -> state preparing >Thread-26214::INFO::2012-12-02 14:09:07,396::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26214::INFO::2012-12-02 14:09:07,397::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007493019104', 'lastCheck': 1354450144.7503941, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00182890892029', 'lastCheck': 1354450139.6315489, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00207281112671', 'lastCheck': 1354450142.890646, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821304321289', 'lastCheck': 1354450139.6494889, 'code': 0, 'valid': True}} >Thread-26214::DEBUG::2012-12-02 14:09:07,397::task::1151::TaskManager.Task::(prepare) Task=`1ba4fe00-bb7d-43ba-b1dd-1f3f59853dc6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007493019104', 'lastCheck': 1354450144.7503941, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00182890892029', 'lastCheck': 1354450139.6315489, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00207281112671', 'lastCheck': 1354450142.890646, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821304321289', 'lastCheck': 1354450139.6494889, 'code': 0, 'valid': True}} >Thread-26214::DEBUG::2012-12-02 14:09:07,397::task::568::TaskManager.Task::(_updateState) Task=`1ba4fe00-bb7d-43ba-b1dd-1f3f59853dc6`::moving from state preparing -> state finished >Thread-26214::DEBUG::2012-12-02 14:09:07,397::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26214::DEBUG::2012-12-02 14:09:07,397::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26214::DEBUG::2012-12-02 14:09:07,397::task::957::TaskManager.Task::(_decref) Task=`1ba4fe00-bb7d-43ba-b1dd-1f3f59853dc6`::ref 0 aborting False >Thread-26218::DEBUG::2012-12-02 14:09:11,683::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26218::DEBUG::2012-12-02 14:09:11,684::task::568::TaskManager.Task::(_updateState) Task=`49472497-ef81-433c-b350-5ee7afa22a77`::moving from state init -> state preparing >Thread-26218::INFO::2012-12-02 14:09:11,684::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26218::INFO::2012-12-02 14:09:11,684::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26218::DEBUG::2012-12-02 14:09:11,684::task::1151::TaskManager.Task::(prepare) Task=`49472497-ef81-433c-b350-5ee7afa22a77`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26218::DEBUG::2012-12-02 14:09:11,684::task::568::TaskManager.Task::(_updateState) Task=`49472497-ef81-433c-b350-5ee7afa22a77`::moving from state preparing -> state finished >Thread-26218::DEBUG::2012-12-02 14:09:11,684::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26218::DEBUG::2012-12-02 14:09:11,685::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26218::DEBUG::2012-12-02 14:09:11,685::task::957::TaskManager.Task::(_decref) Task=`49472497-ef81-433c-b350-5ee7afa22a77`::ref 0 aborting False >Thread-26219::DEBUG::2012-12-02 14:09:11,687::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26219::DEBUG::2012-12-02 14:09:11,687::task::568::TaskManager.Task::(_updateState) Task=`0c86b541-552b-4d13-86de-0796525ad547`::moving from state init -> state preparing >Thread-26219::INFO::2012-12-02 14:09:11,687::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26219::DEBUG::2012-12-02 14:09:11,688::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`92c13f75-d5f5-48f3-9aac-f49a6fa25f27`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26219::DEBUG::2012-12-02 14:09:11,688::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26219::DEBUG::2012-12-02 14:09:11,688::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26219::DEBUG::2012-12-02 14:09:11,688::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`92c13f75-d5f5-48f3-9aac-f49a6fa25f27`::Granted request >Thread-26219::DEBUG::2012-12-02 14:09:11,688::task::794::TaskManager.Task::(resourceAcquired) Task=`0c86b541-552b-4d13-86de-0796525ad547`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26219::DEBUG::2012-12-02 14:09:11,689::task::957::TaskManager.Task::(_decref) Task=`0c86b541-552b-4d13-86de-0796525ad547`::ref 1 aborting False >Thread-26219::INFO::2012-12-02 14:09:11,695::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26219::DEBUG::2012-12-02 14:09:11,695::task::1151::TaskManager.Task::(prepare) Task=`0c86b541-552b-4d13-86de-0796525ad547`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26219::DEBUG::2012-12-02 14:09:11,696::task::568::TaskManager.Task::(_updateState) Task=`0c86b541-552b-4d13-86de-0796525ad547`::moving from state preparing -> state finished >Thread-26219::DEBUG::2012-12-02 14:09:11,696::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26219::DEBUG::2012-12-02 14:09:11,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26219::DEBUG::2012-12-02 14:09:11,696::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26219::DEBUG::2012-12-02 14:09:11,696::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26219::DEBUG::2012-12-02 14:09:11,696::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26219::DEBUG::2012-12-02 14:09:11,696::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26219::DEBUG::2012-12-02 14:09:11,696::task::957::TaskManager.Task::(_decref) Task=`0c86b541-552b-4d13-86de-0796525ad547`::ref 0 aborting False >Thread-26222::DEBUG::2012-12-02 14:09:17,466::task::568::TaskManager.Task::(_updateState) Task=`6afb1d69-1a39-43cb-b4ca-2177bc0a4768`::moving from state init -> state preparing >Thread-26222::INFO::2012-12-02 14:09:17,466::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26222::INFO::2012-12-02 14:09:17,466::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00987696647644', 'lastCheck': 1354450154.7695899, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00361609458923', 'lastCheck': 1354450149.64573, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00306510925293', 'lastCheck': 1354450152.9041109, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00751900672913', 'lastCheck': 1354450149.665941, 'code': 0, 'valid': True}} >Thread-26222::DEBUG::2012-12-02 14:09:17,466::task::1151::TaskManager.Task::(prepare) Task=`6afb1d69-1a39-43cb-b4ca-2177bc0a4768`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00987696647644', 'lastCheck': 1354450154.7695899, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00361609458923', 'lastCheck': 1354450149.64573, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00306510925293', 'lastCheck': 1354450152.9041109, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00751900672913', 'lastCheck': 1354450149.665941, 'code': 0, 'valid': True}} >Thread-26222::DEBUG::2012-12-02 14:09:17,467::task::568::TaskManager.Task::(_updateState) Task=`6afb1d69-1a39-43cb-b4ca-2177bc0a4768`::moving from state preparing -> state finished >Thread-26222::DEBUG::2012-12-02 14:09:17,467::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26222::DEBUG::2012-12-02 14:09:17,467::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26222::DEBUG::2012-12-02 14:09:17,467::task::957::TaskManager.Task::(_decref) Task=`6afb1d69-1a39-43cb-b4ca-2177bc0a4768`::ref 0 aborting False >Thread-26226::DEBUG::2012-12-02 14:09:21,887::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26226::DEBUG::2012-12-02 14:09:21,887::task::568::TaskManager.Task::(_updateState) Task=`c8c39611-0dc1-471a-92a8-7d697f293a56`::moving from state init -> state preparing >Thread-26226::INFO::2012-12-02 14:09:21,887::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26226::INFO::2012-12-02 14:09:21,887::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26226::DEBUG::2012-12-02 14:09:21,888::task::1151::TaskManager.Task::(prepare) Task=`c8c39611-0dc1-471a-92a8-7d697f293a56`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26226::DEBUG::2012-12-02 14:09:21,888::task::568::TaskManager.Task::(_updateState) Task=`c8c39611-0dc1-471a-92a8-7d697f293a56`::moving from state preparing -> state finished >Thread-26226::DEBUG::2012-12-02 14:09:21,888::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26226::DEBUG::2012-12-02 14:09:21,888::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26226::DEBUG::2012-12-02 14:09:21,888::task::957::TaskManager.Task::(_decref) Task=`c8c39611-0dc1-471a-92a8-7d697f293a56`::ref 0 aborting False >Thread-26227::DEBUG::2012-12-02 14:09:21,890::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26227::DEBUG::2012-12-02 14:09:21,891::task::568::TaskManager.Task::(_updateState) Task=`d3456ceb-5f4c-4a62-8bb1-585a12fc04e6`::moving from state init -> state preparing >Thread-26227::INFO::2012-12-02 14:09:21,891::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26227::DEBUG::2012-12-02 14:09:21,891::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`35439c18-d7d6-4f0d-9542-619a4024ae61`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26227::DEBUG::2012-12-02 14:09:21,891::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26227::DEBUG::2012-12-02 14:09:21,892::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26227::DEBUG::2012-12-02 14:09:21,892::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`35439c18-d7d6-4f0d-9542-619a4024ae61`::Granted request >Thread-26227::DEBUG::2012-12-02 14:09:21,892::task::794::TaskManager.Task::(resourceAcquired) Task=`d3456ceb-5f4c-4a62-8bb1-585a12fc04e6`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26227::DEBUG::2012-12-02 14:09:21,892::task::957::TaskManager.Task::(_decref) Task=`d3456ceb-5f4c-4a62-8bb1-585a12fc04e6`::ref 1 aborting False >Thread-26227::INFO::2012-12-02 14:09:21,898::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26227::DEBUG::2012-12-02 14:09:21,898::task::1151::TaskManager.Task::(prepare) Task=`d3456ceb-5f4c-4a62-8bb1-585a12fc04e6`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26227::DEBUG::2012-12-02 14:09:21,898::task::568::TaskManager.Task::(_updateState) Task=`d3456ceb-5f4c-4a62-8bb1-585a12fc04e6`::moving from state preparing -> state finished >Thread-26227::DEBUG::2012-12-02 14:09:21,898::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26227::DEBUG::2012-12-02 14:09:21,898::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26227::DEBUG::2012-12-02 14:09:21,898::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26227::DEBUG::2012-12-02 14:09:21,898::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26227::DEBUG::2012-12-02 14:09:21,898::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26227::DEBUG::2012-12-02 14:09:21,898::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26227::DEBUG::2012-12-02 14:09:21,899::task::957::TaskManager.Task::(_decref) Task=`d3456ceb-5f4c-4a62-8bb1-585a12fc04e6`::ref 0 aborting False >Thread-26230::DEBUG::2012-12-02 14:09:27,526::task::568::TaskManager.Task::(_updateState) Task=`f8536061-111b-44d6-bbe8-684b83fbb67c`::moving from state init -> state preparing >Thread-26230::INFO::2012-12-02 14:09:27,526::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26230::INFO::2012-12-02 14:09:27,526::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00820207595825', 'lastCheck': 1354450164.78773, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00250601768494', 'lastCheck': 1354450159.6620691, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018618106842', 'lastCheck': 1354450162.915977, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812196731567', 'lastCheck': 1354450159.6830249, 'code': 0, 'valid': True}} >Thread-26230::DEBUG::2012-12-02 14:09:27,527::task::1151::TaskManager.Task::(prepare) Task=`f8536061-111b-44d6-bbe8-684b83fbb67c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00820207595825', 'lastCheck': 1354450164.78773, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00250601768494', 'lastCheck': 1354450159.6620691, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018618106842', 'lastCheck': 1354450162.915977, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812196731567', 'lastCheck': 1354450159.6830249, 'code': 0, 'valid': True}} >Thread-26230::DEBUG::2012-12-02 14:09:27,527::task::568::TaskManager.Task::(_updateState) Task=`f8536061-111b-44d6-bbe8-684b83fbb67c`::moving from state preparing -> state finished >Thread-26230::DEBUG::2012-12-02 14:09:27,527::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26230::DEBUG::2012-12-02 14:09:27,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26230::DEBUG::2012-12-02 14:09:27,527::task::957::TaskManager.Task::(_decref) Task=`f8536061-111b-44d6-bbe8-684b83fbb67c`::ref 0 aborting False >Thread-26234::DEBUG::2012-12-02 14:09:32,049::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26234::DEBUG::2012-12-02 14:09:32,049::task::568::TaskManager.Task::(_updateState) Task=`6506972d-cc78-4b81-8a7c-25e6183b2792`::moving from state init -> state preparing >Thread-26234::INFO::2012-12-02 14:09:32,049::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26234::INFO::2012-12-02 14:09:32,049::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26234::DEBUG::2012-12-02 14:09:32,049::task::1151::TaskManager.Task::(prepare) Task=`6506972d-cc78-4b81-8a7c-25e6183b2792`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26234::DEBUG::2012-12-02 14:09:32,050::task::568::TaskManager.Task::(_updateState) Task=`6506972d-cc78-4b81-8a7c-25e6183b2792`::moving from state preparing -> state finished >Thread-26234::DEBUG::2012-12-02 14:09:32,050::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26234::DEBUG::2012-12-02 14:09:32,050::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26234::DEBUG::2012-12-02 14:09:32,050::task::957::TaskManager.Task::(_decref) Task=`6506972d-cc78-4b81-8a7c-25e6183b2792`::ref 0 aborting False >Thread-26235::DEBUG::2012-12-02 14:09:32,053::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26235::DEBUG::2012-12-02 14:09:32,053::task::568::TaskManager.Task::(_updateState) Task=`2f390eda-af24-4bcf-9b07-ea56ab802a13`::moving from state init -> state preparing >Thread-26235::INFO::2012-12-02 14:09:32,053::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26235::DEBUG::2012-12-02 14:09:32,053::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`912216cb-050d-4da0-bcbf-bf3a474f3145`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26235::DEBUG::2012-12-02 14:09:32,053::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26235::DEBUG::2012-12-02 14:09:32,054::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26235::DEBUG::2012-12-02 14:09:32,054::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`912216cb-050d-4da0-bcbf-bf3a474f3145`::Granted request >Thread-26235::DEBUG::2012-12-02 14:09:32,054::task::794::TaskManager.Task::(resourceAcquired) Task=`2f390eda-af24-4bcf-9b07-ea56ab802a13`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26235::DEBUG::2012-12-02 14:09:32,054::task::957::TaskManager.Task::(_decref) Task=`2f390eda-af24-4bcf-9b07-ea56ab802a13`::ref 1 aborting False >Thread-26235::INFO::2012-12-02 14:09:32,059::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26235::DEBUG::2012-12-02 14:09:32,059::task::1151::TaskManager.Task::(prepare) Task=`2f390eda-af24-4bcf-9b07-ea56ab802a13`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26235::DEBUG::2012-12-02 14:09:32,059::task::568::TaskManager.Task::(_updateState) Task=`2f390eda-af24-4bcf-9b07-ea56ab802a13`::moving from state preparing -> state finished >Thread-26235::DEBUG::2012-12-02 14:09:32,059::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26235::DEBUG::2012-12-02 14:09:32,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26235::DEBUG::2012-12-02 14:09:32,059::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26235::DEBUG::2012-12-02 14:09:32,059::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26235::DEBUG::2012-12-02 14:09:32,060::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26235::DEBUG::2012-12-02 14:09:32,060::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26235::DEBUG::2012-12-02 14:09:32,060::task::957::TaskManager.Task::(_decref) Task=`2f390eda-af24-4bcf-9b07-ea56ab802a13`::ref 0 aborting False >Thread-26238::DEBUG::2012-12-02 14:09:37,690::task::568::TaskManager.Task::(_updateState) Task=`2de77814-326b-4924-bce5-8f5d1ac8422a`::moving from state init -> state preparing >Thread-26238::INFO::2012-12-02 14:09:37,690::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26238::INFO::2012-12-02 14:09:37,690::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00878405570984', 'lastCheck': 1354450174.8099091, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00341391563416', 'lastCheck': 1354450169.674454, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00305390357971', 'lastCheck': 1354450172.930021, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00709009170532', 'lastCheck': 1354450169.6978791, 'code': 0, 'valid': True}} >Thread-26238::DEBUG::2012-12-02 14:09:37,690::task::1151::TaskManager.Task::(prepare) Task=`2de77814-326b-4924-bce5-8f5d1ac8422a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00878405570984', 'lastCheck': 1354450174.8099091, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00341391563416', 'lastCheck': 1354450169.674454, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00305390357971', 'lastCheck': 1354450172.930021, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00709009170532', 'lastCheck': 1354450169.6978791, 'code': 0, 'valid': True}} >Thread-26238::DEBUG::2012-12-02 14:09:37,690::task::568::TaskManager.Task::(_updateState) Task=`2de77814-326b-4924-bce5-8f5d1ac8422a`::moving from state preparing -> state finished >Thread-26238::DEBUG::2012-12-02 14:09:37,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26238::DEBUG::2012-12-02 14:09:37,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26238::DEBUG::2012-12-02 14:09:37,691::task::957::TaskManager.Task::(_decref) Task=`2de77814-326b-4924-bce5-8f5d1ac8422a`::ref 0 aborting False >Thread-26242::DEBUG::2012-12-02 14:09:42,188::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26242::DEBUG::2012-12-02 14:09:42,188::task::568::TaskManager.Task::(_updateState) Task=`53ada093-9a73-4998-9ae4-1ef277fbd0ec`::moving from state init -> state preparing >Thread-26242::INFO::2012-12-02 14:09:42,189::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26242::INFO::2012-12-02 14:09:42,189::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26242::DEBUG::2012-12-02 14:09:42,189::task::1151::TaskManager.Task::(prepare) Task=`53ada093-9a73-4998-9ae4-1ef277fbd0ec`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26242::DEBUG::2012-12-02 14:09:42,189::task::568::TaskManager.Task::(_updateState) Task=`53ada093-9a73-4998-9ae4-1ef277fbd0ec`::moving from state preparing -> state finished >Thread-26242::DEBUG::2012-12-02 14:09:42,189::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26242::DEBUG::2012-12-02 14:09:42,189::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26242::DEBUG::2012-12-02 14:09:42,189::task::957::TaskManager.Task::(_decref) Task=`53ada093-9a73-4998-9ae4-1ef277fbd0ec`::ref 0 aborting False >Thread-26243::DEBUG::2012-12-02 14:09:42,192::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26243::DEBUG::2012-12-02 14:09:42,192::task::568::TaskManager.Task::(_updateState) Task=`277feff4-6d9c-42c3-91d1-54bb734b49f7`::moving from state init -> state preparing >Thread-26243::INFO::2012-12-02 14:09:42,192::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26243::DEBUG::2012-12-02 14:09:42,193::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e877e222-7374-4fc3-bb01-7ab1dd56ef94`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26243::DEBUG::2012-12-02 14:09:42,193::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26243::DEBUG::2012-12-02 14:09:42,193::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26243::DEBUG::2012-12-02 14:09:42,193::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e877e222-7374-4fc3-bb01-7ab1dd56ef94`::Granted request >Thread-26243::DEBUG::2012-12-02 14:09:42,193::task::794::TaskManager.Task::(resourceAcquired) Task=`277feff4-6d9c-42c3-91d1-54bb734b49f7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26243::DEBUG::2012-12-02 14:09:42,193::task::957::TaskManager.Task::(_decref) Task=`277feff4-6d9c-42c3-91d1-54bb734b49f7`::ref 1 aborting False >Thread-26243::INFO::2012-12-02 14:09:42,198::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26243::DEBUG::2012-12-02 14:09:42,198::task::1151::TaskManager.Task::(prepare) Task=`277feff4-6d9c-42c3-91d1-54bb734b49f7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26243::DEBUG::2012-12-02 14:09:42,199::task::568::TaskManager.Task::(_updateState) Task=`277feff4-6d9c-42c3-91d1-54bb734b49f7`::moving from state preparing -> state finished >Thread-26243::DEBUG::2012-12-02 14:09:42,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26243::DEBUG::2012-12-02 14:09:42,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26243::DEBUG::2012-12-02 14:09:42,199::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26243::DEBUG::2012-12-02 14:09:42,199::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26243::DEBUG::2012-12-02 14:09:42,199::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26243::DEBUG::2012-12-02 14:09:42,199::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26243::DEBUG::2012-12-02 14:09:42,199::task::957::TaskManager.Task::(_decref) Task=`277feff4-6d9c-42c3-91d1-54bb734b49f7`::ref 0 aborting False >Thread-26246::DEBUG::2012-12-02 14:09:47,964::task::568::TaskManager.Task::(_updateState) Task=`e7cfb9d4-d67a-4fac-88d1-f499245a43fc`::moving from state init -> state preparing >Thread-26246::INFO::2012-12-02 14:09:47,964::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26246::INFO::2012-12-02 14:09:47,964::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00787305831909', 'lastCheck': 1354450184.826791, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00236797332764', 'lastCheck': 1354450179.689724, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00189113616943', 'lastCheck': 1354450182.941649, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00665998458862', 'lastCheck': 1354450179.7153111, 'code': 0, 'valid': True}} >Thread-26246::DEBUG::2012-12-02 14:09:47,965::task::1151::TaskManager.Task::(prepare) Task=`e7cfb9d4-d67a-4fac-88d1-f499245a43fc`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00787305831909', 'lastCheck': 1354450184.826791, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00236797332764', 'lastCheck': 1354450179.689724, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00189113616943', 'lastCheck': 1354450182.941649, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00665998458862', 'lastCheck': 1354450179.7153111, 'code': 0, 'valid': True}} >Thread-26246::DEBUG::2012-12-02 14:09:47,965::task::568::TaskManager.Task::(_updateState) Task=`e7cfb9d4-d67a-4fac-88d1-f499245a43fc`::moving from state preparing -> state finished >Thread-26246::DEBUG::2012-12-02 14:09:47,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26246::DEBUG::2012-12-02 14:09:47,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26246::DEBUG::2012-12-02 14:09:47,965::task::957::TaskManager.Task::(_decref) Task=`e7cfb9d4-d67a-4fac-88d1-f499245a43fc`::ref 0 aborting False >Thread-26250::DEBUG::2012-12-02 14:09:52,321::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26250::DEBUG::2012-12-02 14:09:52,322::task::568::TaskManager.Task::(_updateState) Task=`d3a88b69-daff-449e-8b16-87d6f413dd5c`::moving from state init -> state preparing >Thread-26250::INFO::2012-12-02 14:09:52,322::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26250::INFO::2012-12-02 14:09:52,322::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26250::DEBUG::2012-12-02 14:09:52,322::task::1151::TaskManager.Task::(prepare) Task=`d3a88b69-daff-449e-8b16-87d6f413dd5c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26250::DEBUG::2012-12-02 14:09:52,323::task::568::TaskManager.Task::(_updateState) Task=`d3a88b69-daff-449e-8b16-87d6f413dd5c`::moving from state preparing -> state finished >Thread-26250::DEBUG::2012-12-02 14:09:52,323::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26250::DEBUG::2012-12-02 14:09:52,323::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26250::DEBUG::2012-12-02 14:09:52,323::task::957::TaskManager.Task::(_decref) Task=`d3a88b69-daff-449e-8b16-87d6f413dd5c`::ref 0 aborting False >Thread-26251::DEBUG::2012-12-02 14:09:52,325::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26251::DEBUG::2012-12-02 14:09:52,326::task::568::TaskManager.Task::(_updateState) Task=`9348f6de-cda1-42cb-916b-d687e3a320a3`::moving from state init -> state preparing >Thread-26251::INFO::2012-12-02 14:09:52,326::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26251::DEBUG::2012-12-02 14:09:52,326::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cc4fa5a3-eedc-4e27-8e97-e0f3c6750c95`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26251::DEBUG::2012-12-02 14:09:52,326::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26251::DEBUG::2012-12-02 14:09:52,327::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26251::DEBUG::2012-12-02 14:09:52,327::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cc4fa5a3-eedc-4e27-8e97-e0f3c6750c95`::Granted request >Thread-26251::DEBUG::2012-12-02 14:09:52,327::task::794::TaskManager.Task::(resourceAcquired) Task=`9348f6de-cda1-42cb-916b-d687e3a320a3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26251::DEBUG::2012-12-02 14:09:52,327::task::957::TaskManager.Task::(_decref) Task=`9348f6de-cda1-42cb-916b-d687e3a320a3`::ref 1 aborting False >Thread-26251::INFO::2012-12-02 14:09:52,334::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26251::DEBUG::2012-12-02 14:09:52,334::task::1151::TaskManager.Task::(prepare) Task=`9348f6de-cda1-42cb-916b-d687e3a320a3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26251::DEBUG::2012-12-02 14:09:52,334::task::568::TaskManager.Task::(_updateState) Task=`9348f6de-cda1-42cb-916b-d687e3a320a3`::moving from state preparing -> state finished >Thread-26251::DEBUG::2012-12-02 14:09:52,334::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26251::DEBUG::2012-12-02 14:09:52,334::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26251::DEBUG::2012-12-02 14:09:52,334::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26251::DEBUG::2012-12-02 14:09:52,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26251::DEBUG::2012-12-02 14:09:52,335::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26251::DEBUG::2012-12-02 14:09:52,335::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26251::DEBUG::2012-12-02 14:09:52,335::task::957::TaskManager.Task::(_decref) Task=`9348f6de-cda1-42cb-916b-d687e3a320a3`::ref 0 aborting False >Thread-26254::DEBUG::2012-12-02 14:09:58,026::task::568::TaskManager.Task::(_updateState) Task=`6eaa72b2-79fd-4394-83f3-c042f9c77555`::moving from state init -> state preparing >Thread-26254::INFO::2012-12-02 14:09:58,027::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26254::INFO::2012-12-02 14:09:58,027::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00854301452637', 'lastCheck': 1354450194.844245, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00188612937927', 'lastCheck': 1354450189.701189, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238800048828', 'lastCheck': 1354450192.9528689, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00759100914001', 'lastCheck': 1354450189.7320471, 'code': 0, 'valid': True}} >Thread-26254::DEBUG::2012-12-02 14:09:58,027::task::1151::TaskManager.Task::(prepare) Task=`6eaa72b2-79fd-4394-83f3-c042f9c77555`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00854301452637', 'lastCheck': 1354450194.844245, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00188612937927', 'lastCheck': 1354450189.701189, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238800048828', 'lastCheck': 1354450192.9528689, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00759100914001', 'lastCheck': 1354450189.7320471, 'code': 0, 'valid': True}} >Thread-26254::DEBUG::2012-12-02 14:09:58,027::task::568::TaskManager.Task::(_updateState) Task=`6eaa72b2-79fd-4394-83f3-c042f9c77555`::moving from state preparing -> state finished >Thread-26254::DEBUG::2012-12-02 14:09:58,027::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26254::DEBUG::2012-12-02 14:09:58,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26254::DEBUG::2012-12-02 14:09:58,027::task::957::TaskManager.Task::(_decref) Task=`6eaa72b2-79fd-4394-83f3-c042f9c77555`::ref 0 aborting False >Thread-26258::DEBUG::2012-12-02 14:10:02,473::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26258::DEBUG::2012-12-02 14:10:02,474::task::568::TaskManager.Task::(_updateState) Task=`39ae4927-c07d-4841-9326-f72cb7c189ad`::moving from state init -> state preparing >Thread-26258::INFO::2012-12-02 14:10:02,474::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26258::INFO::2012-12-02 14:10:02,474::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26258::DEBUG::2012-12-02 14:10:02,474::task::1151::TaskManager.Task::(prepare) Task=`39ae4927-c07d-4841-9326-f72cb7c189ad`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26258::DEBUG::2012-12-02 14:10:02,474::task::568::TaskManager.Task::(_updateState) Task=`39ae4927-c07d-4841-9326-f72cb7c189ad`::moving from state preparing -> state finished >Thread-26258::DEBUG::2012-12-02 14:10:02,475::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26258::DEBUG::2012-12-02 14:10:02,475::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26258::DEBUG::2012-12-02 14:10:02,475::task::957::TaskManager.Task::(_decref) Task=`39ae4927-c07d-4841-9326-f72cb7c189ad`::ref 0 aborting False >Thread-26259::DEBUG::2012-12-02 14:10:02,477::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26259::DEBUG::2012-12-02 14:10:02,477::task::568::TaskManager.Task::(_updateState) Task=`01721fac-9103-4857-baae-95d2b2283701`::moving from state init -> state preparing >Thread-26259::INFO::2012-12-02 14:10:02,478::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26259::DEBUG::2012-12-02 14:10:02,478::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4dbb5832-7d45-4d29-8a56-ddb0a15a32b6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26259::DEBUG::2012-12-02 14:10:02,478::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26259::DEBUG::2012-12-02 14:10:02,478::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26259::DEBUG::2012-12-02 14:10:02,478::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4dbb5832-7d45-4d29-8a56-ddb0a15a32b6`::Granted request >Thread-26259::DEBUG::2012-12-02 14:10:02,479::task::794::TaskManager.Task::(resourceAcquired) Task=`01721fac-9103-4857-baae-95d2b2283701`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26259::DEBUG::2012-12-02 14:10:02,479::task::957::TaskManager.Task::(_decref) Task=`01721fac-9103-4857-baae-95d2b2283701`::ref 1 aborting False >Thread-26259::INFO::2012-12-02 14:10:02,486::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26259::DEBUG::2012-12-02 14:10:02,486::task::1151::TaskManager.Task::(prepare) Task=`01721fac-9103-4857-baae-95d2b2283701`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26259::DEBUG::2012-12-02 14:10:02,486::task::568::TaskManager.Task::(_updateState) Task=`01721fac-9103-4857-baae-95d2b2283701`::moving from state preparing -> state finished >Thread-26259::DEBUG::2012-12-02 14:10:02,486::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26259::DEBUG::2012-12-02 14:10:02,486::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26259::DEBUG::2012-12-02 14:10:02,486::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26259::DEBUG::2012-12-02 14:10:02,486::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26259::DEBUG::2012-12-02 14:10:02,486::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26259::DEBUG::2012-12-02 14:10:02,486::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26259::DEBUG::2012-12-02 14:10:02,486::task::957::TaskManager.Task::(_decref) Task=`01721fac-9103-4857-baae-95d2b2283701`::ref 0 aborting False >Thread-26262::DEBUG::2012-12-02 14:10:08,104::task::568::TaskManager.Task::(_updateState) Task=`7b138381-e67f-4d19-bca5-c0ac3d8369e9`::moving from state init -> state preparing >Thread-26262::INFO::2012-12-02 14:10:08,104::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26262::INFO::2012-12-02 14:10:08,104::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00774693489075', 'lastCheck': 1354450204.8615961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0022280216217', 'lastCheck': 1354450199.7126911, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253582000732', 'lastCheck': 1354450202.9658771, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00717997550964', 'lastCheck': 1354450199.7470181, 'code': 0, 'valid': True}} >Thread-26262::DEBUG::2012-12-02 14:10:08,105::task::1151::TaskManager.Task::(prepare) Task=`7b138381-e67f-4d19-bca5-c0ac3d8369e9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00774693489075', 'lastCheck': 1354450204.8615961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0022280216217', 'lastCheck': 1354450199.7126911, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253582000732', 'lastCheck': 1354450202.9658771, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00717997550964', 'lastCheck': 1354450199.7470181, 'code': 0, 'valid': True}} >Thread-26262::DEBUG::2012-12-02 14:10:08,105::task::568::TaskManager.Task::(_updateState) Task=`7b138381-e67f-4d19-bca5-c0ac3d8369e9`::moving from state preparing -> state finished >Thread-26262::DEBUG::2012-12-02 14:10:08,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26262::DEBUG::2012-12-02 14:10:08,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26262::DEBUG::2012-12-02 14:10:08,105::task::957::TaskManager.Task::(_decref) Task=`7b138381-e67f-4d19-bca5-c0ac3d8369e9`::ref 0 aborting False >Thread-26264::DEBUG::2012-12-02 14:10:12,609::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26264::DEBUG::2012-12-02 14:10:12,610::task::568::TaskManager.Task::(_updateState) Task=`7ac53fa0-220d-41b4-8983-ffbf2fa87fcf`::moving from state init -> state preparing >Thread-26264::INFO::2012-12-02 14:10:12,610::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26264::INFO::2012-12-02 14:10:12,610::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26264::DEBUG::2012-12-02 14:10:12,610::task::1151::TaskManager.Task::(prepare) Task=`7ac53fa0-220d-41b4-8983-ffbf2fa87fcf`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26264::DEBUG::2012-12-02 14:10:12,610::task::568::TaskManager.Task::(_updateState) Task=`7ac53fa0-220d-41b4-8983-ffbf2fa87fcf`::moving from state preparing -> state finished >Thread-26264::DEBUG::2012-12-02 14:10:12,611::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26264::DEBUG::2012-12-02 14:10:12,611::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26264::DEBUG::2012-12-02 14:10:12,611::task::957::TaskManager.Task::(_decref) Task=`7ac53fa0-220d-41b4-8983-ffbf2fa87fcf`::ref 0 aborting False >Thread-26265::DEBUG::2012-12-02 14:10:12,613::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26265::DEBUG::2012-12-02 14:10:12,613::task::568::TaskManager.Task::(_updateState) Task=`da0d2e1a-6855-42d3-b8fd-33adbda2d4de`::moving from state init -> state preparing >Thread-26265::INFO::2012-12-02 14:10:12,614::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26265::DEBUG::2012-12-02 14:10:12,614::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`042c64d8-c575-4268-addc-85a20ba271aa`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26265::DEBUG::2012-12-02 14:10:12,614::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26265::DEBUG::2012-12-02 14:10:12,614::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26265::DEBUG::2012-12-02 14:10:12,614::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`042c64d8-c575-4268-addc-85a20ba271aa`::Granted request >Thread-26265::DEBUG::2012-12-02 14:10:12,615::task::794::TaskManager.Task::(resourceAcquired) Task=`da0d2e1a-6855-42d3-b8fd-33adbda2d4de`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26265::DEBUG::2012-12-02 14:10:12,615::task::957::TaskManager.Task::(_decref) Task=`da0d2e1a-6855-42d3-b8fd-33adbda2d4de`::ref 1 aborting False >Thread-26265::INFO::2012-12-02 14:10:12,620::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26265::DEBUG::2012-12-02 14:10:12,620::task::1151::TaskManager.Task::(prepare) Task=`da0d2e1a-6855-42d3-b8fd-33adbda2d4de`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26265::DEBUG::2012-12-02 14:10:12,620::task::568::TaskManager.Task::(_updateState) Task=`da0d2e1a-6855-42d3-b8fd-33adbda2d4de`::moving from state preparing -> state finished >Thread-26265::DEBUG::2012-12-02 14:10:12,620::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26265::DEBUG::2012-12-02 14:10:12,620::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26265::DEBUG::2012-12-02 14:10:12,620::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26265::DEBUG::2012-12-02 14:10:12,620::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26265::DEBUG::2012-12-02 14:10:12,620::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26265::DEBUG::2012-12-02 14:10:12,620::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26265::DEBUG::2012-12-02 14:10:12,620::task::957::TaskManager.Task::(_decref) Task=`da0d2e1a-6855-42d3-b8fd-33adbda2d4de`::ref 0 aborting False >Thread-26270::DEBUG::2012-12-02 14:10:22,252::task::568::TaskManager.Task::(_updateState) Task=`7d87904b-d831-412c-b65e-a849e14edef8`::moving from state init -> state preparing >Thread-26270::INFO::2012-12-02 14:10:22,252::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26270::INFO::2012-12-02 14:10:22,252::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00815916061401', 'lastCheck': 1354450214.879236, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00203514099121', 'lastCheck': 1354450219.7412679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0028669834137', 'lastCheck': 1354450212.9789081, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00655198097229', 'lastCheck': 1354450219.7775879, 'code': 0, 'valid': True}} >Thread-26270::DEBUG::2012-12-02 14:10:22,252::task::1151::TaskManager.Task::(prepare) Task=`7d87904b-d831-412c-b65e-a849e14edef8`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00815916061401', 'lastCheck': 1354450214.879236, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00203514099121', 'lastCheck': 1354450219.7412679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0028669834137', 'lastCheck': 1354450212.9789081, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00655198097229', 'lastCheck': 1354450219.7775879, 'code': 0, 'valid': True}} >Thread-26270::DEBUG::2012-12-02 14:10:22,252::task::568::TaskManager.Task::(_updateState) Task=`7d87904b-d831-412c-b65e-a849e14edef8`::moving from state preparing -> state finished >Thread-26270::DEBUG::2012-12-02 14:10:22,252::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26270::DEBUG::2012-12-02 14:10:22,253::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26270::DEBUG::2012-12-02 14:10:22,253::task::957::TaskManager.Task::(_decref) Task=`7d87904b-d831-412c-b65e-a849e14edef8`::ref 0 aborting False >Thread-26272::DEBUG::2012-12-02 14:10:22,866::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26272::DEBUG::2012-12-02 14:10:22,867::task::568::TaskManager.Task::(_updateState) Task=`bd025086-e491-4168-9927-e0653cfb7836`::moving from state init -> state preparing >Thread-26272::INFO::2012-12-02 14:10:22,867::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26272::INFO::2012-12-02 14:10:22,867::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26272::DEBUG::2012-12-02 14:10:22,867::task::1151::TaskManager.Task::(prepare) Task=`bd025086-e491-4168-9927-e0653cfb7836`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26272::DEBUG::2012-12-02 14:10:22,867::task::568::TaskManager.Task::(_updateState) Task=`bd025086-e491-4168-9927-e0653cfb7836`::moving from state preparing -> state finished >Thread-26272::DEBUG::2012-12-02 14:10:22,868::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26272::DEBUG::2012-12-02 14:10:22,868::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26272::DEBUG::2012-12-02 14:10:22,868::task::957::TaskManager.Task::(_decref) Task=`bd025086-e491-4168-9927-e0653cfb7836`::ref 0 aborting False >Thread-26273::DEBUG::2012-12-02 14:10:22,870::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26273::DEBUG::2012-12-02 14:10:22,871::task::568::TaskManager.Task::(_updateState) Task=`8ff74cfb-6a96-4ae5-b725-e351beeea063`::moving from state init -> state preparing >Thread-26273::INFO::2012-12-02 14:10:22,871::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26273::DEBUG::2012-12-02 14:10:22,871::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1a211e87-de9a-4f97-81e3-2171306d015b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26273::DEBUG::2012-12-02 14:10:22,871::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26273::DEBUG::2012-12-02 14:10:22,871::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26273::DEBUG::2012-12-02 14:10:22,872::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1a211e87-de9a-4f97-81e3-2171306d015b`::Granted request >Thread-26273::DEBUG::2012-12-02 14:10:22,872::task::794::TaskManager.Task::(resourceAcquired) Task=`8ff74cfb-6a96-4ae5-b725-e351beeea063`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26273::DEBUG::2012-12-02 14:10:22,872::task::957::TaskManager.Task::(_decref) Task=`8ff74cfb-6a96-4ae5-b725-e351beeea063`::ref 1 aborting False >Thread-26273::INFO::2012-12-02 14:10:22,877::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26273::DEBUG::2012-12-02 14:10:22,877::task::1151::TaskManager.Task::(prepare) Task=`8ff74cfb-6a96-4ae5-b725-e351beeea063`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26273::DEBUG::2012-12-02 14:10:22,877::task::568::TaskManager.Task::(_updateState) Task=`8ff74cfb-6a96-4ae5-b725-e351beeea063`::moving from state preparing -> state finished >Thread-26273::DEBUG::2012-12-02 14:10:22,878::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26273::DEBUG::2012-12-02 14:10:22,878::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26273::DEBUG::2012-12-02 14:10:22,878::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26273::DEBUG::2012-12-02 14:10:22,878::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26273::DEBUG::2012-12-02 14:10:22,878::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26273::DEBUG::2012-12-02 14:10:22,878::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26273::DEBUG::2012-12-02 14:10:22,878::task::957::TaskManager.Task::(_decref) Task=`8ff74cfb-6a96-4ae5-b725-e351beeea063`::ref 0 aborting False >Thread-26278::DEBUG::2012-12-02 14:10:32,401::task::568::TaskManager.Task::(_updateState) Task=`08eb515d-339c-4c10-84e9-3f3f26eb0c0a`::moving from state init -> state preparing >Thread-26278::INFO::2012-12-02 14:10:32,401::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26278::INFO::2012-12-02 14:10:32,402::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794792175293', 'lastCheck': 1354450224.8959949, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00192308425903', 'lastCheck': 1354450229.842123, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183606147766', 'lastCheck': 1354450222.990658, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00350594520569', 'lastCheck': 1354450229.8427041, 'code': 0, 'valid': True}} >Thread-26278::DEBUG::2012-12-02 14:10:32,402::task::1151::TaskManager.Task::(prepare) Task=`08eb515d-339c-4c10-84e9-3f3f26eb0c0a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794792175293', 'lastCheck': 1354450224.8959949, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00192308425903', 'lastCheck': 1354450229.842123, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183606147766', 'lastCheck': 1354450222.990658, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00350594520569', 'lastCheck': 1354450229.8427041, 'code': 0, 'valid': True}} >Thread-26278::DEBUG::2012-12-02 14:10:32,402::task::568::TaskManager.Task::(_updateState) Task=`08eb515d-339c-4c10-84e9-3f3f26eb0c0a`::moving from state preparing -> state finished >Thread-26278::DEBUG::2012-12-02 14:10:32,402::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26278::DEBUG::2012-12-02 14:10:32,402::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26278::DEBUG::2012-12-02 14:10:32,402::task::957::TaskManager.Task::(_decref) Task=`08eb515d-339c-4c10-84e9-3f3f26eb0c0a`::ref 0 aborting False >Thread-26280::DEBUG::2012-12-02 14:10:33,035::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26280::DEBUG::2012-12-02 14:10:33,035::task::568::TaskManager.Task::(_updateState) Task=`18aaff56-8f02-46e6-b7fd-076b628371ee`::moving from state init -> state preparing >Thread-26280::INFO::2012-12-02 14:10:33,035::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26280::INFO::2012-12-02 14:10:33,036::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26280::DEBUG::2012-12-02 14:10:33,036::task::1151::TaskManager.Task::(prepare) Task=`18aaff56-8f02-46e6-b7fd-076b628371ee`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26280::DEBUG::2012-12-02 14:10:33,036::task::568::TaskManager.Task::(_updateState) Task=`18aaff56-8f02-46e6-b7fd-076b628371ee`::moving from state preparing -> state finished >Thread-26280::DEBUG::2012-12-02 14:10:33,036::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26280::DEBUG::2012-12-02 14:10:33,036::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26280::DEBUG::2012-12-02 14:10:33,036::task::957::TaskManager.Task::(_decref) Task=`18aaff56-8f02-46e6-b7fd-076b628371ee`::ref 0 aborting False >Thread-26281::DEBUG::2012-12-02 14:10:33,039::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26281::DEBUG::2012-12-02 14:10:33,039::task::568::TaskManager.Task::(_updateState) Task=`5a683867-9fdf-4602-941f-4ee9cfdc15f4`::moving from state init -> state preparing >Thread-26281::INFO::2012-12-02 14:10:33,039::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26281::DEBUG::2012-12-02 14:10:33,039::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`45d5d6ba-acae-400a-a09d-53d255df6b8c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26281::DEBUG::2012-12-02 14:10:33,040::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26281::DEBUG::2012-12-02 14:10:33,040::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26281::DEBUG::2012-12-02 14:10:33,040::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`45d5d6ba-acae-400a-a09d-53d255df6b8c`::Granted request >Thread-26281::DEBUG::2012-12-02 14:10:33,040::task::794::TaskManager.Task::(resourceAcquired) Task=`5a683867-9fdf-4602-941f-4ee9cfdc15f4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26281::DEBUG::2012-12-02 14:10:33,040::task::957::TaskManager.Task::(_decref) Task=`5a683867-9fdf-4602-941f-4ee9cfdc15f4`::ref 1 aborting False >Thread-26281::INFO::2012-12-02 14:10:33,046::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26281::DEBUG::2012-12-02 14:10:33,046::task::1151::TaskManager.Task::(prepare) Task=`5a683867-9fdf-4602-941f-4ee9cfdc15f4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26281::DEBUG::2012-12-02 14:10:33,046::task::568::TaskManager.Task::(_updateState) Task=`5a683867-9fdf-4602-941f-4ee9cfdc15f4`::moving from state preparing -> state finished >Thread-26281::DEBUG::2012-12-02 14:10:33,046::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26281::DEBUG::2012-12-02 14:10:33,046::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26281::DEBUG::2012-12-02 14:10:33,047::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26281::DEBUG::2012-12-02 14:10:33,047::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26281::DEBUG::2012-12-02 14:10:33,047::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26281::DEBUG::2012-12-02 14:10:33,047::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26281::DEBUG::2012-12-02 14:10:33,047::task::957::TaskManager.Task::(_decref) Task=`5a683867-9fdf-4602-941f-4ee9cfdc15f4`::ref 0 aborting False >Thread-26286::DEBUG::2012-12-02 14:10:42,473::task::568::TaskManager.Task::(_updateState) Task=`86584e8a-e1ee-40a0-9fe0-c1699430d5a2`::moving from state init -> state preparing >Thread-26286::INFO::2012-12-02 14:10:42,473::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26286::INFO::2012-12-02 14:10:42,473::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0039370059967', 'lastCheck': 1354450234.909833, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00268697738647', 'lastCheck': 1354450239.8621919, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00202703475952', 'lastCheck': 1354450233.0551109, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842189788818', 'lastCheck': 1354450239.8683469, 'code': 0, 'valid': True}} >Thread-26286::DEBUG::2012-12-02 14:10:42,474::task::1151::TaskManager.Task::(prepare) Task=`86584e8a-e1ee-40a0-9fe0-c1699430d5a2`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0039370059967', 'lastCheck': 1354450234.909833, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00268697738647', 'lastCheck': 1354450239.8621919, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00202703475952', 'lastCheck': 1354450233.0551109, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842189788818', 'lastCheck': 1354450239.8683469, 'code': 0, 'valid': True}} >Thread-26286::DEBUG::2012-12-02 14:10:42,474::task::568::TaskManager.Task::(_updateState) Task=`86584e8a-e1ee-40a0-9fe0-c1699430d5a2`::moving from state preparing -> state finished >Thread-26286::DEBUG::2012-12-02 14:10:42,474::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26286::DEBUG::2012-12-02 14:10:42,474::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26286::DEBUG::2012-12-02 14:10:42,474::task::957::TaskManager.Task::(_decref) Task=`86584e8a-e1ee-40a0-9fe0-c1699430d5a2`::ref 0 aborting False >Thread-26288::DEBUG::2012-12-02 14:10:43,177::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26288::DEBUG::2012-12-02 14:10:43,177::task::568::TaskManager.Task::(_updateState) Task=`1bd47389-d4f1-4476-95fb-f7c918f5ee15`::moving from state init -> state preparing >Thread-26288::INFO::2012-12-02 14:10:43,177::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26288::INFO::2012-12-02 14:10:43,177::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26288::DEBUG::2012-12-02 14:10:43,178::task::1151::TaskManager.Task::(prepare) Task=`1bd47389-d4f1-4476-95fb-f7c918f5ee15`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26288::DEBUG::2012-12-02 14:10:43,178::task::568::TaskManager.Task::(_updateState) Task=`1bd47389-d4f1-4476-95fb-f7c918f5ee15`::moving from state preparing -> state finished >Thread-26288::DEBUG::2012-12-02 14:10:43,178::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26288::DEBUG::2012-12-02 14:10:43,178::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26288::DEBUG::2012-12-02 14:10:43,178::task::957::TaskManager.Task::(_decref) Task=`1bd47389-d4f1-4476-95fb-f7c918f5ee15`::ref 0 aborting False >Thread-26289::DEBUG::2012-12-02 14:10:43,180::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26289::DEBUG::2012-12-02 14:10:43,181::task::568::TaskManager.Task::(_updateState) Task=`efbf1b1c-98bc-476d-8cfd-bbaf5320cd29`::moving from state init -> state preparing >Thread-26289::INFO::2012-12-02 14:10:43,181::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26289::DEBUG::2012-12-02 14:10:43,181::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`93980361-42c2-4cee-873b-609a63d7586d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26289::DEBUG::2012-12-02 14:10:43,181::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26289::DEBUG::2012-12-02 14:10:43,182::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26289::DEBUG::2012-12-02 14:10:43,182::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`93980361-42c2-4cee-873b-609a63d7586d`::Granted request >Thread-26289::DEBUG::2012-12-02 14:10:43,182::task::794::TaskManager.Task::(resourceAcquired) Task=`efbf1b1c-98bc-476d-8cfd-bbaf5320cd29`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26289::DEBUG::2012-12-02 14:10:43,182::task::957::TaskManager.Task::(_decref) Task=`efbf1b1c-98bc-476d-8cfd-bbaf5320cd29`::ref 1 aborting False >Thread-26289::INFO::2012-12-02 14:10:43,188::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26289::DEBUG::2012-12-02 14:10:43,188::task::1151::TaskManager.Task::(prepare) Task=`efbf1b1c-98bc-476d-8cfd-bbaf5320cd29`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26289::DEBUG::2012-12-02 14:10:43,188::task::568::TaskManager.Task::(_updateState) Task=`efbf1b1c-98bc-476d-8cfd-bbaf5320cd29`::moving from state preparing -> state finished >Thread-26289::DEBUG::2012-12-02 14:10:43,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26289::DEBUG::2012-12-02 14:10:43,189::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26289::DEBUG::2012-12-02 14:10:43,189::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26289::DEBUG::2012-12-02 14:10:43,189::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26289::DEBUG::2012-12-02 14:10:43,189::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26289::DEBUG::2012-12-02 14:10:43,189::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26289::DEBUG::2012-12-02 14:10:43,189::task::957::TaskManager.Task::(_decref) Task=`efbf1b1c-98bc-476d-8cfd-bbaf5320cd29`::ref 0 aborting False >Thread-26294::DEBUG::2012-12-02 14:10:52,562::task::568::TaskManager.Task::(_updateState) Task=`be5a0c27-6d16-4b1e-a43d-3316a0b0a0c4`::moving from state init -> state preparing >Thread-26294::INFO::2012-12-02 14:10:52,562::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26294::INFO::2012-12-02 14:10:52,562::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00752687454224', 'lastCheck': 1354450244.926734, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0044801235199', 'lastCheck': 1354450249.8797879, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022828578949', 'lastCheck': 1354450243.068686, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00698113441467', 'lastCheck': 1354450249.8872311, 'code': 0, 'valid': True}} >Thread-26294::DEBUG::2012-12-02 14:10:52,562::task::1151::TaskManager.Task::(prepare) Task=`be5a0c27-6d16-4b1e-a43d-3316a0b0a0c4`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00752687454224', 'lastCheck': 1354450244.926734, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0044801235199', 'lastCheck': 1354450249.8797879, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022828578949', 'lastCheck': 1354450243.068686, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00698113441467', 'lastCheck': 1354450249.8872311, 'code': 0, 'valid': True}} >Thread-26294::DEBUG::2012-12-02 14:10:52,562::task::568::TaskManager.Task::(_updateState) Task=`be5a0c27-6d16-4b1e-a43d-3316a0b0a0c4`::moving from state preparing -> state finished >Thread-26294::DEBUG::2012-12-02 14:10:52,563::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26294::DEBUG::2012-12-02 14:10:52,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26294::DEBUG::2012-12-02 14:10:52,563::task::957::TaskManager.Task::(_decref) Task=`be5a0c27-6d16-4b1e-a43d-3316a0b0a0c4`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:10:53,068::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:10:53,069::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:10:53,069::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:10:53,292::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:10:53,294::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:10:53,295::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:10:53,331::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:10:53,332::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:10:53,334::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:10:53,337::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:10:53,337::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:10:53,337::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-26357::DEBUG::2012-12-02 14:10:53,498::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26357::DEBUG::2012-12-02 14:10:53,498::task::568::TaskManager.Task::(_updateState) Task=`18bc07dd-4549-4d38-aa61-741934902515`::moving from state init -> state preparing >Thread-26357::INFO::2012-12-02 14:10:53,498::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26357::INFO::2012-12-02 14:10:53,498::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26357::DEBUG::2012-12-02 14:10:53,499::task::1151::TaskManager.Task::(prepare) Task=`18bc07dd-4549-4d38-aa61-741934902515`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26357::DEBUG::2012-12-02 14:10:53,499::task::568::TaskManager.Task::(_updateState) Task=`18bc07dd-4549-4d38-aa61-741934902515`::moving from state preparing -> state finished >Thread-26357::DEBUG::2012-12-02 14:10:53,499::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26357::DEBUG::2012-12-02 14:10:53,499::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26357::DEBUG::2012-12-02 14:10:53,499::task::957::TaskManager.Task::(_decref) Task=`18bc07dd-4549-4d38-aa61-741934902515`::ref 0 aborting False >Thread-26358::DEBUG::2012-12-02 14:10:53,502::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26358::DEBUG::2012-12-02 14:10:53,502::task::568::TaskManager.Task::(_updateState) Task=`5d3afb88-bc0d-438e-b7b6-6220f45ada73`::moving from state init -> state preparing >Thread-26358::INFO::2012-12-02 14:10:53,502::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26358::DEBUG::2012-12-02 14:10:53,502::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6a66272e-ceaf-4b50-8ffa-23284176d843`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26358::DEBUG::2012-12-02 14:10:53,503::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26358::DEBUG::2012-12-02 14:10:53,503::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26358::DEBUG::2012-12-02 14:10:53,503::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6a66272e-ceaf-4b50-8ffa-23284176d843`::Granted request >Thread-26358::DEBUG::2012-12-02 14:10:53,503::task::794::TaskManager.Task::(resourceAcquired) Task=`5d3afb88-bc0d-438e-b7b6-6220f45ada73`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26358::DEBUG::2012-12-02 14:10:53,503::task::957::TaskManager.Task::(_decref) Task=`5d3afb88-bc0d-438e-b7b6-6220f45ada73`::ref 1 aborting False >Thread-26358::INFO::2012-12-02 14:10:53,509::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26358::DEBUG::2012-12-02 14:10:53,509::task::1151::TaskManager.Task::(prepare) Task=`5d3afb88-bc0d-438e-b7b6-6220f45ada73`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26358::DEBUG::2012-12-02 14:10:53,509::task::568::TaskManager.Task::(_updateState) Task=`5d3afb88-bc0d-438e-b7b6-6220f45ada73`::moving from state preparing -> state finished >Thread-26358::DEBUG::2012-12-02 14:10:53,509::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26358::DEBUG::2012-12-02 14:10:53,509::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26358::DEBUG::2012-12-02 14:10:53,509::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26358::DEBUG::2012-12-02 14:10:53,509::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26358::DEBUG::2012-12-02 14:10:53,509::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26358::DEBUG::2012-12-02 14:10:53,509::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26358::DEBUG::2012-12-02 14:10:53,509::task::957::TaskManager.Task::(_decref) Task=`5d3afb88-bc0d-438e-b7b6-6220f45ada73`::ref 0 aborting False >Thread-11821::DEBUG::2012-12-02 14:10:54,926::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:10:54,927::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:10:54,927::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:10:55,139::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:10:55,140::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:10:55,140::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:10:55,175::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:10:55,175::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:10:55,183::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:10:55,186::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:10:55,186::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:10:55,186::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-11822::DEBUG::2012-12-02 14:10:59,879::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:10:59,880::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:10:59,880::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:11:00,087::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:11:00,089::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:11:00,089::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:11:00,124::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:11:00,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:11:00,126::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:11:00,130::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:11:00,130::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:11:00,130::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:11:00,130::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:11:00,131::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:11:00,133::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:11:00,336::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:11:00,338::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:11:00,338::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:11:00,373::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:11:00,373::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:11:00,382::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:11:00,385::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:11:00,385::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:11:00,385::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-26546::DEBUG::2012-12-02 14:11:02,649::task::568::TaskManager.Task::(_updateState) Task=`83837da9-aa7b-4722-87a9-cfd7c426ab42`::moving from state init -> state preparing >Thread-26546::INFO::2012-12-02 14:11:02,650::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26546::INFO::2012-12-02 14:11:02,650::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00681209564209', 'lastCheck': 1354450255.2020559, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00415897369385', 'lastCheck': 1354450260.394088, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00173497200012', 'lastCheck': 1354450253.348196, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00615692138672', 'lastCheck': 1354450260.4020679, 'code': 0, 'valid': True}} >Thread-26546::DEBUG::2012-12-02 14:11:02,650::task::1151::TaskManager.Task::(prepare) Task=`83837da9-aa7b-4722-87a9-cfd7c426ab42`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00681209564209', 'lastCheck': 1354450255.2020559, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00415897369385', 'lastCheck': 1354450260.394088, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00173497200012', 'lastCheck': 1354450253.348196, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00615692138672', 'lastCheck': 1354450260.4020679, 'code': 0, 'valid': True}} >Thread-26546::DEBUG::2012-12-02 14:11:02,650::task::568::TaskManager.Task::(_updateState) Task=`83837da9-aa7b-4722-87a9-cfd7c426ab42`::moving from state preparing -> state finished >Thread-26546::DEBUG::2012-12-02 14:11:02,650::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26546::DEBUG::2012-12-02 14:11:02,650::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26546::DEBUG::2012-12-02 14:11:02,651::task::957::TaskManager.Task::(_decref) Task=`83837da9-aa7b-4722-87a9-cfd7c426ab42`::ref 0 aborting False >Thread-26548::DEBUG::2012-12-02 14:11:03,603::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26548::DEBUG::2012-12-02 14:11:03,603::task::568::TaskManager.Task::(_updateState) Task=`d8c86172-334f-495a-9a55-0cf3cd8ab6f8`::moving from state init -> state preparing >Thread-26548::INFO::2012-12-02 14:11:03,603::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26548::INFO::2012-12-02 14:11:03,603::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26548::DEBUG::2012-12-02 14:11:03,603::task::1151::TaskManager.Task::(prepare) Task=`d8c86172-334f-495a-9a55-0cf3cd8ab6f8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26548::DEBUG::2012-12-02 14:11:03,604::task::568::TaskManager.Task::(_updateState) Task=`d8c86172-334f-495a-9a55-0cf3cd8ab6f8`::moving from state preparing -> state finished >Thread-26548::DEBUG::2012-12-02 14:11:03,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26548::DEBUG::2012-12-02 14:11:03,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26548::DEBUG::2012-12-02 14:11:03,604::task::957::TaskManager.Task::(_decref) Task=`d8c86172-334f-495a-9a55-0cf3cd8ab6f8`::ref 0 aborting False >Thread-26549::DEBUG::2012-12-02 14:11:03,606::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26549::DEBUG::2012-12-02 14:11:03,607::task::568::TaskManager.Task::(_updateState) Task=`dadad0e7-7145-4ee7-aa02-313213392ce4`::moving from state init -> state preparing >Thread-26549::INFO::2012-12-02 14:11:03,607::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26549::DEBUG::2012-12-02 14:11:03,607::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f1c135d7-2278-4dba-a4c0-53eac0b33eef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26549::DEBUG::2012-12-02 14:11:03,607::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26549::DEBUG::2012-12-02 14:11:03,607::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26549::DEBUG::2012-12-02 14:11:03,607::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f1c135d7-2278-4dba-a4c0-53eac0b33eef`::Granted request >Thread-26549::DEBUG::2012-12-02 14:11:03,608::task::794::TaskManager.Task::(resourceAcquired) Task=`dadad0e7-7145-4ee7-aa02-313213392ce4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26549::DEBUG::2012-12-02 14:11:03,608::task::957::TaskManager.Task::(_decref) Task=`dadad0e7-7145-4ee7-aa02-313213392ce4`::ref 1 aborting False >Thread-26549::INFO::2012-12-02 14:11:03,612::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26549::DEBUG::2012-12-02 14:11:03,612::task::1151::TaskManager.Task::(prepare) Task=`dadad0e7-7145-4ee7-aa02-313213392ce4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26549::DEBUG::2012-12-02 14:11:03,612::task::568::TaskManager.Task::(_updateState) Task=`dadad0e7-7145-4ee7-aa02-313213392ce4`::moving from state preparing -> state finished >Thread-26549::DEBUG::2012-12-02 14:11:03,612::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26549::DEBUG::2012-12-02 14:11:03,612::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26549::DEBUG::2012-12-02 14:11:03,612::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26549::DEBUG::2012-12-02 14:11:03,613::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26549::DEBUG::2012-12-02 14:11:03,613::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26549::DEBUG::2012-12-02 14:11:03,613::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26549::DEBUG::2012-12-02 14:11:03,613::task::957::TaskManager.Task::(_decref) Task=`dadad0e7-7145-4ee7-aa02-313213392ce4`::ref 0 aborting False >Thread-26554::DEBUG::2012-12-02 14:11:12,745::task::568::TaskManager.Task::(_updateState) Task=`38bd30ac-95cb-4f69-b5e5-ec56cd6c3c14`::moving from state init -> state preparing >Thread-26554::INFO::2012-12-02 14:11:12,746::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26554::INFO::2012-12-02 14:11:12,746::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00943207740784', 'lastCheck': 1354450265.2214291, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00272512435913', 'lastCheck': 1354450270.410259, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213313102722', 'lastCheck': 1354450263.36163, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00327014923096', 'lastCheck': 1354450270.414819, 'code': 0, 'valid': True}} >Thread-26554::DEBUG::2012-12-02 14:11:12,746::task::1151::TaskManager.Task::(prepare) Task=`38bd30ac-95cb-4f69-b5e5-ec56cd6c3c14`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00943207740784', 'lastCheck': 1354450265.2214291, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00272512435913', 'lastCheck': 1354450270.410259, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213313102722', 'lastCheck': 1354450263.36163, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00327014923096', 'lastCheck': 1354450270.414819, 'code': 0, 'valid': True}} >Thread-26554::DEBUG::2012-12-02 14:11:12,746::task::568::TaskManager.Task::(_updateState) Task=`38bd30ac-95cb-4f69-b5e5-ec56cd6c3c14`::moving from state preparing -> state finished >Thread-26554::DEBUG::2012-12-02 14:11:12,746::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26554::DEBUG::2012-12-02 14:11:12,746::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26554::DEBUG::2012-12-02 14:11:12,747::task::957::TaskManager.Task::(_decref) Task=`38bd30ac-95cb-4f69-b5e5-ec56cd6c3c14`::ref 0 aborting False >Thread-26556::DEBUG::2012-12-02 14:11:14,037::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26556::DEBUG::2012-12-02 14:11:14,038::task::568::TaskManager.Task::(_updateState) Task=`e0d8c3ac-9248-4bfa-93ba-f46e77144d5f`::moving from state init -> state preparing >Thread-26556::INFO::2012-12-02 14:11:14,038::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26556::INFO::2012-12-02 14:11:14,038::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26556::DEBUG::2012-12-02 14:11:14,038::task::1151::TaskManager.Task::(prepare) Task=`e0d8c3ac-9248-4bfa-93ba-f46e77144d5f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26556::DEBUG::2012-12-02 14:11:14,038::task::568::TaskManager.Task::(_updateState) Task=`e0d8c3ac-9248-4bfa-93ba-f46e77144d5f`::moving from state preparing -> state finished >Thread-26556::DEBUG::2012-12-02 14:11:14,039::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26556::DEBUG::2012-12-02 14:11:14,039::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26556::DEBUG::2012-12-02 14:11:14,039::task::957::TaskManager.Task::(_decref) Task=`e0d8c3ac-9248-4bfa-93ba-f46e77144d5f`::ref 0 aborting False >Thread-26557::DEBUG::2012-12-02 14:11:14,041::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26557::DEBUG::2012-12-02 14:11:14,042::task::568::TaskManager.Task::(_updateState) Task=`b23dac53-e49c-4dc4-9748-a314beefdf83`::moving from state init -> state preparing >Thread-26557::INFO::2012-12-02 14:11:14,042::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26557::DEBUG::2012-12-02 14:11:14,042::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`092ca176-e9a9-453b-9618-391f8c1e9cf6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26557::DEBUG::2012-12-02 14:11:14,042::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26557::DEBUG::2012-12-02 14:11:14,042::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26557::DEBUG::2012-12-02 14:11:14,042::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`092ca176-e9a9-453b-9618-391f8c1e9cf6`::Granted request >Thread-26557::DEBUG::2012-12-02 14:11:14,043::task::794::TaskManager.Task::(resourceAcquired) Task=`b23dac53-e49c-4dc4-9748-a314beefdf83`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26557::DEBUG::2012-12-02 14:11:14,043::task::957::TaskManager.Task::(_decref) Task=`b23dac53-e49c-4dc4-9748-a314beefdf83`::ref 1 aborting False >Thread-26557::INFO::2012-12-02 14:11:14,048::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26557::DEBUG::2012-12-02 14:11:14,048::task::1151::TaskManager.Task::(prepare) Task=`b23dac53-e49c-4dc4-9748-a314beefdf83`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26557::DEBUG::2012-12-02 14:11:14,048::task::568::TaskManager.Task::(_updateState) Task=`b23dac53-e49c-4dc4-9748-a314beefdf83`::moving from state preparing -> state finished >Thread-26557::DEBUG::2012-12-02 14:11:14,049::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26557::DEBUG::2012-12-02 14:11:14,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26557::DEBUG::2012-12-02 14:11:14,049::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26557::DEBUG::2012-12-02 14:11:14,049::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26557::DEBUG::2012-12-02 14:11:14,049::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26557::DEBUG::2012-12-02 14:11:14,049::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26557::DEBUG::2012-12-02 14:11:14,049::task::957::TaskManager.Task::(_decref) Task=`b23dac53-e49c-4dc4-9748-a314beefdf83`::ref 0 aborting False >Thread-26562::DEBUG::2012-12-02 14:11:22,808::task::568::TaskManager.Task::(_updateState) Task=`eb2c5d64-27e0-4901-8bf8-54e1184d5db3`::moving from state init -> state preparing >Thread-26562::INFO::2012-12-02 14:11:22,809::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26562::INFO::2012-12-02 14:11:22,809::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00806188583374', 'lastCheck': 1354450275.240267, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00441098213196', 'lastCheck': 1354450280.4312699, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0032069683075', 'lastCheck': 1354450273.3745961, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00870704650879', 'lastCheck': 1354450280.436162, 'code': 0, 'valid': True}} >Thread-26562::DEBUG::2012-12-02 14:11:22,809::task::1151::TaskManager.Task::(prepare) Task=`eb2c5d64-27e0-4901-8bf8-54e1184d5db3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00806188583374', 'lastCheck': 1354450275.240267, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00441098213196', 'lastCheck': 1354450280.4312699, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0032069683075', 'lastCheck': 1354450273.3745961, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00870704650879', 'lastCheck': 1354450280.436162, 'code': 0, 'valid': True}} >Thread-26562::DEBUG::2012-12-02 14:11:22,809::task::568::TaskManager.Task::(_updateState) Task=`eb2c5d64-27e0-4901-8bf8-54e1184d5db3`::moving from state preparing -> state finished >Thread-26562::DEBUG::2012-12-02 14:11:22,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26562::DEBUG::2012-12-02 14:11:22,810::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26562::DEBUG::2012-12-02 14:11:22,810::task::957::TaskManager.Task::(_decref) Task=`eb2c5d64-27e0-4901-8bf8-54e1184d5db3`::ref 0 aborting False >Thread-26564::DEBUG::2012-12-02 14:11:24,142::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26564::DEBUG::2012-12-02 14:11:24,143::task::568::TaskManager.Task::(_updateState) Task=`a7b47bcd-c98c-471e-95d1-8888d0fe8758`::moving from state init -> state preparing >Thread-26564::INFO::2012-12-02 14:11:24,143::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26564::INFO::2012-12-02 14:11:24,143::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26564::DEBUG::2012-12-02 14:11:24,143::task::1151::TaskManager.Task::(prepare) Task=`a7b47bcd-c98c-471e-95d1-8888d0fe8758`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26564::DEBUG::2012-12-02 14:11:24,143::task::568::TaskManager.Task::(_updateState) Task=`a7b47bcd-c98c-471e-95d1-8888d0fe8758`::moving from state preparing -> state finished >Thread-26564::DEBUG::2012-12-02 14:11:24,143::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26564::DEBUG::2012-12-02 14:11:24,143::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26564::DEBUG::2012-12-02 14:11:24,144::task::957::TaskManager.Task::(_decref) Task=`a7b47bcd-c98c-471e-95d1-8888d0fe8758`::ref 0 aborting False >Thread-26565::DEBUG::2012-12-02 14:11:24,146::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26565::DEBUG::2012-12-02 14:11:24,146::task::568::TaskManager.Task::(_updateState) Task=`2866eba1-9fbf-4839-9016-e43de0658609`::moving from state init -> state preparing >Thread-26565::INFO::2012-12-02 14:11:24,146::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26565::DEBUG::2012-12-02 14:11:24,147::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`afbca05c-d877-415f-94f8-e65eeaeec24f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26565::DEBUG::2012-12-02 14:11:24,147::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26565::DEBUG::2012-12-02 14:11:24,147::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26565::DEBUG::2012-12-02 14:11:24,147::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`afbca05c-d877-415f-94f8-e65eeaeec24f`::Granted request >Thread-26565::DEBUG::2012-12-02 14:11:24,147::task::794::TaskManager.Task::(resourceAcquired) Task=`2866eba1-9fbf-4839-9016-e43de0658609`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26565::DEBUG::2012-12-02 14:11:24,147::task::957::TaskManager.Task::(_decref) Task=`2866eba1-9fbf-4839-9016-e43de0658609`::ref 1 aborting False >Thread-26565::INFO::2012-12-02 14:11:24,151::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26565::DEBUG::2012-12-02 14:11:24,152::task::1151::TaskManager.Task::(prepare) Task=`2866eba1-9fbf-4839-9016-e43de0658609`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26565::DEBUG::2012-12-02 14:11:24,152::task::568::TaskManager.Task::(_updateState) Task=`2866eba1-9fbf-4839-9016-e43de0658609`::moving from state preparing -> state finished >Thread-26565::DEBUG::2012-12-02 14:11:24,152::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26565::DEBUG::2012-12-02 14:11:24,152::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26565::DEBUG::2012-12-02 14:11:24,152::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26565::DEBUG::2012-12-02 14:11:24,152::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26565::DEBUG::2012-12-02 14:11:24,152::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26565::DEBUG::2012-12-02 14:11:24,152::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26565::DEBUG::2012-12-02 14:11:24,152::task::957::TaskManager.Task::(_decref) Task=`2866eba1-9fbf-4839-9016-e43de0658609`::ref 0 aborting False >Thread-26570::DEBUG::2012-12-02 14:11:32,913::task::568::TaskManager.Task::(_updateState) Task=`ec57bd66-c36e-4b76-ab14-6709878ef303`::moving from state init -> state preparing >Thread-26570::INFO::2012-12-02 14:11:32,914::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26570::INFO::2012-12-02 14:11:32,914::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00823521614075', 'lastCheck': 1354450285.2576139, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00563812255859', 'lastCheck': 1354450290.44978, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00209498405457', 'lastCheck': 1354450283.3862491, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0063910484314', 'lastCheck': 1354450290.4565899, 'code': 0, 'valid': True}} >Thread-26570::DEBUG::2012-12-02 14:11:32,914::task::1151::TaskManager.Task::(prepare) Task=`ec57bd66-c36e-4b76-ab14-6709878ef303`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00823521614075', 'lastCheck': 1354450285.2576139, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00563812255859', 'lastCheck': 1354450290.44978, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00209498405457', 'lastCheck': 1354450283.3862491, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0063910484314', 'lastCheck': 1354450290.4565899, 'code': 0, 'valid': True}} >Thread-26570::DEBUG::2012-12-02 14:11:32,914::task::568::TaskManager.Task::(_updateState) Task=`ec57bd66-c36e-4b76-ab14-6709878ef303`::moving from state preparing -> state finished >Thread-26570::DEBUG::2012-12-02 14:11:32,914::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26570::DEBUG::2012-12-02 14:11:32,915::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26570::DEBUG::2012-12-02 14:11:32,915::task::957::TaskManager.Task::(_decref) Task=`ec57bd66-c36e-4b76-ab14-6709878ef303`::ref 0 aborting False >Thread-26572::DEBUG::2012-12-02 14:11:34,246::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26572::DEBUG::2012-12-02 14:11:34,247::task::568::TaskManager.Task::(_updateState) Task=`c4ab30ef-e34e-465f-beb5-aa4d9e1f08e3`::moving from state init -> state preparing >Thread-26572::INFO::2012-12-02 14:11:34,247::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26572::INFO::2012-12-02 14:11:34,247::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26572::DEBUG::2012-12-02 14:11:34,247::task::1151::TaskManager.Task::(prepare) Task=`c4ab30ef-e34e-465f-beb5-aa4d9e1f08e3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26572::DEBUG::2012-12-02 14:11:34,247::task::568::TaskManager.Task::(_updateState) Task=`c4ab30ef-e34e-465f-beb5-aa4d9e1f08e3`::moving from state preparing -> state finished >Thread-26572::DEBUG::2012-12-02 14:11:34,247::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26572::DEBUG::2012-12-02 14:11:34,247::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26572::DEBUG::2012-12-02 14:11:34,248::task::957::TaskManager.Task::(_decref) Task=`c4ab30ef-e34e-465f-beb5-aa4d9e1f08e3`::ref 0 aborting False >Thread-26573::DEBUG::2012-12-02 14:11:34,250::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26573::DEBUG::2012-12-02 14:11:34,250::task::568::TaskManager.Task::(_updateState) Task=`241ee569-415f-4d84-a5d3-e66ff7d77cee`::moving from state init -> state preparing >Thread-26573::INFO::2012-12-02 14:11:34,251::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26573::DEBUG::2012-12-02 14:11:34,251::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e34b9b40-a872-4385-a315-b9ee5aee0f93`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26573::DEBUG::2012-12-02 14:11:34,251::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26573::DEBUG::2012-12-02 14:11:34,251::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26573::DEBUG::2012-12-02 14:11:34,251::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e34b9b40-a872-4385-a315-b9ee5aee0f93`::Granted request >Thread-26573::DEBUG::2012-12-02 14:11:34,252::task::794::TaskManager.Task::(resourceAcquired) Task=`241ee569-415f-4d84-a5d3-e66ff7d77cee`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26573::DEBUG::2012-12-02 14:11:34,252::task::957::TaskManager.Task::(_decref) Task=`241ee569-415f-4d84-a5d3-e66ff7d77cee`::ref 1 aborting False >Thread-26573::INFO::2012-12-02 14:11:34,257::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26573::DEBUG::2012-12-02 14:11:34,257::task::1151::TaskManager.Task::(prepare) Task=`241ee569-415f-4d84-a5d3-e66ff7d77cee`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26573::DEBUG::2012-12-02 14:11:34,257::task::568::TaskManager.Task::(_updateState) Task=`241ee569-415f-4d84-a5d3-e66ff7d77cee`::moving from state preparing -> state finished >Thread-26573::DEBUG::2012-12-02 14:11:34,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26573::DEBUG::2012-12-02 14:11:34,258::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26573::DEBUG::2012-12-02 14:11:34,258::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26573::DEBUG::2012-12-02 14:11:34,258::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26573::DEBUG::2012-12-02 14:11:34,258::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26573::DEBUG::2012-12-02 14:11:34,258::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26573::DEBUG::2012-12-02 14:11:34,258::task::957::TaskManager.Task::(_decref) Task=`241ee569-415f-4d84-a5d3-e66ff7d77cee`::ref 0 aborting False >Thread-26578::DEBUG::2012-12-02 14:11:43,079::task::568::TaskManager.Task::(_updateState) Task=`46b540d1-cb61-4a79-9f91-5dee0c5242f1`::moving from state init -> state preparing >Thread-26578::INFO::2012-12-02 14:11:43,079::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26578::INFO::2012-12-02 14:11:43,079::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00874495506287', 'lastCheck': 1354450295.279103, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00605511665344', 'lastCheck': 1354450300.4701011, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00323796272278', 'lastCheck': 1354450293.400876, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0138010978699', 'lastCheck': 1354450300.486166, 'code': 0, 'valid': True}} >Thread-26578::DEBUG::2012-12-02 14:11:43,079::task::1151::TaskManager.Task::(prepare) Task=`46b540d1-cb61-4a79-9f91-5dee0c5242f1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00874495506287', 'lastCheck': 1354450295.279103, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00605511665344', 'lastCheck': 1354450300.4701011, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00323796272278', 'lastCheck': 1354450293.400876, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0138010978699', 'lastCheck': 1354450300.486166, 'code': 0, 'valid': True}} >Thread-26578::DEBUG::2012-12-02 14:11:43,079::task::568::TaskManager.Task::(_updateState) Task=`46b540d1-cb61-4a79-9f91-5dee0c5242f1`::moving from state preparing -> state finished >Thread-26578::DEBUG::2012-12-02 14:11:43,080::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26578::DEBUG::2012-12-02 14:11:43,080::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26578::DEBUG::2012-12-02 14:11:43,080::task::957::TaskManager.Task::(_decref) Task=`46b540d1-cb61-4a79-9f91-5dee0c5242f1`::ref 0 aborting False >Thread-26580::DEBUG::2012-12-02 14:11:44,432::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26580::DEBUG::2012-12-02 14:11:44,433::task::568::TaskManager.Task::(_updateState) Task=`acea0943-97e0-4281-9d0b-f6c0a0fdb44f`::moving from state init -> state preparing >Thread-26580::INFO::2012-12-02 14:11:44,433::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26580::INFO::2012-12-02 14:11:44,433::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26580::DEBUG::2012-12-02 14:11:44,433::task::1151::TaskManager.Task::(prepare) Task=`acea0943-97e0-4281-9d0b-f6c0a0fdb44f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26580::DEBUG::2012-12-02 14:11:44,433::task::568::TaskManager.Task::(_updateState) Task=`acea0943-97e0-4281-9d0b-f6c0a0fdb44f`::moving from state preparing -> state finished >Thread-26580::DEBUG::2012-12-02 14:11:44,433::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26580::DEBUG::2012-12-02 14:11:44,434::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26580::DEBUG::2012-12-02 14:11:44,434::task::957::TaskManager.Task::(_decref) Task=`acea0943-97e0-4281-9d0b-f6c0a0fdb44f`::ref 0 aborting False >Thread-26581::DEBUG::2012-12-02 14:11:44,436::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26581::DEBUG::2012-12-02 14:11:44,437::task::568::TaskManager.Task::(_updateState) Task=`49a66833-6c49-4f04-8104-a7f0acc43b2a`::moving from state init -> state preparing >Thread-26581::INFO::2012-12-02 14:11:44,437::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26581::DEBUG::2012-12-02 14:11:44,437::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`765c1e3c-d570-4fff-b4e6-4a5be2c01b32`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26581::DEBUG::2012-12-02 14:11:44,437::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26581::DEBUG::2012-12-02 14:11:44,437::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26581::DEBUG::2012-12-02 14:11:44,437::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`765c1e3c-d570-4fff-b4e6-4a5be2c01b32`::Granted request >Thread-26581::DEBUG::2012-12-02 14:11:44,438::task::794::TaskManager.Task::(resourceAcquired) Task=`49a66833-6c49-4f04-8104-a7f0acc43b2a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26581::DEBUG::2012-12-02 14:11:44,438::task::957::TaskManager.Task::(_decref) Task=`49a66833-6c49-4f04-8104-a7f0acc43b2a`::ref 1 aborting False >Thread-26581::INFO::2012-12-02 14:11:44,444::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26581::DEBUG::2012-12-02 14:11:44,444::task::1151::TaskManager.Task::(prepare) Task=`49a66833-6c49-4f04-8104-a7f0acc43b2a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26581::DEBUG::2012-12-02 14:11:44,445::task::568::TaskManager.Task::(_updateState) Task=`49a66833-6c49-4f04-8104-a7f0acc43b2a`::moving from state preparing -> state finished >Thread-26581::DEBUG::2012-12-02 14:11:44,445::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26581::DEBUG::2012-12-02 14:11:44,445::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26581::DEBUG::2012-12-02 14:11:44,445::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26581::DEBUG::2012-12-02 14:11:44,445::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26581::DEBUG::2012-12-02 14:11:44,445::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26581::DEBUG::2012-12-02 14:11:44,445::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26581::DEBUG::2012-12-02 14:11:44,445::task::957::TaskManager.Task::(_decref) Task=`49a66833-6c49-4f04-8104-a7f0acc43b2a`::ref 0 aborting False >Thread-26586::DEBUG::2012-12-02 14:11:53,171::task::568::TaskManager.Task::(_updateState) Task=`cd2fef85-d5f7-4f16-bc38-05c03a6fb5b5`::moving from state init -> state preparing >Thread-26586::INFO::2012-12-02 14:11:53,171::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26586::INFO::2012-12-02 14:11:53,171::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0090639591217', 'lastCheck': 1354450305.299257, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00222587585449', 'lastCheck': 1354450310.484446, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.001797914505', 'lastCheck': 1354450303.4128611, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00783896446228', 'lastCheck': 1354450310.5029981, 'code': 0, 'valid': True}} >Thread-26586::DEBUG::2012-12-02 14:11:53,171::task::1151::TaskManager.Task::(prepare) Task=`cd2fef85-d5f7-4f16-bc38-05c03a6fb5b5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0090639591217', 'lastCheck': 1354450305.299257, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00222587585449', 'lastCheck': 1354450310.484446, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.001797914505', 'lastCheck': 1354450303.4128611, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00783896446228', 'lastCheck': 1354450310.5029981, 'code': 0, 'valid': True}} >Thread-26586::DEBUG::2012-12-02 14:11:53,172::task::568::TaskManager.Task::(_updateState) Task=`cd2fef85-d5f7-4f16-bc38-05c03a6fb5b5`::moving from state preparing -> state finished >Thread-26586::DEBUG::2012-12-02 14:11:53,172::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26586::DEBUG::2012-12-02 14:11:53,172::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26586::DEBUG::2012-12-02 14:11:53,172::task::957::TaskManager.Task::(_decref) Task=`cd2fef85-d5f7-4f16-bc38-05c03a6fb5b5`::ref 0 aborting False >Thread-26588::DEBUG::2012-12-02 14:11:54,588::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26588::DEBUG::2012-12-02 14:11:54,589::task::568::TaskManager.Task::(_updateState) Task=`345ecb93-ec96-49eb-bce0-b7c3803cce66`::moving from state init -> state preparing >Thread-26588::INFO::2012-12-02 14:11:54,589::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26588::INFO::2012-12-02 14:11:54,589::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26588::DEBUG::2012-12-02 14:11:54,589::task::1151::TaskManager.Task::(prepare) Task=`345ecb93-ec96-49eb-bce0-b7c3803cce66`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26588::DEBUG::2012-12-02 14:11:54,590::task::568::TaskManager.Task::(_updateState) Task=`345ecb93-ec96-49eb-bce0-b7c3803cce66`::moving from state preparing -> state finished >Thread-26588::DEBUG::2012-12-02 14:11:54,590::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26588::DEBUG::2012-12-02 14:11:54,590::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26588::DEBUG::2012-12-02 14:11:54,590::task::957::TaskManager.Task::(_decref) Task=`345ecb93-ec96-49eb-bce0-b7c3803cce66`::ref 0 aborting False >Thread-26589::DEBUG::2012-12-02 14:11:54,592::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26589::DEBUG::2012-12-02 14:11:54,593::task::568::TaskManager.Task::(_updateState) Task=`f71e6238-0da8-4685-8148-6991389671ea`::moving from state init -> state preparing >Thread-26589::INFO::2012-12-02 14:11:54,593::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26589::DEBUG::2012-12-02 14:11:54,593::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`043446e4-fba4-4bdb-85bf-e925da106ebb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26589::DEBUG::2012-12-02 14:11:54,593::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26589::DEBUG::2012-12-02 14:11:54,594::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26589::DEBUG::2012-12-02 14:11:54,594::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`043446e4-fba4-4bdb-85bf-e925da106ebb`::Granted request >Thread-26589::DEBUG::2012-12-02 14:11:54,594::task::794::TaskManager.Task::(resourceAcquired) Task=`f71e6238-0da8-4685-8148-6991389671ea`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26589::DEBUG::2012-12-02 14:11:54,594::task::957::TaskManager.Task::(_decref) Task=`f71e6238-0da8-4685-8148-6991389671ea`::ref 1 aborting False >Thread-26589::INFO::2012-12-02 14:11:54,599::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26589::DEBUG::2012-12-02 14:11:54,599::task::1151::TaskManager.Task::(prepare) Task=`f71e6238-0da8-4685-8148-6991389671ea`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26589::DEBUG::2012-12-02 14:11:54,599::task::568::TaskManager.Task::(_updateState) Task=`f71e6238-0da8-4685-8148-6991389671ea`::moving from state preparing -> state finished >Thread-26589::DEBUG::2012-12-02 14:11:54,599::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26589::DEBUG::2012-12-02 14:11:54,599::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26589::DEBUG::2012-12-02 14:11:54,599::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26589::DEBUG::2012-12-02 14:11:54,599::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26589::DEBUG::2012-12-02 14:11:54,600::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26589::DEBUG::2012-12-02 14:11:54,600::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26589::DEBUG::2012-12-02 14:11:54,600::task::957::TaskManager.Task::(_decref) Task=`f71e6238-0da8-4685-8148-6991389671ea`::ref 0 aborting False >Thread-26594::DEBUG::2012-12-02 14:12:03,268::task::568::TaskManager.Task::(_updateState) Task=`f67fec26-a16f-405d-b171-17d1879ac28a`::moving from state init -> state preparing >Thread-26594::INFO::2012-12-02 14:12:03,269::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26594::INFO::2012-12-02 14:12:03,269::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00938582420349', 'lastCheck': 1354450315.3199191, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00282692909241', 'lastCheck': 1354450320.495398, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181698799133', 'lastCheck': 1354450313.4247351, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828504562378', 'lastCheck': 1354450320.5201621, 'code': 0, 'valid': True}} >Thread-26594::DEBUG::2012-12-02 14:12:03,269::task::1151::TaskManager.Task::(prepare) Task=`f67fec26-a16f-405d-b171-17d1879ac28a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00938582420349', 'lastCheck': 1354450315.3199191, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00282692909241', 'lastCheck': 1354450320.495398, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181698799133', 'lastCheck': 1354450313.4247351, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828504562378', 'lastCheck': 1354450320.5201621, 'code': 0, 'valid': True}} >Thread-26594::DEBUG::2012-12-02 14:12:03,269::task::568::TaskManager.Task::(_updateState) Task=`f67fec26-a16f-405d-b171-17d1879ac28a`::moving from state preparing -> state finished >Thread-26594::DEBUG::2012-12-02 14:12:03,269::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26594::DEBUG::2012-12-02 14:12:03,269::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26594::DEBUG::2012-12-02 14:12:03,270::task::957::TaskManager.Task::(_decref) Task=`f67fec26-a16f-405d-b171-17d1879ac28a`::ref 0 aborting False >Thread-26596::DEBUG::2012-12-02 14:12:04,711::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26596::DEBUG::2012-12-02 14:12:04,711::task::568::TaskManager.Task::(_updateState) Task=`22d96d3e-450a-4a89-8830-fcd58d63ec75`::moving from state init -> state preparing >Thread-26596::INFO::2012-12-02 14:12:04,712::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26596::INFO::2012-12-02 14:12:04,712::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26596::DEBUG::2012-12-02 14:12:04,712::task::1151::TaskManager.Task::(prepare) Task=`22d96d3e-450a-4a89-8830-fcd58d63ec75`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26596::DEBUG::2012-12-02 14:12:04,712::task::568::TaskManager.Task::(_updateState) Task=`22d96d3e-450a-4a89-8830-fcd58d63ec75`::moving from state preparing -> state finished >Thread-26596::DEBUG::2012-12-02 14:12:04,712::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26596::DEBUG::2012-12-02 14:12:04,712::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26596::DEBUG::2012-12-02 14:12:04,712::task::957::TaskManager.Task::(_decref) Task=`22d96d3e-450a-4a89-8830-fcd58d63ec75`::ref 0 aborting False >Thread-26597::DEBUG::2012-12-02 14:12:04,715::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26597::DEBUG::2012-12-02 14:12:04,715::task::568::TaskManager.Task::(_updateState) Task=`2d7b390b-e1b0-4406-a026-e823d3af27ba`::moving from state init -> state preparing >Thread-26597::INFO::2012-12-02 14:12:04,715::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26597::DEBUG::2012-12-02 14:12:04,716::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`09e6dc03-75e7-4b73-a91b-6fb20627ca5b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26597::DEBUG::2012-12-02 14:12:04,716::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26597::DEBUG::2012-12-02 14:12:04,716::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26597::DEBUG::2012-12-02 14:12:04,716::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`09e6dc03-75e7-4b73-a91b-6fb20627ca5b`::Granted request >Thread-26597::DEBUG::2012-12-02 14:12:04,717::task::794::TaskManager.Task::(resourceAcquired) Task=`2d7b390b-e1b0-4406-a026-e823d3af27ba`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26597::DEBUG::2012-12-02 14:12:04,717::task::957::TaskManager.Task::(_decref) Task=`2d7b390b-e1b0-4406-a026-e823d3af27ba`::ref 1 aborting False >Thread-26597::INFO::2012-12-02 14:12:04,723::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26597::DEBUG::2012-12-02 14:12:04,723::task::1151::TaskManager.Task::(prepare) Task=`2d7b390b-e1b0-4406-a026-e823d3af27ba`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26597::DEBUG::2012-12-02 14:12:04,723::task::568::TaskManager.Task::(_updateState) Task=`2d7b390b-e1b0-4406-a026-e823d3af27ba`::moving from state preparing -> state finished >Thread-26597::DEBUG::2012-12-02 14:12:04,723::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26597::DEBUG::2012-12-02 14:12:04,724::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26597::DEBUG::2012-12-02 14:12:04,724::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26597::DEBUG::2012-12-02 14:12:04,724::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26597::DEBUG::2012-12-02 14:12:04,724::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26597::DEBUG::2012-12-02 14:12:04,724::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26597::DEBUG::2012-12-02 14:12:04,724::task::957::TaskManager.Task::(_decref) Task=`2d7b390b-e1b0-4406-a026-e823d3af27ba`::ref 0 aborting False >Thread-26602::DEBUG::2012-12-02 14:12:13,365::task::568::TaskManager.Task::(_updateState) Task=`b568a927-c22b-4892-a84a-79e3ae85ae61`::moving from state init -> state preparing >Thread-26602::INFO::2012-12-02 14:12:13,365::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26602::INFO::2012-12-02 14:12:13,365::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00806522369385', 'lastCheck': 1354450325.3375249, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00389504432678', 'lastCheck': 1354450330.5136769, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00212502479553', 'lastCheck': 1354450323.436873, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00730991363525', 'lastCheck': 1354450330.536865, 'code': 0, 'valid': True}} >Thread-26602::DEBUG::2012-12-02 14:12:13,365::task::1151::TaskManager.Task::(prepare) Task=`b568a927-c22b-4892-a84a-79e3ae85ae61`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00806522369385', 'lastCheck': 1354450325.3375249, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00389504432678', 'lastCheck': 1354450330.5136769, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00212502479553', 'lastCheck': 1354450323.436873, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00730991363525', 'lastCheck': 1354450330.536865, 'code': 0, 'valid': True}} >Thread-26602::DEBUG::2012-12-02 14:12:13,365::task::568::TaskManager.Task::(_updateState) Task=`b568a927-c22b-4892-a84a-79e3ae85ae61`::moving from state preparing -> state finished >Thread-26602::DEBUG::2012-12-02 14:12:13,365::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26602::DEBUG::2012-12-02 14:12:13,366::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26602::DEBUG::2012-12-02 14:12:13,366::task::957::TaskManager.Task::(_decref) Task=`b568a927-c22b-4892-a84a-79e3ae85ae61`::ref 0 aborting False >Thread-26604::DEBUG::2012-12-02 14:12:14,854::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26604::DEBUG::2012-12-02 14:12:14,854::task::568::TaskManager.Task::(_updateState) Task=`4724ae24-734a-4038-830f-b19a1b9442ca`::moving from state init -> state preparing >Thread-26604::INFO::2012-12-02 14:12:14,854::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26604::INFO::2012-12-02 14:12:14,854::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26604::DEBUG::2012-12-02 14:12:14,855::task::1151::TaskManager.Task::(prepare) Task=`4724ae24-734a-4038-830f-b19a1b9442ca`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26604::DEBUG::2012-12-02 14:12:14,855::task::568::TaskManager.Task::(_updateState) Task=`4724ae24-734a-4038-830f-b19a1b9442ca`::moving from state preparing -> state finished >Thread-26604::DEBUG::2012-12-02 14:12:14,855::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26604::DEBUG::2012-12-02 14:12:14,855::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26604::DEBUG::2012-12-02 14:12:14,855::task::957::TaskManager.Task::(_decref) Task=`4724ae24-734a-4038-830f-b19a1b9442ca`::ref 0 aborting False >Thread-26605::DEBUG::2012-12-02 14:12:14,868::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26605::DEBUG::2012-12-02 14:12:14,868::task::568::TaskManager.Task::(_updateState) Task=`590673c2-edd3-4a49-aa16-be6480b70932`::moving from state init -> state preparing >Thread-26605::INFO::2012-12-02 14:12:14,868::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26605::DEBUG::2012-12-02 14:12:14,869::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a5f691f-4230-4e31-b44c-8a5d83f1fb5c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26605::DEBUG::2012-12-02 14:12:14,869::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26605::DEBUG::2012-12-02 14:12:14,869::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26605::DEBUG::2012-12-02 14:12:14,869::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a5f691f-4230-4e31-b44c-8a5d83f1fb5c`::Granted request >Thread-26605::DEBUG::2012-12-02 14:12:14,869::task::794::TaskManager.Task::(resourceAcquired) Task=`590673c2-edd3-4a49-aa16-be6480b70932`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26605::DEBUG::2012-12-02 14:12:14,870::task::957::TaskManager.Task::(_decref) Task=`590673c2-edd3-4a49-aa16-be6480b70932`::ref 1 aborting False >Thread-26605::INFO::2012-12-02 14:12:14,876::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26605::DEBUG::2012-12-02 14:12:14,876::task::1151::TaskManager.Task::(prepare) Task=`590673c2-edd3-4a49-aa16-be6480b70932`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26605::DEBUG::2012-12-02 14:12:14,876::task::568::TaskManager.Task::(_updateState) Task=`590673c2-edd3-4a49-aa16-be6480b70932`::moving from state preparing -> state finished >Thread-26605::DEBUG::2012-12-02 14:12:14,876::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26605::DEBUG::2012-12-02 14:12:14,876::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26605::DEBUG::2012-12-02 14:12:14,876::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26605::DEBUG::2012-12-02 14:12:14,876::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26605::DEBUG::2012-12-02 14:12:14,876::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26605::DEBUG::2012-12-02 14:12:14,877::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26605::DEBUG::2012-12-02 14:12:14,877::task::957::TaskManager.Task::(_decref) Task=`590673c2-edd3-4a49-aa16-be6480b70932`::ref 0 aborting False >Thread-26610::DEBUG::2012-12-02 14:12:23,428::task::568::TaskManager.Task::(_updateState) Task=`a67ff1e2-e801-466e-aa14-491bf4dadb7d`::moving from state init -> state preparing >Thread-26610::INFO::2012-12-02 14:12:23,429::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26610::INFO::2012-12-02 14:12:23,429::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00846982002258', 'lastCheck': 1354450335.3556731, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0029981136322', 'lastCheck': 1354450340.5291591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00327014923096', 'lastCheck': 1354450333.4501419, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00689601898193', 'lastCheck': 1354450340.552803, 'code': 0, 'valid': True}} >Thread-26610::DEBUG::2012-12-02 14:12:23,429::task::1151::TaskManager.Task::(prepare) Task=`a67ff1e2-e801-466e-aa14-491bf4dadb7d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00846982002258', 'lastCheck': 1354450335.3556731, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0029981136322', 'lastCheck': 1354450340.5291591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00327014923096', 'lastCheck': 1354450333.4501419, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00689601898193', 'lastCheck': 1354450340.552803, 'code': 0, 'valid': True}} >Thread-26610::DEBUG::2012-12-02 14:12:23,429::task::568::TaskManager.Task::(_updateState) Task=`a67ff1e2-e801-466e-aa14-491bf4dadb7d`::moving from state preparing -> state finished >Thread-26610::DEBUG::2012-12-02 14:12:23,429::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26610::DEBUG::2012-12-02 14:12:23,429::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26610::DEBUG::2012-12-02 14:12:23,429::task::957::TaskManager.Task::(_decref) Task=`a67ff1e2-e801-466e-aa14-491bf4dadb7d`::ref 0 aborting False >Thread-26612::DEBUG::2012-12-02 14:12:25,019::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26612::DEBUG::2012-12-02 14:12:25,020::task::568::TaskManager.Task::(_updateState) Task=`2fd9a6ef-2bba-446b-b2f2-a76a21d1c4d1`::moving from state init -> state preparing >Thread-26612::INFO::2012-12-02 14:12:25,020::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26612::INFO::2012-12-02 14:12:25,020::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26612::DEBUG::2012-12-02 14:12:25,020::task::1151::TaskManager.Task::(prepare) Task=`2fd9a6ef-2bba-446b-b2f2-a76a21d1c4d1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26612::DEBUG::2012-12-02 14:12:25,020::task::568::TaskManager.Task::(_updateState) Task=`2fd9a6ef-2bba-446b-b2f2-a76a21d1c4d1`::moving from state preparing -> state finished >Thread-26612::DEBUG::2012-12-02 14:12:25,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26612::DEBUG::2012-12-02 14:12:25,020::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26612::DEBUG::2012-12-02 14:12:25,021::task::957::TaskManager.Task::(_decref) Task=`2fd9a6ef-2bba-446b-b2f2-a76a21d1c4d1`::ref 0 aborting False >Thread-26613::DEBUG::2012-12-02 14:12:25,023::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26613::DEBUG::2012-12-02 14:12:25,023::task::568::TaskManager.Task::(_updateState) Task=`0a17349e-f4d6-4c6e-99a7-460a16670de2`::moving from state init -> state preparing >Thread-26613::INFO::2012-12-02 14:12:25,024::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26613::DEBUG::2012-12-02 14:12:25,024::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5c24f1e0-6989-4dd6-986a-dabd2cdd4f3e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26613::DEBUG::2012-12-02 14:12:25,024::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26613::DEBUG::2012-12-02 14:12:25,024::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26613::DEBUG::2012-12-02 14:12:25,024::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5c24f1e0-6989-4dd6-986a-dabd2cdd4f3e`::Granted request >Thread-26613::DEBUG::2012-12-02 14:12:25,025::task::794::TaskManager.Task::(resourceAcquired) Task=`0a17349e-f4d6-4c6e-99a7-460a16670de2`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26613::DEBUG::2012-12-02 14:12:25,025::task::957::TaskManager.Task::(_decref) Task=`0a17349e-f4d6-4c6e-99a7-460a16670de2`::ref 1 aborting False >Thread-26613::INFO::2012-12-02 14:12:25,029::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26613::DEBUG::2012-12-02 14:12:25,029::task::1151::TaskManager.Task::(prepare) Task=`0a17349e-f4d6-4c6e-99a7-460a16670de2`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26613::DEBUG::2012-12-02 14:12:25,029::task::568::TaskManager.Task::(_updateState) Task=`0a17349e-f4d6-4c6e-99a7-460a16670de2`::moving from state preparing -> state finished >Thread-26613::DEBUG::2012-12-02 14:12:25,030::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26613::DEBUG::2012-12-02 14:12:25,030::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26613::DEBUG::2012-12-02 14:12:25,030::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26613::DEBUG::2012-12-02 14:12:25,030::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26613::DEBUG::2012-12-02 14:12:25,030::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26613::DEBUG::2012-12-02 14:12:25,030::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26613::DEBUG::2012-12-02 14:12:25,030::task::957::TaskManager.Task::(_decref) Task=`0a17349e-f4d6-4c6e-99a7-460a16670de2`::ref 0 aborting False >Thread-26618::DEBUG::2012-12-02 14:12:33,527::task::568::TaskManager.Task::(_updateState) Task=`38d218f1-39ef-47a1-9003-9dda455d58af`::moving from state init -> state preparing >Thread-26618::INFO::2012-12-02 14:12:33,527::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26618::INFO::2012-12-02 14:12:33,528::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00702214241028', 'lastCheck': 1354450345.3724501, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0037088394165', 'lastCheck': 1354450350.5435281, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00496220588684', 'lastCheck': 1354450353.4799809, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00718784332275', 'lastCheck': 1354450350.567672, 'code': 0, 'valid': True}} >Thread-26618::DEBUG::2012-12-02 14:12:33,528::task::1151::TaskManager.Task::(prepare) Task=`38d218f1-39ef-47a1-9003-9dda455d58af`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00702214241028', 'lastCheck': 1354450345.3724501, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0037088394165', 'lastCheck': 1354450350.5435281, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00496220588684', 'lastCheck': 1354450353.4799809, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00718784332275', 'lastCheck': 1354450350.567672, 'code': 0, 'valid': True}} >Thread-26618::DEBUG::2012-12-02 14:12:33,528::task::568::TaskManager.Task::(_updateState) Task=`38d218f1-39ef-47a1-9003-9dda455d58af`::moving from state preparing -> state finished >Thread-26618::DEBUG::2012-12-02 14:12:33,528::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26618::DEBUG::2012-12-02 14:12:33,528::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26618::DEBUG::2012-12-02 14:12:33,528::task::957::TaskManager.Task::(_decref) Task=`38d218f1-39ef-47a1-9003-9dda455d58af`::ref 0 aborting False >Thread-26620::DEBUG::2012-12-02 14:12:35,157::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26620::DEBUG::2012-12-02 14:12:35,157::task::568::TaskManager.Task::(_updateState) Task=`b7a09712-4192-47f4-be3f-e403ed983dca`::moving from state init -> state preparing >Thread-26620::INFO::2012-12-02 14:12:35,157::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26620::INFO::2012-12-02 14:12:35,158::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26620::DEBUG::2012-12-02 14:12:35,158::task::1151::TaskManager.Task::(prepare) Task=`b7a09712-4192-47f4-be3f-e403ed983dca`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26620::DEBUG::2012-12-02 14:12:35,158::task::568::TaskManager.Task::(_updateState) Task=`b7a09712-4192-47f4-be3f-e403ed983dca`::moving from state preparing -> state finished >Thread-26620::DEBUG::2012-12-02 14:12:35,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26620::DEBUG::2012-12-02 14:12:35,158::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26620::DEBUG::2012-12-02 14:12:35,158::task::957::TaskManager.Task::(_decref) Task=`b7a09712-4192-47f4-be3f-e403ed983dca`::ref 0 aborting False >Thread-26621::DEBUG::2012-12-02 14:12:35,161::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26621::DEBUG::2012-12-02 14:12:35,161::task::568::TaskManager.Task::(_updateState) Task=`6b480b57-fd42-4b0b-ac59-d6ec0c64746d`::moving from state init -> state preparing >Thread-26621::INFO::2012-12-02 14:12:35,161::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26621::DEBUG::2012-12-02 14:12:35,161::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3b436ce9-7d63-4732-ab6a-3d0e3702f1bd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26621::DEBUG::2012-12-02 14:12:35,162::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26621::DEBUG::2012-12-02 14:12:35,162::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26621::DEBUG::2012-12-02 14:12:35,162::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3b436ce9-7d63-4732-ab6a-3d0e3702f1bd`::Granted request >Thread-26621::DEBUG::2012-12-02 14:12:35,162::task::794::TaskManager.Task::(resourceAcquired) Task=`6b480b57-fd42-4b0b-ac59-d6ec0c64746d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26621::DEBUG::2012-12-02 14:12:35,162::task::957::TaskManager.Task::(_decref) Task=`6b480b57-fd42-4b0b-ac59-d6ec0c64746d`::ref 1 aborting False >Thread-26621::INFO::2012-12-02 14:12:35,168::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26621::DEBUG::2012-12-02 14:12:35,168::task::1151::TaskManager.Task::(prepare) Task=`6b480b57-fd42-4b0b-ac59-d6ec0c64746d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26621::DEBUG::2012-12-02 14:12:35,168::task::568::TaskManager.Task::(_updateState) Task=`6b480b57-fd42-4b0b-ac59-d6ec0c64746d`::moving from state preparing -> state finished >Thread-26621::DEBUG::2012-12-02 14:12:35,168::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26621::DEBUG::2012-12-02 14:12:35,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26621::DEBUG::2012-12-02 14:12:35,168::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26621::DEBUG::2012-12-02 14:12:35,169::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26621::DEBUG::2012-12-02 14:12:35,169::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26621::DEBUG::2012-12-02 14:12:35,169::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26621::DEBUG::2012-12-02 14:12:35,169::task::957::TaskManager.Task::(_decref) Task=`6b480b57-fd42-4b0b-ac59-d6ec0c64746d`::ref 0 aborting False >Thread-26626::DEBUG::2012-12-02 14:12:43,655::task::568::TaskManager.Task::(_updateState) Task=`f9cd022f-2646-473f-8d30-715a7ada3ece`::moving from state init -> state preparing >Thread-26626::INFO::2012-12-02 14:12:43,655::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26626::INFO::2012-12-02 14:12:43,656::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00825595855713', 'lastCheck': 1354450355.392097, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00245809555054', 'lastCheck': 1354450360.5591049, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00133800506592', 'lastCheck': 1354450363.487561, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00850296020508', 'lastCheck': 1354450360.5872729, 'code': 0, 'valid': True}} >Thread-26626::DEBUG::2012-12-02 14:12:43,656::task::1151::TaskManager.Task::(prepare) Task=`f9cd022f-2646-473f-8d30-715a7ada3ece`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00825595855713', 'lastCheck': 1354450355.392097, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00245809555054', 'lastCheck': 1354450360.5591049, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00133800506592', 'lastCheck': 1354450363.487561, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00850296020508', 'lastCheck': 1354450360.5872729, 'code': 0, 'valid': True}} >Thread-26626::DEBUG::2012-12-02 14:12:43,656::task::568::TaskManager.Task::(_updateState) Task=`f9cd022f-2646-473f-8d30-715a7ada3ece`::moving from state preparing -> state finished >Thread-26626::DEBUG::2012-12-02 14:12:43,656::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26626::DEBUG::2012-12-02 14:12:43,656::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26626::DEBUG::2012-12-02 14:12:43,656::task::957::TaskManager.Task::(_decref) Task=`f9cd022f-2646-473f-8d30-715a7ada3ece`::ref 0 aborting False >Thread-26628::DEBUG::2012-12-02 14:12:45,361::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26628::DEBUG::2012-12-02 14:12:45,362::task::568::TaskManager.Task::(_updateState) Task=`1e3c82b4-309a-4677-b86e-5f754d4038c9`::moving from state init -> state preparing >Thread-26628::INFO::2012-12-02 14:12:45,362::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26628::INFO::2012-12-02 14:12:45,362::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26628::DEBUG::2012-12-02 14:12:45,362::task::1151::TaskManager.Task::(prepare) Task=`1e3c82b4-309a-4677-b86e-5f754d4038c9`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26628::DEBUG::2012-12-02 14:12:45,362::task::568::TaskManager.Task::(_updateState) Task=`1e3c82b4-309a-4677-b86e-5f754d4038c9`::moving from state preparing -> state finished >Thread-26628::DEBUG::2012-12-02 14:12:45,363::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26628::DEBUG::2012-12-02 14:12:45,363::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26628::DEBUG::2012-12-02 14:12:45,363::task::957::TaskManager.Task::(_decref) Task=`1e3c82b4-309a-4677-b86e-5f754d4038c9`::ref 0 aborting False >Thread-26629::DEBUG::2012-12-02 14:12:45,365::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26629::DEBUG::2012-12-02 14:12:45,366::task::568::TaskManager.Task::(_updateState) Task=`6cbd6ea4-532c-400b-a019-36a9265d8136`::moving from state init -> state preparing >Thread-26629::INFO::2012-12-02 14:12:45,366::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26629::DEBUG::2012-12-02 14:12:45,366::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`33bb2ca8-844d-4cef-93bf-200b701c68ae`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26629::DEBUG::2012-12-02 14:12:45,366::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26629::DEBUG::2012-12-02 14:12:45,366::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26629::DEBUG::2012-12-02 14:12:45,366::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`33bb2ca8-844d-4cef-93bf-200b701c68ae`::Granted request >Thread-26629::DEBUG::2012-12-02 14:12:45,367::task::794::TaskManager.Task::(resourceAcquired) Task=`6cbd6ea4-532c-400b-a019-36a9265d8136`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26629::DEBUG::2012-12-02 14:12:45,367::task::957::TaskManager.Task::(_decref) Task=`6cbd6ea4-532c-400b-a019-36a9265d8136`::ref 1 aborting False >Thread-26629::INFO::2012-12-02 14:12:45,373::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26629::DEBUG::2012-12-02 14:12:45,373::task::1151::TaskManager.Task::(prepare) Task=`6cbd6ea4-532c-400b-a019-36a9265d8136`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26629::DEBUG::2012-12-02 14:12:45,374::task::568::TaskManager.Task::(_updateState) Task=`6cbd6ea4-532c-400b-a019-36a9265d8136`::moving from state preparing -> state finished >Thread-26629::DEBUG::2012-12-02 14:12:45,374::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26629::DEBUG::2012-12-02 14:12:45,374::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26629::DEBUG::2012-12-02 14:12:45,374::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26629::DEBUG::2012-12-02 14:12:45,374::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26629::DEBUG::2012-12-02 14:12:45,374::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26629::DEBUG::2012-12-02 14:12:45,374::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26629::DEBUG::2012-12-02 14:12:45,375::task::957::TaskManager.Task::(_decref) Task=`6cbd6ea4-532c-400b-a019-36a9265d8136`::ref 0 aborting False >Thread-26634::DEBUG::2012-12-02 14:12:53,725::task::568::TaskManager.Task::(_updateState) Task=`4e6aa98c-b867-4a2c-ac45-6cdfd953b854`::moving from state init -> state preparing >Thread-26634::INFO::2012-12-02 14:12:53,725::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26634::INFO::2012-12-02 14:12:53,725::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00930500030518', 'lastCheck': 1354450365.409348, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00279903411865', 'lastCheck': 1354450370.5718131, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00284719467163', 'lastCheck': 1354450373.499718, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00699305534363', 'lastCheck': 1354450370.602752, 'code': 0, 'valid': True}} >Thread-26634::DEBUG::2012-12-02 14:12:53,725::task::1151::TaskManager.Task::(prepare) Task=`4e6aa98c-b867-4a2c-ac45-6cdfd953b854`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00930500030518', 'lastCheck': 1354450365.409348, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00279903411865', 'lastCheck': 1354450370.5718131, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00284719467163', 'lastCheck': 1354450373.499718, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00699305534363', 'lastCheck': 1354450370.602752, 'code': 0, 'valid': True}} >Thread-26634::DEBUG::2012-12-02 14:12:53,726::task::568::TaskManager.Task::(_updateState) Task=`4e6aa98c-b867-4a2c-ac45-6cdfd953b854`::moving from state preparing -> state finished >Thread-26634::DEBUG::2012-12-02 14:12:53,726::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26634::DEBUG::2012-12-02 14:12:53,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26634::DEBUG::2012-12-02 14:12:53,726::task::957::TaskManager.Task::(_decref) Task=`4e6aa98c-b867-4a2c-ac45-6cdfd953b854`::ref 0 aborting False >Thread-26636::DEBUG::2012-12-02 14:12:55,515::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26636::DEBUG::2012-12-02 14:12:55,515::task::568::TaskManager.Task::(_updateState) Task=`2b450c1c-ac8e-43aa-9195-760ee9c808d7`::moving from state init -> state preparing >Thread-26636::INFO::2012-12-02 14:12:55,515::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26636::INFO::2012-12-02 14:12:55,516::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26636::DEBUG::2012-12-02 14:12:55,516::task::1151::TaskManager.Task::(prepare) Task=`2b450c1c-ac8e-43aa-9195-760ee9c808d7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26636::DEBUG::2012-12-02 14:12:55,516::task::568::TaskManager.Task::(_updateState) Task=`2b450c1c-ac8e-43aa-9195-760ee9c808d7`::moving from state preparing -> state finished >Thread-26636::DEBUG::2012-12-02 14:12:55,516::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26636::DEBUG::2012-12-02 14:12:55,516::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26636::DEBUG::2012-12-02 14:12:55,516::task::957::TaskManager.Task::(_decref) Task=`2b450c1c-ac8e-43aa-9195-760ee9c808d7`::ref 0 aborting False >Thread-26637::DEBUG::2012-12-02 14:12:55,519::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26637::DEBUG::2012-12-02 14:12:55,519::task::568::TaskManager.Task::(_updateState) Task=`49325f5f-e330-4548-8c8e-3ede1b3bbeaa`::moving from state init -> state preparing >Thread-26637::INFO::2012-12-02 14:12:55,519::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26637::DEBUG::2012-12-02 14:12:55,519::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`40ca1d5b-7cf0-4a51-992c-b2cabbf94412`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26637::DEBUG::2012-12-02 14:12:55,520::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26637::DEBUG::2012-12-02 14:12:55,520::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26637::DEBUG::2012-12-02 14:12:55,520::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`40ca1d5b-7cf0-4a51-992c-b2cabbf94412`::Granted request >Thread-26637::DEBUG::2012-12-02 14:12:55,521::task::794::TaskManager.Task::(resourceAcquired) Task=`49325f5f-e330-4548-8c8e-3ede1b3bbeaa`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26637::DEBUG::2012-12-02 14:12:55,521::task::957::TaskManager.Task::(_decref) Task=`49325f5f-e330-4548-8c8e-3ede1b3bbeaa`::ref 1 aborting False >Thread-26637::INFO::2012-12-02 14:12:55,527::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26637::DEBUG::2012-12-02 14:12:55,527::task::1151::TaskManager.Task::(prepare) Task=`49325f5f-e330-4548-8c8e-3ede1b3bbeaa`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26637::DEBUG::2012-12-02 14:12:55,527::task::568::TaskManager.Task::(_updateState) Task=`49325f5f-e330-4548-8c8e-3ede1b3bbeaa`::moving from state preparing -> state finished >Thread-26637::DEBUG::2012-12-02 14:12:55,527::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26637::DEBUG::2012-12-02 14:12:55,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26637::DEBUG::2012-12-02 14:12:55,527::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26637::DEBUG::2012-12-02 14:12:55,527::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26637::DEBUG::2012-12-02 14:12:55,527::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26637::DEBUG::2012-12-02 14:12:55,527::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26637::DEBUG::2012-12-02 14:12:55,527::task::957::TaskManager.Task::(_decref) Task=`49325f5f-e330-4548-8c8e-3ede1b3bbeaa`::ref 0 aborting False >Thread-26642::DEBUG::2012-12-02 14:13:03,797::task::568::TaskManager.Task::(_updateState) Task=`429c79f3-78ad-4026-b703-c20b2ff453a7`::moving from state init -> state preparing >Thread-26642::INFO::2012-12-02 14:13:03,797::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26642::INFO::2012-12-02 14:13:03,798::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0192439556122', 'lastCheck': 1354450375.4415319, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00215911865234', 'lastCheck': 1354450380.584255, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00168299674988', 'lastCheck': 1354450383.5099559, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778198242188', 'lastCheck': 1354450380.619606, 'code': 0, 'valid': True}} >Thread-26642::DEBUG::2012-12-02 14:13:03,798::task::1151::TaskManager.Task::(prepare) Task=`429c79f3-78ad-4026-b703-c20b2ff453a7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0192439556122', 'lastCheck': 1354450375.4415319, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00215911865234', 'lastCheck': 1354450380.584255, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00168299674988', 'lastCheck': 1354450383.5099559, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778198242188', 'lastCheck': 1354450380.619606, 'code': 0, 'valid': True}} >Thread-26642::DEBUG::2012-12-02 14:13:03,798::task::568::TaskManager.Task::(_updateState) Task=`429c79f3-78ad-4026-b703-c20b2ff453a7`::moving from state preparing -> state finished >Thread-26642::DEBUG::2012-12-02 14:13:03,798::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26642::DEBUG::2012-12-02 14:13:03,798::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26642::DEBUG::2012-12-02 14:13:03,798::task::957::TaskManager.Task::(_decref) Task=`429c79f3-78ad-4026-b703-c20b2ff453a7`::ref 0 aborting False >Thread-26644::DEBUG::2012-12-02 14:13:05,635::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26644::DEBUG::2012-12-02 14:13:05,636::task::568::TaskManager.Task::(_updateState) Task=`ba34634c-c783-4ad8-afd6-594822a38386`::moving from state init -> state preparing >Thread-26644::INFO::2012-12-02 14:13:05,636::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26644::INFO::2012-12-02 14:13:05,636::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26644::DEBUG::2012-12-02 14:13:05,636::task::1151::TaskManager.Task::(prepare) Task=`ba34634c-c783-4ad8-afd6-594822a38386`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26644::DEBUG::2012-12-02 14:13:05,636::task::568::TaskManager.Task::(_updateState) Task=`ba34634c-c783-4ad8-afd6-594822a38386`::moving from state preparing -> state finished >Thread-26644::DEBUG::2012-12-02 14:13:05,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26644::DEBUG::2012-12-02 14:13:05,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26644::DEBUG::2012-12-02 14:13:05,637::task::957::TaskManager.Task::(_decref) Task=`ba34634c-c783-4ad8-afd6-594822a38386`::ref 0 aborting False >Thread-26645::DEBUG::2012-12-02 14:13:05,639::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26645::DEBUG::2012-12-02 14:13:05,640::task::568::TaskManager.Task::(_updateState) Task=`50c1f55a-dff2-4b7c-a316-52ad84b5e9b1`::moving from state init -> state preparing >Thread-26645::INFO::2012-12-02 14:13:05,640::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26645::DEBUG::2012-12-02 14:13:05,640::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b3f80391-6734-4318-8ec4-a3ad7740eaef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26645::DEBUG::2012-12-02 14:13:05,640::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26645::DEBUG::2012-12-02 14:13:05,640::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26645::DEBUG::2012-12-02 14:13:05,640::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b3f80391-6734-4318-8ec4-a3ad7740eaef`::Granted request >Thread-26645::DEBUG::2012-12-02 14:13:05,641::task::794::TaskManager.Task::(resourceAcquired) Task=`50c1f55a-dff2-4b7c-a316-52ad84b5e9b1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26645::DEBUG::2012-12-02 14:13:05,641::task::957::TaskManager.Task::(_decref) Task=`50c1f55a-dff2-4b7c-a316-52ad84b5e9b1`::ref 1 aborting False >Thread-26645::INFO::2012-12-02 14:13:05,647::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26645::DEBUG::2012-12-02 14:13:05,647::task::1151::TaskManager.Task::(prepare) Task=`50c1f55a-dff2-4b7c-a316-52ad84b5e9b1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26645::DEBUG::2012-12-02 14:13:05,647::task::568::TaskManager.Task::(_updateState) Task=`50c1f55a-dff2-4b7c-a316-52ad84b5e9b1`::moving from state preparing -> state finished >Thread-26645::DEBUG::2012-12-02 14:13:05,647::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26645::DEBUG::2012-12-02 14:13:05,647::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26645::DEBUG::2012-12-02 14:13:05,647::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26645::DEBUG::2012-12-02 14:13:05,647::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26645::DEBUG::2012-12-02 14:13:05,647::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26645::DEBUG::2012-12-02 14:13:05,647::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26645::DEBUG::2012-12-02 14:13:05,647::task::957::TaskManager.Task::(_decref) Task=`50c1f55a-dff2-4b7c-a316-52ad84b5e9b1`::ref 0 aborting False >Thread-26650::DEBUG::2012-12-02 14:13:13,896::task::568::TaskManager.Task::(_updateState) Task=`cf6c80fb-3f4c-4afd-8fa0-9303f0797a21`::moving from state init -> state preparing >Thread-26650::INFO::2012-12-02 14:13:13,896::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26650::INFO::2012-12-02 14:13:13,896::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830006599426', 'lastCheck': 1354450385.4585979, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00502014160156', 'lastCheck': 1354450390.6013319, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00346779823303', 'lastCheck': 1354450393.5250421, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00764513015747', 'lastCheck': 1354450390.636359, 'code': 0, 'valid': True}} >Thread-26650::DEBUG::2012-12-02 14:13:13,896::task::1151::TaskManager.Task::(prepare) Task=`cf6c80fb-3f4c-4afd-8fa0-9303f0797a21`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830006599426', 'lastCheck': 1354450385.4585979, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00502014160156', 'lastCheck': 1354450390.6013319, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00346779823303', 'lastCheck': 1354450393.5250421, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00764513015747', 'lastCheck': 1354450390.636359, 'code': 0, 'valid': True}} >Thread-26650::DEBUG::2012-12-02 14:13:13,896::task::568::TaskManager.Task::(_updateState) Task=`cf6c80fb-3f4c-4afd-8fa0-9303f0797a21`::moving from state preparing -> state finished >Thread-26650::DEBUG::2012-12-02 14:13:13,897::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26650::DEBUG::2012-12-02 14:13:13,897::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26650::DEBUG::2012-12-02 14:13:13,897::task::957::TaskManager.Task::(_decref) Task=`cf6c80fb-3f4c-4afd-8fa0-9303f0797a21`::ref 0 aborting False >Thread-26652::DEBUG::2012-12-02 14:13:15,776::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26652::DEBUG::2012-12-02 14:13:15,777::task::568::TaskManager.Task::(_updateState) Task=`3023967e-3152-4a8d-a67a-455ec5872ae2`::moving from state init -> state preparing >Thread-26652::INFO::2012-12-02 14:13:15,777::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26652::INFO::2012-12-02 14:13:15,777::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26652::DEBUG::2012-12-02 14:13:15,777::task::1151::TaskManager.Task::(prepare) Task=`3023967e-3152-4a8d-a67a-455ec5872ae2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26652::DEBUG::2012-12-02 14:13:15,778::task::568::TaskManager.Task::(_updateState) Task=`3023967e-3152-4a8d-a67a-455ec5872ae2`::moving from state preparing -> state finished >Thread-26652::DEBUG::2012-12-02 14:13:15,778::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26652::DEBUG::2012-12-02 14:13:15,778::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26652::DEBUG::2012-12-02 14:13:15,778::task::957::TaskManager.Task::(_decref) Task=`3023967e-3152-4a8d-a67a-455ec5872ae2`::ref 0 aborting False >Thread-26653::DEBUG::2012-12-02 14:13:15,780::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26653::DEBUG::2012-12-02 14:13:15,781::task::568::TaskManager.Task::(_updateState) Task=`bf118a0d-62b7-4e2e-a83f-dccc38d54454`::moving from state init -> state preparing >Thread-26653::INFO::2012-12-02 14:13:15,781::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26653::DEBUG::2012-12-02 14:13:15,781::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ff7b96c1-90e6-41d4-afd8-2949675b03bd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26653::DEBUG::2012-12-02 14:13:15,781::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26653::DEBUG::2012-12-02 14:13:15,781::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26653::DEBUG::2012-12-02 14:13:15,782::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ff7b96c1-90e6-41d4-afd8-2949675b03bd`::Granted request >Thread-26653::DEBUG::2012-12-02 14:13:15,782::task::794::TaskManager.Task::(resourceAcquired) Task=`bf118a0d-62b7-4e2e-a83f-dccc38d54454`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26653::DEBUG::2012-12-02 14:13:15,782::task::957::TaskManager.Task::(_decref) Task=`bf118a0d-62b7-4e2e-a83f-dccc38d54454`::ref 1 aborting False >Thread-26653::INFO::2012-12-02 14:13:15,788::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26653::DEBUG::2012-12-02 14:13:15,788::task::1151::TaskManager.Task::(prepare) Task=`bf118a0d-62b7-4e2e-a83f-dccc38d54454`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26653::DEBUG::2012-12-02 14:13:15,788::task::568::TaskManager.Task::(_updateState) Task=`bf118a0d-62b7-4e2e-a83f-dccc38d54454`::moving from state preparing -> state finished >Thread-26653::DEBUG::2012-12-02 14:13:15,788::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26653::DEBUG::2012-12-02 14:13:15,788::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26653::DEBUG::2012-12-02 14:13:15,788::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26653::DEBUG::2012-12-02 14:13:15,788::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26653::DEBUG::2012-12-02 14:13:15,788::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26653::DEBUG::2012-12-02 14:13:15,788::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26653::DEBUG::2012-12-02 14:13:15,789::task::957::TaskManager.Task::(_decref) Task=`bf118a0d-62b7-4e2e-a83f-dccc38d54454`::ref 0 aborting False >Thread-26658::DEBUG::2012-12-02 14:13:23,978::task::568::TaskManager.Task::(_updateState) Task=`c12a96e3-f866-4c14-8698-705ab67c899a`::moving from state init -> state preparing >Thread-26658::INFO::2012-12-02 14:13:23,978::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26658::INFO::2012-12-02 14:13:23,978::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104730129242', 'lastCheck': 1354450395.478338, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00229001045227', 'lastCheck': 1354450400.6146901, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00248003005981', 'lastCheck': 1354450403.537503, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00663685798645', 'lastCheck': 1354450400.651917, 'code': 0, 'valid': True}} >Thread-26658::DEBUG::2012-12-02 14:13:23,979::task::1151::TaskManager.Task::(prepare) Task=`c12a96e3-f866-4c14-8698-705ab67c899a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104730129242', 'lastCheck': 1354450395.478338, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00229001045227', 'lastCheck': 1354450400.6146901, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00248003005981', 'lastCheck': 1354450403.537503, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00663685798645', 'lastCheck': 1354450400.651917, 'code': 0, 'valid': True}} >Thread-26658::DEBUG::2012-12-02 14:13:23,979::task::568::TaskManager.Task::(_updateState) Task=`c12a96e3-f866-4c14-8698-705ab67c899a`::moving from state preparing -> state finished >Thread-26658::DEBUG::2012-12-02 14:13:23,979::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26658::DEBUG::2012-12-02 14:13:23,979::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26658::DEBUG::2012-12-02 14:13:23,979::task::957::TaskManager.Task::(_decref) Task=`c12a96e3-f866-4c14-8698-705ab67c899a`::ref 0 aborting False >Thread-26660::DEBUG::2012-12-02 14:13:25,927::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26660::DEBUG::2012-12-02 14:13:25,927::task::568::TaskManager.Task::(_updateState) Task=`d25d05bf-e922-4c8b-8d7e-858b56482c89`::moving from state init -> state preparing >Thread-26660::INFO::2012-12-02 14:13:25,927::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26660::INFO::2012-12-02 14:13:25,928::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26660::DEBUG::2012-12-02 14:13:25,928::task::1151::TaskManager.Task::(prepare) Task=`d25d05bf-e922-4c8b-8d7e-858b56482c89`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26660::DEBUG::2012-12-02 14:13:25,928::task::568::TaskManager.Task::(_updateState) Task=`d25d05bf-e922-4c8b-8d7e-858b56482c89`::moving from state preparing -> state finished >Thread-26660::DEBUG::2012-12-02 14:13:25,928::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26660::DEBUG::2012-12-02 14:13:25,928::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26660::DEBUG::2012-12-02 14:13:25,928::task::957::TaskManager.Task::(_decref) Task=`d25d05bf-e922-4c8b-8d7e-858b56482c89`::ref 0 aborting False >Thread-26661::DEBUG::2012-12-02 14:13:25,931::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26661::DEBUG::2012-12-02 14:13:25,931::task::568::TaskManager.Task::(_updateState) Task=`2641ebe6-2dfe-4ff3-8943-27ee5e66405d`::moving from state init -> state preparing >Thread-26661::INFO::2012-12-02 14:13:25,931::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26661::DEBUG::2012-12-02 14:13:25,931::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7d3ba350-8ef7-441d-8e33-41f9f7b1af66`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26661::DEBUG::2012-12-02 14:13:25,932::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26661::DEBUG::2012-12-02 14:13:25,932::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26661::DEBUG::2012-12-02 14:13:25,932::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7d3ba350-8ef7-441d-8e33-41f9f7b1af66`::Granted request >Thread-26661::DEBUG::2012-12-02 14:13:25,932::task::794::TaskManager.Task::(resourceAcquired) Task=`2641ebe6-2dfe-4ff3-8943-27ee5e66405d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26661::DEBUG::2012-12-02 14:13:25,932::task::957::TaskManager.Task::(_decref) Task=`2641ebe6-2dfe-4ff3-8943-27ee5e66405d`::ref 1 aborting False >Thread-26661::INFO::2012-12-02 14:13:25,938::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26661::DEBUG::2012-12-02 14:13:25,938::task::1151::TaskManager.Task::(prepare) Task=`2641ebe6-2dfe-4ff3-8943-27ee5e66405d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26661::DEBUG::2012-12-02 14:13:25,938::task::568::TaskManager.Task::(_updateState) Task=`2641ebe6-2dfe-4ff3-8943-27ee5e66405d`::moving from state preparing -> state finished >Thread-26661::DEBUG::2012-12-02 14:13:25,938::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26661::DEBUG::2012-12-02 14:13:25,938::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26661::DEBUG::2012-12-02 14:13:25,938::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26661::DEBUG::2012-12-02 14:13:25,938::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26661::DEBUG::2012-12-02 14:13:25,938::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26661::DEBUG::2012-12-02 14:13:25,939::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26661::DEBUG::2012-12-02 14:13:25,939::task::957::TaskManager.Task::(_decref) Task=`2641ebe6-2dfe-4ff3-8943-27ee5e66405d`::ref 0 aborting False >Thread-26666::DEBUG::2012-12-02 14:13:34,133::task::568::TaskManager.Task::(_updateState) Task=`a6c102df-a160-4e5d-9eae-c220df5b037a`::moving from state init -> state preparing >Thread-26666::INFO::2012-12-02 14:13:34,134::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26666::INFO::2012-12-02 14:13:34,134::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0081479549408', 'lastCheck': 1354450405.496347, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00319385528564', 'lastCheck': 1354450410.627553, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00309300422668', 'lastCheck': 1354450413.5513201, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00644588470459', 'lastCheck': 1354450410.6670749, 'code': 0, 'valid': True}} >Thread-26666::DEBUG::2012-12-02 14:13:34,134::task::1151::TaskManager.Task::(prepare) Task=`a6c102df-a160-4e5d-9eae-c220df5b037a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0081479549408', 'lastCheck': 1354450405.496347, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00319385528564', 'lastCheck': 1354450410.627553, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00309300422668', 'lastCheck': 1354450413.5513201, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00644588470459', 'lastCheck': 1354450410.6670749, 'code': 0, 'valid': True}} >Thread-26666::DEBUG::2012-12-02 14:13:34,134::task::568::TaskManager.Task::(_updateState) Task=`a6c102df-a160-4e5d-9eae-c220df5b037a`::moving from state preparing -> state finished >Thread-26666::DEBUG::2012-12-02 14:13:34,134::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26666::DEBUG::2012-12-02 14:13:34,134::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26666::DEBUG::2012-12-02 14:13:34,135::task::957::TaskManager.Task::(_decref) Task=`a6c102df-a160-4e5d-9eae-c220df5b037a`::ref 0 aborting False >Thread-26668::DEBUG::2012-12-02 14:13:36,082::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26668::DEBUG::2012-12-02 14:13:36,082::task::568::TaskManager.Task::(_updateState) Task=`76c30606-b9d6-4ad1-82c7-390bafde1db5`::moving from state init -> state preparing >Thread-26668::INFO::2012-12-02 14:13:36,082::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26668::INFO::2012-12-02 14:13:36,082::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26668::DEBUG::2012-12-02 14:13:36,083::task::1151::TaskManager.Task::(prepare) Task=`76c30606-b9d6-4ad1-82c7-390bafde1db5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26668::DEBUG::2012-12-02 14:13:36,083::task::568::TaskManager.Task::(_updateState) Task=`76c30606-b9d6-4ad1-82c7-390bafde1db5`::moving from state preparing -> state finished >Thread-26668::DEBUG::2012-12-02 14:13:36,083::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26668::DEBUG::2012-12-02 14:13:36,083::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26668::DEBUG::2012-12-02 14:13:36,083::task::957::TaskManager.Task::(_decref) Task=`76c30606-b9d6-4ad1-82c7-390bafde1db5`::ref 0 aborting False >Thread-26669::DEBUG::2012-12-02 14:13:36,086::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26669::DEBUG::2012-12-02 14:13:36,086::task::568::TaskManager.Task::(_updateState) Task=`b541510a-8e95-433a-b8f7-9bfbb99db5c7`::moving from state init -> state preparing >Thread-26669::INFO::2012-12-02 14:13:36,086::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26669::DEBUG::2012-12-02 14:13:36,086::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a47138a0-7782-4557-b5d7-af3c7956ceea`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26669::DEBUG::2012-12-02 14:13:36,086::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26669::DEBUG::2012-12-02 14:13:36,087::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26669::DEBUG::2012-12-02 14:13:36,087::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a47138a0-7782-4557-b5d7-af3c7956ceea`::Granted request >Thread-26669::DEBUG::2012-12-02 14:13:36,087::task::794::TaskManager.Task::(resourceAcquired) Task=`b541510a-8e95-433a-b8f7-9bfbb99db5c7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26669::DEBUG::2012-12-02 14:13:36,087::task::957::TaskManager.Task::(_decref) Task=`b541510a-8e95-433a-b8f7-9bfbb99db5c7`::ref 1 aborting False >Thread-26669::INFO::2012-12-02 14:13:36,094::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26669::DEBUG::2012-12-02 14:13:36,094::task::1151::TaskManager.Task::(prepare) Task=`b541510a-8e95-433a-b8f7-9bfbb99db5c7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26669::DEBUG::2012-12-02 14:13:36,094::task::568::TaskManager.Task::(_updateState) Task=`b541510a-8e95-433a-b8f7-9bfbb99db5c7`::moving from state preparing -> state finished >Thread-26669::DEBUG::2012-12-02 14:13:36,094::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26669::DEBUG::2012-12-02 14:13:36,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26669::DEBUG::2012-12-02 14:13:36,094::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26669::DEBUG::2012-12-02 14:13:36,095::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26669::DEBUG::2012-12-02 14:13:36,095::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26669::DEBUG::2012-12-02 14:13:36,095::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26669::DEBUG::2012-12-02 14:13:36,095::task::957::TaskManager.Task::(_decref) Task=`b541510a-8e95-433a-b8f7-9bfbb99db5c7`::ref 0 aborting False >Thread-26674::DEBUG::2012-12-02 14:13:44,199::task::568::TaskManager.Task::(_updateState) Task=`6aac3b20-f1b4-4e69-abbb-4cdefd28a79b`::moving from state init -> state preparing >Thread-26674::INFO::2012-12-02 14:13:44,199::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26674::INFO::2012-12-02 14:13:44,200::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00943422317505', 'lastCheck': 1354450415.517323, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0021378993988', 'lastCheck': 1354450420.6431739, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00237011909485', 'lastCheck': 1354450423.5636899, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00859904289246', 'lastCheck': 1354450420.6872699, 'code': 0, 'valid': True}} >Thread-26674::DEBUG::2012-12-02 14:13:44,200::task::1151::TaskManager.Task::(prepare) Task=`6aac3b20-f1b4-4e69-abbb-4cdefd28a79b`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00943422317505', 'lastCheck': 1354450415.517323, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0021378993988', 'lastCheck': 1354450420.6431739, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00237011909485', 'lastCheck': 1354450423.5636899, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00859904289246', 'lastCheck': 1354450420.6872699, 'code': 0, 'valid': True}} >Thread-26674::DEBUG::2012-12-02 14:13:44,200::task::568::TaskManager.Task::(_updateState) Task=`6aac3b20-f1b4-4e69-abbb-4cdefd28a79b`::moving from state preparing -> state finished >Thread-26674::DEBUG::2012-12-02 14:13:44,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26674::DEBUG::2012-12-02 14:13:44,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26674::DEBUG::2012-12-02 14:13:44,200::task::957::TaskManager.Task::(_decref) Task=`6aac3b20-f1b4-4e69-abbb-4cdefd28a79b`::ref 0 aborting False >Thread-26676::DEBUG::2012-12-02 14:13:46,203::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26676::DEBUG::2012-12-02 14:13:46,203::task::568::TaskManager.Task::(_updateState) Task=`cf7bc306-fa57-4129-a51a-44625e823825`::moving from state init -> state preparing >Thread-26676::INFO::2012-12-02 14:13:46,203::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26676::INFO::2012-12-02 14:13:46,204::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26676::DEBUG::2012-12-02 14:13:46,204::task::1151::TaskManager.Task::(prepare) Task=`cf7bc306-fa57-4129-a51a-44625e823825`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26676::DEBUG::2012-12-02 14:13:46,204::task::568::TaskManager.Task::(_updateState) Task=`cf7bc306-fa57-4129-a51a-44625e823825`::moving from state preparing -> state finished >Thread-26676::DEBUG::2012-12-02 14:13:46,204::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26676::DEBUG::2012-12-02 14:13:46,204::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26676::DEBUG::2012-12-02 14:13:46,204::task::957::TaskManager.Task::(_decref) Task=`cf7bc306-fa57-4129-a51a-44625e823825`::ref 0 aborting False >Thread-26677::DEBUG::2012-12-02 14:13:46,207::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26677::DEBUG::2012-12-02 14:13:46,207::task::568::TaskManager.Task::(_updateState) Task=`d106cd54-4e7b-4184-8668-b8767d6b49b8`::moving from state init -> state preparing >Thread-26677::INFO::2012-12-02 14:13:46,207::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26677::DEBUG::2012-12-02 14:13:46,207::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c1bcf91c-1af7-42bd-847f-0ca32068f898`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26677::DEBUG::2012-12-02 14:13:46,208::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26677::DEBUG::2012-12-02 14:13:46,208::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26677::DEBUG::2012-12-02 14:13:46,208::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c1bcf91c-1af7-42bd-847f-0ca32068f898`::Granted request >Thread-26677::DEBUG::2012-12-02 14:13:46,208::task::794::TaskManager.Task::(resourceAcquired) Task=`d106cd54-4e7b-4184-8668-b8767d6b49b8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26677::DEBUG::2012-12-02 14:13:46,208::task::957::TaskManager.Task::(_decref) Task=`d106cd54-4e7b-4184-8668-b8767d6b49b8`::ref 1 aborting False >Thread-26677::INFO::2012-12-02 14:13:46,215::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26677::DEBUG::2012-12-02 14:13:46,215::task::1151::TaskManager.Task::(prepare) Task=`d106cd54-4e7b-4184-8668-b8767d6b49b8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26677::DEBUG::2012-12-02 14:13:46,215::task::568::TaskManager.Task::(_updateState) Task=`d106cd54-4e7b-4184-8668-b8767d6b49b8`::moving from state preparing -> state finished >Thread-26677::DEBUG::2012-12-02 14:13:46,215::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26677::DEBUG::2012-12-02 14:13:46,215::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26677::DEBUG::2012-12-02 14:13:46,215::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26677::DEBUG::2012-12-02 14:13:46,215::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26677::DEBUG::2012-12-02 14:13:46,215::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26677::DEBUG::2012-12-02 14:13:46,216::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26677::DEBUG::2012-12-02 14:13:46,216::task::957::TaskManager.Task::(_decref) Task=`d106cd54-4e7b-4184-8668-b8767d6b49b8`::ref 0 aborting False >Thread-26682::DEBUG::2012-12-02 14:13:54,270::task::568::TaskManager.Task::(_updateState) Task=`438ab054-d706-4f18-ac3b-c490531937be`::moving from state init -> state preparing >Thread-26682::INFO::2012-12-02 14:13:54,270::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26682::INFO::2012-12-02 14:13:54,270::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00825095176697', 'lastCheck': 1354450425.5360241, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00194501876831', 'lastCheck': 1354450430.6538351, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00226712226868', 'lastCheck': 1354450433.5754149, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00632405281067', 'lastCheck': 1354450430.702563, 'code': 0, 'valid': True}} >Thread-26682::DEBUG::2012-12-02 14:13:54,270::task::1151::TaskManager.Task::(prepare) Task=`438ab054-d706-4f18-ac3b-c490531937be`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00825095176697', 'lastCheck': 1354450425.5360241, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00194501876831', 'lastCheck': 1354450430.6538351, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00226712226868', 'lastCheck': 1354450433.5754149, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00632405281067', 'lastCheck': 1354450430.702563, 'code': 0, 'valid': True}} >Thread-26682::DEBUG::2012-12-02 14:13:54,270::task::568::TaskManager.Task::(_updateState) Task=`438ab054-d706-4f18-ac3b-c490531937be`::moving from state preparing -> state finished >Thread-26682::DEBUG::2012-12-02 14:13:54,270::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26682::DEBUG::2012-12-02 14:13:54,271::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26682::DEBUG::2012-12-02 14:13:54,271::task::957::TaskManager.Task::(_decref) Task=`438ab054-d706-4f18-ac3b-c490531937be`::ref 0 aborting False >Thread-26684::DEBUG::2012-12-02 14:13:56,375::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26684::DEBUG::2012-12-02 14:13:56,376::task::568::TaskManager.Task::(_updateState) Task=`189bbd5f-6db3-4e03-b1c5-4c8ec181c2a7`::moving from state init -> state preparing >Thread-26684::INFO::2012-12-02 14:13:56,376::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26684::INFO::2012-12-02 14:13:56,376::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26684::DEBUG::2012-12-02 14:13:56,376::task::1151::TaskManager.Task::(prepare) Task=`189bbd5f-6db3-4e03-b1c5-4c8ec181c2a7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26684::DEBUG::2012-12-02 14:13:56,376::task::568::TaskManager.Task::(_updateState) Task=`189bbd5f-6db3-4e03-b1c5-4c8ec181c2a7`::moving from state preparing -> state finished >Thread-26684::DEBUG::2012-12-02 14:13:56,377::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26684::DEBUG::2012-12-02 14:13:56,377::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26684::DEBUG::2012-12-02 14:13:56,377::task::957::TaskManager.Task::(_decref) Task=`189bbd5f-6db3-4e03-b1c5-4c8ec181c2a7`::ref 0 aborting False >Thread-26685::DEBUG::2012-12-02 14:13:56,389::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26685::DEBUG::2012-12-02 14:13:56,389::task::568::TaskManager.Task::(_updateState) Task=`f4bbd7a4-b770-4cfe-ae82-830a2de15ec1`::moving from state init -> state preparing >Thread-26685::INFO::2012-12-02 14:13:56,390::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26685::DEBUG::2012-12-02 14:13:56,390::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2e6fb3c0-1cab-4c14-90e0-41b8c20e6856`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26685::DEBUG::2012-12-02 14:13:56,390::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26685::DEBUG::2012-12-02 14:13:56,390::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26685::DEBUG::2012-12-02 14:13:56,390::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2e6fb3c0-1cab-4c14-90e0-41b8c20e6856`::Granted request >Thread-26685::DEBUG::2012-12-02 14:13:56,391::task::794::TaskManager.Task::(resourceAcquired) Task=`f4bbd7a4-b770-4cfe-ae82-830a2de15ec1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26685::DEBUG::2012-12-02 14:13:56,391::task::957::TaskManager.Task::(_decref) Task=`f4bbd7a4-b770-4cfe-ae82-830a2de15ec1`::ref 1 aborting False >Thread-26685::INFO::2012-12-02 14:13:56,396::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26685::DEBUG::2012-12-02 14:13:56,397::task::1151::TaskManager.Task::(prepare) Task=`f4bbd7a4-b770-4cfe-ae82-830a2de15ec1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26685::DEBUG::2012-12-02 14:13:56,397::task::568::TaskManager.Task::(_updateState) Task=`f4bbd7a4-b770-4cfe-ae82-830a2de15ec1`::moving from state preparing -> state finished >Thread-26685::DEBUG::2012-12-02 14:13:56,397::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26685::DEBUG::2012-12-02 14:13:56,397::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26685::DEBUG::2012-12-02 14:13:56,397::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26685::DEBUG::2012-12-02 14:13:56,397::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26685::DEBUG::2012-12-02 14:13:56,397::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26685::DEBUG::2012-12-02 14:13:56,397::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26685::DEBUG::2012-12-02 14:13:56,397::task::957::TaskManager.Task::(_decref) Task=`f4bbd7a4-b770-4cfe-ae82-830a2de15ec1`::ref 0 aborting False >Thread-26690::DEBUG::2012-12-02 14:14:04,424::task::568::TaskManager.Task::(_updateState) Task=`4c1ed3d5-e020-42c6-8578-2a8cc5c23029`::moving from state init -> state preparing >Thread-26690::INFO::2012-12-02 14:14:04,424::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26690::INFO::2012-12-02 14:14:04,425::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0106580257416', 'lastCheck': 1354450435.5558741, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00185203552246', 'lastCheck': 1354450440.6655171, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213003158569', 'lastCheck': 1354450443.5869391, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00741910934448', 'lastCheck': 1354450440.7187021, 'code': 0, 'valid': True}} >Thread-26690::DEBUG::2012-12-02 14:14:04,425::task::1151::TaskManager.Task::(prepare) Task=`4c1ed3d5-e020-42c6-8578-2a8cc5c23029`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0106580257416', 'lastCheck': 1354450435.5558741, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00185203552246', 'lastCheck': 1354450440.6655171, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213003158569', 'lastCheck': 1354450443.5869391, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00741910934448', 'lastCheck': 1354450440.7187021, 'code': 0, 'valid': True}} >Thread-26690::DEBUG::2012-12-02 14:14:04,425::task::568::TaskManager.Task::(_updateState) Task=`4c1ed3d5-e020-42c6-8578-2a8cc5c23029`::moving from state preparing -> state finished >Thread-26690::DEBUG::2012-12-02 14:14:04,425::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26690::DEBUG::2012-12-02 14:14:04,425::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26690::DEBUG::2012-12-02 14:14:04,425::task::957::TaskManager.Task::(_decref) Task=`4c1ed3d5-e020-42c6-8578-2a8cc5c23029`::ref 0 aborting False >Thread-26693::DEBUG::2012-12-02 14:14:06,569::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26693::DEBUG::2012-12-02 14:14:06,569::task::568::TaskManager.Task::(_updateState) Task=`e99852cc-954f-4007-8a67-e14d49862eb2`::moving from state init -> state preparing >Thread-26693::INFO::2012-12-02 14:14:06,569::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26693::INFO::2012-12-02 14:14:06,570::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26693::DEBUG::2012-12-02 14:14:06,570::task::1151::TaskManager.Task::(prepare) Task=`e99852cc-954f-4007-8a67-e14d49862eb2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26693::DEBUG::2012-12-02 14:14:06,570::task::568::TaskManager.Task::(_updateState) Task=`e99852cc-954f-4007-8a67-e14d49862eb2`::moving from state preparing -> state finished >Thread-26693::DEBUG::2012-12-02 14:14:06,570::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26693::DEBUG::2012-12-02 14:14:06,570::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26693::DEBUG::2012-12-02 14:14:06,570::task::957::TaskManager.Task::(_decref) Task=`e99852cc-954f-4007-8a67-e14d49862eb2`::ref 0 aborting False >Thread-26694::DEBUG::2012-12-02 14:14:06,573::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26694::DEBUG::2012-12-02 14:14:06,573::task::568::TaskManager.Task::(_updateState) Task=`b02e96de-9603-436a-aa63-1c3f353c5a35`::moving from state init -> state preparing >Thread-26694::INFO::2012-12-02 14:14:06,573::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26694::DEBUG::2012-12-02 14:14:06,573::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5c64f360-6fd3-4147-8c71-b39a73b00a9c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26694::DEBUG::2012-12-02 14:14:06,573::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26694::DEBUG::2012-12-02 14:14:06,574::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26694::DEBUG::2012-12-02 14:14:06,574::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5c64f360-6fd3-4147-8c71-b39a73b00a9c`::Granted request >Thread-26694::DEBUG::2012-12-02 14:14:06,574::task::794::TaskManager.Task::(resourceAcquired) Task=`b02e96de-9603-436a-aa63-1c3f353c5a35`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26694::DEBUG::2012-12-02 14:14:06,574::task::957::TaskManager.Task::(_decref) Task=`b02e96de-9603-436a-aa63-1c3f353c5a35`::ref 1 aborting False >Thread-26694::INFO::2012-12-02 14:14:06,579::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26694::DEBUG::2012-12-02 14:14:06,579::task::1151::TaskManager.Task::(prepare) Task=`b02e96de-9603-436a-aa63-1c3f353c5a35`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26694::DEBUG::2012-12-02 14:14:06,579::task::568::TaskManager.Task::(_updateState) Task=`b02e96de-9603-436a-aa63-1c3f353c5a35`::moving from state preparing -> state finished >Thread-26694::DEBUG::2012-12-02 14:14:06,579::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26694::DEBUG::2012-12-02 14:14:06,580::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26694::DEBUG::2012-12-02 14:14:06,580::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26694::DEBUG::2012-12-02 14:14:06,580::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26694::DEBUG::2012-12-02 14:14:06,580::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26694::DEBUG::2012-12-02 14:14:06,580::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26694::DEBUG::2012-12-02 14:14:06,580::task::957::TaskManager.Task::(_decref) Task=`b02e96de-9603-436a-aa63-1c3f353c5a35`::ref 0 aborting False >Thread-26698::DEBUG::2012-12-02 14:14:14,487::task::568::TaskManager.Task::(_updateState) Task=`fccdfabf-3e30-4df6-b464-2c28201b2fee`::moving from state init -> state preparing >Thread-26698::INFO::2012-12-02 14:14:14,487::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26698::INFO::2012-12-02 14:14:14,487::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00784802436829', 'lastCheck': 1354450445.5730929, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00323104858398', 'lastCheck': 1354450450.6795039, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00294399261475', 'lastCheck': 1354450453.6000021, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00702810287476', 'lastCheck': 1354450450.7347021, 'code': 0, 'valid': True}} >Thread-26698::DEBUG::2012-12-02 14:14:14,487::task::1151::TaskManager.Task::(prepare) Task=`fccdfabf-3e30-4df6-b464-2c28201b2fee`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00784802436829', 'lastCheck': 1354450445.5730929, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00323104858398', 'lastCheck': 1354450450.6795039, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00294399261475', 'lastCheck': 1354450453.6000021, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00702810287476', 'lastCheck': 1354450450.7347021, 'code': 0, 'valid': True}} >Thread-26698::DEBUG::2012-12-02 14:14:14,488::task::568::TaskManager.Task::(_updateState) Task=`fccdfabf-3e30-4df6-b464-2c28201b2fee`::moving from state preparing -> state finished >Thread-26698::DEBUG::2012-12-02 14:14:14,488::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26698::DEBUG::2012-12-02 14:14:14,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26698::DEBUG::2012-12-02 14:14:14,488::task::957::TaskManager.Task::(_decref) Task=`fccdfabf-3e30-4df6-b464-2c28201b2fee`::ref 0 aborting False >Thread-26701::DEBUG::2012-12-02 14:14:16,673::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26701::DEBUG::2012-12-02 14:14:16,673::task::568::TaskManager.Task::(_updateState) Task=`e7935184-2ee4-4f6e-9965-b8ac299725d0`::moving from state init -> state preparing >Thread-26701::INFO::2012-12-02 14:14:16,674::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26701::INFO::2012-12-02 14:14:16,674::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26701::DEBUG::2012-12-02 14:14:16,674::task::1151::TaskManager.Task::(prepare) Task=`e7935184-2ee4-4f6e-9965-b8ac299725d0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26701::DEBUG::2012-12-02 14:14:16,674::task::568::TaskManager.Task::(_updateState) Task=`e7935184-2ee4-4f6e-9965-b8ac299725d0`::moving from state preparing -> state finished >Thread-26701::DEBUG::2012-12-02 14:14:16,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26701::DEBUG::2012-12-02 14:14:16,674::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26701::DEBUG::2012-12-02 14:14:16,674::task::957::TaskManager.Task::(_decref) Task=`e7935184-2ee4-4f6e-9965-b8ac299725d0`::ref 0 aborting False >Thread-26702::DEBUG::2012-12-02 14:14:16,677::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26702::DEBUG::2012-12-02 14:14:16,677::task::568::TaskManager.Task::(_updateState) Task=`4b7eb40e-03f0-485f-89f4-0080e7d28207`::moving from state init -> state preparing >Thread-26702::INFO::2012-12-02 14:14:16,677::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26702::DEBUG::2012-12-02 14:14:16,677::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2812983a-c266-475d-a203-4c75ff01475e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26702::DEBUG::2012-12-02 14:14:16,678::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26702::DEBUG::2012-12-02 14:14:16,678::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26702::DEBUG::2012-12-02 14:14:16,678::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2812983a-c266-475d-a203-4c75ff01475e`::Granted request >Thread-26702::DEBUG::2012-12-02 14:14:16,678::task::794::TaskManager.Task::(resourceAcquired) Task=`4b7eb40e-03f0-485f-89f4-0080e7d28207`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26702::DEBUG::2012-12-02 14:14:16,678::task::957::TaskManager.Task::(_decref) Task=`4b7eb40e-03f0-485f-89f4-0080e7d28207`::ref 1 aborting False >Thread-26702::INFO::2012-12-02 14:14:16,682::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26702::DEBUG::2012-12-02 14:14:16,683::task::1151::TaskManager.Task::(prepare) Task=`4b7eb40e-03f0-485f-89f4-0080e7d28207`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26702::DEBUG::2012-12-02 14:14:16,683::task::568::TaskManager.Task::(_updateState) Task=`4b7eb40e-03f0-485f-89f4-0080e7d28207`::moving from state preparing -> state finished >Thread-26702::DEBUG::2012-12-02 14:14:16,683::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26702::DEBUG::2012-12-02 14:14:16,683::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26702::DEBUG::2012-12-02 14:14:16,683::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26702::DEBUG::2012-12-02 14:14:16,683::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26702::DEBUG::2012-12-02 14:14:16,683::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26702::DEBUG::2012-12-02 14:14:16,683::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26702::DEBUG::2012-12-02 14:14:16,683::task::957::TaskManager.Task::(_decref) Task=`4b7eb40e-03f0-485f-89f4-0080e7d28207`::ref 0 aborting False >Thread-26706::DEBUG::2012-12-02 14:14:24,690::task::568::TaskManager.Task::(_updateState) Task=`24362ef5-0c1b-419d-8395-469dc9735138`::moving from state init -> state preparing >Thread-26706::INFO::2012-12-02 14:14:24,690::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26706::INFO::2012-12-02 14:14:24,690::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00887703895569', 'lastCheck': 1354450455.58953, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00293707847595', 'lastCheck': 1354450460.697058, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0026159286499', 'lastCheck': 1354450463.6122129, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00628709793091', 'lastCheck': 1354450460.750458, 'code': 0, 'valid': True}} >Thread-26706::DEBUG::2012-12-02 14:14:24,691::task::1151::TaskManager.Task::(prepare) Task=`24362ef5-0c1b-419d-8395-469dc9735138`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00887703895569', 'lastCheck': 1354450455.58953, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00293707847595', 'lastCheck': 1354450460.697058, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0026159286499', 'lastCheck': 1354450463.6122129, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00628709793091', 'lastCheck': 1354450460.750458, 'code': 0, 'valid': True}} >Thread-26706::DEBUG::2012-12-02 14:14:24,691::task::568::TaskManager.Task::(_updateState) Task=`24362ef5-0c1b-419d-8395-469dc9735138`::moving from state preparing -> state finished >Thread-26706::DEBUG::2012-12-02 14:14:24,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26706::DEBUG::2012-12-02 14:14:24,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26706::DEBUG::2012-12-02 14:14:24,691::task::957::TaskManager.Task::(_decref) Task=`24362ef5-0c1b-419d-8395-469dc9735138`::ref 0 aborting False >Thread-26708::DEBUG::2012-12-02 14:14:26,778::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26708::DEBUG::2012-12-02 14:14:26,778::task::568::TaskManager.Task::(_updateState) Task=`62d9f062-0b6f-40b6-a524-44a045874024`::moving from state init -> state preparing >Thread-26708::INFO::2012-12-02 14:14:26,778::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26708::INFO::2012-12-02 14:14:26,779::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26708::DEBUG::2012-12-02 14:14:26,779::task::1151::TaskManager.Task::(prepare) Task=`62d9f062-0b6f-40b6-a524-44a045874024`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26708::DEBUG::2012-12-02 14:14:26,779::task::568::TaskManager.Task::(_updateState) Task=`62d9f062-0b6f-40b6-a524-44a045874024`::moving from state preparing -> state finished >Thread-26708::DEBUG::2012-12-02 14:14:26,779::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26708::DEBUG::2012-12-02 14:14:26,779::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26708::DEBUG::2012-12-02 14:14:26,779::task::957::TaskManager.Task::(_decref) Task=`62d9f062-0b6f-40b6-a524-44a045874024`::ref 0 aborting False >Thread-26709::DEBUG::2012-12-02 14:14:26,782::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26709::DEBUG::2012-12-02 14:14:26,782::task::568::TaskManager.Task::(_updateState) Task=`8f93c117-5611-461d-8bd3-e648f7fe5315`::moving from state init -> state preparing >Thread-26709::INFO::2012-12-02 14:14:26,782::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26709::DEBUG::2012-12-02 14:14:26,782::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ccf0cadd-6d53-4a32-8cb7-bdc7f63cfef1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26709::DEBUG::2012-12-02 14:14:26,782::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26709::DEBUG::2012-12-02 14:14:26,783::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26709::DEBUG::2012-12-02 14:14:26,783::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ccf0cadd-6d53-4a32-8cb7-bdc7f63cfef1`::Granted request >Thread-26709::DEBUG::2012-12-02 14:14:26,783::task::794::TaskManager.Task::(resourceAcquired) Task=`8f93c117-5611-461d-8bd3-e648f7fe5315`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26709::DEBUG::2012-12-02 14:14:26,783::task::957::TaskManager.Task::(_decref) Task=`8f93c117-5611-461d-8bd3-e648f7fe5315`::ref 1 aborting False >Thread-26709::INFO::2012-12-02 14:14:26,788::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26709::DEBUG::2012-12-02 14:14:26,789::task::1151::TaskManager.Task::(prepare) Task=`8f93c117-5611-461d-8bd3-e648f7fe5315`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26709::DEBUG::2012-12-02 14:14:26,789::task::568::TaskManager.Task::(_updateState) Task=`8f93c117-5611-461d-8bd3-e648f7fe5315`::moving from state preparing -> state finished >Thread-26709::DEBUG::2012-12-02 14:14:26,789::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26709::DEBUG::2012-12-02 14:14:26,789::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26709::DEBUG::2012-12-02 14:14:26,789::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26709::DEBUG::2012-12-02 14:14:26,789::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26709::DEBUG::2012-12-02 14:14:26,789::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26709::DEBUG::2012-12-02 14:14:26,789::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26709::DEBUG::2012-12-02 14:14:26,789::task::957::TaskManager.Task::(_decref) Task=`8f93c117-5611-461d-8bd3-e648f7fe5315`::ref 0 aborting False >Thread-26714::DEBUG::2012-12-02 14:14:34,812::task::568::TaskManager.Task::(_updateState) Task=`619f8609-b705-440e-9002-a4a9e565c2c7`::moving from state init -> state preparing >Thread-26714::INFO::2012-12-02 14:14:34,812::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26714::INFO::2012-12-02 14:14:34,812::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0076220035553', 'lastCheck': 1354450465.606472, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00477814674377', 'lastCheck': 1354450470.7118199, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00343918800354', 'lastCheck': 1354450473.626637, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0070059299469', 'lastCheck': 1354450470.766247, 'code': 0, 'valid': True}} >Thread-26714::DEBUG::2012-12-02 14:14:34,812::task::1151::TaskManager.Task::(prepare) Task=`619f8609-b705-440e-9002-a4a9e565c2c7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0076220035553', 'lastCheck': 1354450465.606472, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00477814674377', 'lastCheck': 1354450470.7118199, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00343918800354', 'lastCheck': 1354450473.626637, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0070059299469', 'lastCheck': 1354450470.766247, 'code': 0, 'valid': True}} >Thread-26714::DEBUG::2012-12-02 14:14:34,812::task::568::TaskManager.Task::(_updateState) Task=`619f8609-b705-440e-9002-a4a9e565c2c7`::moving from state preparing -> state finished >Thread-26714::DEBUG::2012-12-02 14:14:34,813::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26714::DEBUG::2012-12-02 14:14:34,813::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26714::DEBUG::2012-12-02 14:14:34,813::task::957::TaskManager.Task::(_decref) Task=`619f8609-b705-440e-9002-a4a9e565c2c7`::ref 0 aborting False >Thread-26717::DEBUG::2012-12-02 14:14:36,907::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26717::DEBUG::2012-12-02 14:14:36,908::task::568::TaskManager.Task::(_updateState) Task=`6604a4c6-b31c-472b-a59a-d8c210091177`::moving from state init -> state preparing >Thread-26717::INFO::2012-12-02 14:14:36,908::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26717::INFO::2012-12-02 14:14:36,908::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26717::DEBUG::2012-12-02 14:14:36,908::task::1151::TaskManager.Task::(prepare) Task=`6604a4c6-b31c-472b-a59a-d8c210091177`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26717::DEBUG::2012-12-02 14:14:36,908::task::568::TaskManager.Task::(_updateState) Task=`6604a4c6-b31c-472b-a59a-d8c210091177`::moving from state preparing -> state finished >Thread-26717::DEBUG::2012-12-02 14:14:36,909::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26717::DEBUG::2012-12-02 14:14:36,909::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26717::DEBUG::2012-12-02 14:14:36,909::task::957::TaskManager.Task::(_decref) Task=`6604a4c6-b31c-472b-a59a-d8c210091177`::ref 0 aborting False >Thread-26718::DEBUG::2012-12-02 14:14:36,911::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26718::DEBUG::2012-12-02 14:14:36,912::task::568::TaskManager.Task::(_updateState) Task=`844384f0-058c-4bb7-b015-4234e081a2f4`::moving from state init -> state preparing >Thread-26718::INFO::2012-12-02 14:14:36,912::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26718::DEBUG::2012-12-02 14:14:36,912::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`18c624ab-ba5a-41dc-b9b8-b79018121a94`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26718::DEBUG::2012-12-02 14:14:36,912::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26718::DEBUG::2012-12-02 14:14:36,912::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26718::DEBUG::2012-12-02 14:14:36,912::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`18c624ab-ba5a-41dc-b9b8-b79018121a94`::Granted request >Thread-26718::DEBUG::2012-12-02 14:14:36,913::task::794::TaskManager.Task::(resourceAcquired) Task=`844384f0-058c-4bb7-b015-4234e081a2f4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26718::DEBUG::2012-12-02 14:14:36,913::task::957::TaskManager.Task::(_decref) Task=`844384f0-058c-4bb7-b015-4234e081a2f4`::ref 1 aborting False >Thread-26718::INFO::2012-12-02 14:14:36,918::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26718::DEBUG::2012-12-02 14:14:36,918::task::1151::TaskManager.Task::(prepare) Task=`844384f0-058c-4bb7-b015-4234e081a2f4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26718::DEBUG::2012-12-02 14:14:36,918::task::568::TaskManager.Task::(_updateState) Task=`844384f0-058c-4bb7-b015-4234e081a2f4`::moving from state preparing -> state finished >Thread-26718::DEBUG::2012-12-02 14:14:36,918::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26718::DEBUG::2012-12-02 14:14:36,919::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26718::DEBUG::2012-12-02 14:14:36,919::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26718::DEBUG::2012-12-02 14:14:36,919::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26718::DEBUG::2012-12-02 14:14:36,919::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26718::DEBUG::2012-12-02 14:14:36,919::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26718::DEBUG::2012-12-02 14:14:36,919::task::957::TaskManager.Task::(_decref) Task=`844384f0-058c-4bb7-b015-4234e081a2f4`::ref 0 aborting False >Thread-26722::DEBUG::2012-12-02 14:14:44,906::task::568::TaskManager.Task::(_updateState) Task=`f3c01ec8-3936-4986-bb00-2806e9e8a73d`::moving from state init -> state preparing >Thread-26722::INFO::2012-12-02 14:14:44,907::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26722::INFO::2012-12-02 14:14:44,907::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0080988407135', 'lastCheck': 1354450475.6263549, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.002366065979', 'lastCheck': 1354450480.7290881, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00215411186218', 'lastCheck': 1354450483.6383779, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00576901435852', 'lastCheck': 1354450480.7827959, 'code': 0, 'valid': True}} >Thread-26722::DEBUG::2012-12-02 14:14:44,907::task::1151::TaskManager.Task::(prepare) Task=`f3c01ec8-3936-4986-bb00-2806e9e8a73d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0080988407135', 'lastCheck': 1354450475.6263549, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.002366065979', 'lastCheck': 1354450480.7290881, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00215411186218', 'lastCheck': 1354450483.6383779, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00576901435852', 'lastCheck': 1354450480.7827959, 'code': 0, 'valid': True}} >Thread-26722::DEBUG::2012-12-02 14:14:44,907::task::568::TaskManager.Task::(_updateState) Task=`f3c01ec8-3936-4986-bb00-2806e9e8a73d`::moving from state preparing -> state finished >Thread-26722::DEBUG::2012-12-02 14:14:44,907::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26722::DEBUG::2012-12-02 14:14:44,907::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26722::DEBUG::2012-12-02 14:14:44,908::task::957::TaskManager.Task::(_decref) Task=`f3c01ec8-3936-4986-bb00-2806e9e8a73d`::ref 0 aborting False >Thread-26725::DEBUG::2012-12-02 14:14:47,019::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26725::DEBUG::2012-12-02 14:14:47,019::task::568::TaskManager.Task::(_updateState) Task=`c76857df-a175-4f0a-a6ff-43de03921040`::moving from state init -> state preparing >Thread-26725::INFO::2012-12-02 14:14:47,019::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26725::INFO::2012-12-02 14:14:47,020::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26725::DEBUG::2012-12-02 14:14:47,020::task::1151::TaskManager.Task::(prepare) Task=`c76857df-a175-4f0a-a6ff-43de03921040`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26725::DEBUG::2012-12-02 14:14:47,020::task::568::TaskManager.Task::(_updateState) Task=`c76857df-a175-4f0a-a6ff-43de03921040`::moving from state preparing -> state finished >Thread-26725::DEBUG::2012-12-02 14:14:47,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26725::DEBUG::2012-12-02 14:14:47,020::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26725::DEBUG::2012-12-02 14:14:47,020::task::957::TaskManager.Task::(_decref) Task=`c76857df-a175-4f0a-a6ff-43de03921040`::ref 0 aborting False >Thread-26726::DEBUG::2012-12-02 14:14:47,023::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26726::DEBUG::2012-12-02 14:14:47,023::task::568::TaskManager.Task::(_updateState) Task=`e53b71fe-d37a-4c0e-85a0-40ec95a8eaeb`::moving from state init -> state preparing >Thread-26726::INFO::2012-12-02 14:14:47,023::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26726::DEBUG::2012-12-02 14:14:47,023::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d68c4e8b-17e8-4db6-a89e-17bc3978a31b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26726::DEBUG::2012-12-02 14:14:47,023::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26726::DEBUG::2012-12-02 14:14:47,024::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26726::DEBUG::2012-12-02 14:14:47,024::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d68c4e8b-17e8-4db6-a89e-17bc3978a31b`::Granted request >Thread-26726::DEBUG::2012-12-02 14:14:47,024::task::794::TaskManager.Task::(resourceAcquired) Task=`e53b71fe-d37a-4c0e-85a0-40ec95a8eaeb`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26726::DEBUG::2012-12-02 14:14:47,024::task::957::TaskManager.Task::(_decref) Task=`e53b71fe-d37a-4c0e-85a0-40ec95a8eaeb`::ref 1 aborting False >Thread-26726::INFO::2012-12-02 14:14:47,029::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26726::DEBUG::2012-12-02 14:14:47,029::task::1151::TaskManager.Task::(prepare) Task=`e53b71fe-d37a-4c0e-85a0-40ec95a8eaeb`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26726::DEBUG::2012-12-02 14:14:47,029::task::568::TaskManager.Task::(_updateState) Task=`e53b71fe-d37a-4c0e-85a0-40ec95a8eaeb`::moving from state preparing -> state finished >Thread-26726::DEBUG::2012-12-02 14:14:47,029::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26726::DEBUG::2012-12-02 14:14:47,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26726::DEBUG::2012-12-02 14:14:47,030::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26726::DEBUG::2012-12-02 14:14:47,030::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26726::DEBUG::2012-12-02 14:14:47,030::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26726::DEBUG::2012-12-02 14:14:47,030::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26726::DEBUG::2012-12-02 14:14:47,030::task::957::TaskManager.Task::(_decref) Task=`e53b71fe-d37a-4c0e-85a0-40ec95a8eaeb`::ref 0 aborting False >Thread-26730::DEBUG::2012-12-02 14:14:55,028::task::568::TaskManager.Task::(_updateState) Task=`abdb90ef-cf2f-45a5-acb2-d680e453bfc9`::moving from state init -> state preparing >Thread-26730::INFO::2012-12-02 14:14:55,028::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26730::INFO::2012-12-02 14:14:55,028::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0072979927063', 'lastCheck': 1354450485.6436551, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00200986862183', 'lastCheck': 1354450490.7407019, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00271105766296', 'lastCheck': 1354450493.6510279, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00617694854736', 'lastCheck': 1354450490.797478, 'code': 0, 'valid': True}} >Thread-26730::DEBUG::2012-12-02 14:14:55,028::task::1151::TaskManager.Task::(prepare) Task=`abdb90ef-cf2f-45a5-acb2-d680e453bfc9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0072979927063', 'lastCheck': 1354450485.6436551, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00200986862183', 'lastCheck': 1354450490.7407019, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00271105766296', 'lastCheck': 1354450493.6510279, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00617694854736', 'lastCheck': 1354450490.797478, 'code': 0, 'valid': True}} >Thread-26730::DEBUG::2012-12-02 14:14:55,029::task::568::TaskManager.Task::(_updateState) Task=`abdb90ef-cf2f-45a5-acb2-d680e453bfc9`::moving from state preparing -> state finished >Thread-26730::DEBUG::2012-12-02 14:14:55,029::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26730::DEBUG::2012-12-02 14:14:55,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26730::DEBUG::2012-12-02 14:14:55,029::task::957::TaskManager.Task::(_decref) Task=`abdb90ef-cf2f-45a5-acb2-d680e453bfc9`::ref 0 aborting False >Thread-26733::DEBUG::2012-12-02 14:14:57,133::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26733::DEBUG::2012-12-02 14:14:57,133::task::568::TaskManager.Task::(_updateState) Task=`7eb0e7d6-348a-4f69-ba8a-1fd56c29fa71`::moving from state init -> state preparing >Thread-26733::INFO::2012-12-02 14:14:57,133::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26733::INFO::2012-12-02 14:14:57,134::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26733::DEBUG::2012-12-02 14:14:57,134::task::1151::TaskManager.Task::(prepare) Task=`7eb0e7d6-348a-4f69-ba8a-1fd56c29fa71`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26733::DEBUG::2012-12-02 14:14:57,134::task::568::TaskManager.Task::(_updateState) Task=`7eb0e7d6-348a-4f69-ba8a-1fd56c29fa71`::moving from state preparing -> state finished >Thread-26733::DEBUG::2012-12-02 14:14:57,134::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26733::DEBUG::2012-12-02 14:14:57,134::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26733::DEBUG::2012-12-02 14:14:57,134::task::957::TaskManager.Task::(_decref) Task=`7eb0e7d6-348a-4f69-ba8a-1fd56c29fa71`::ref 0 aborting False >Thread-26734::DEBUG::2012-12-02 14:14:57,137::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26734::DEBUG::2012-12-02 14:14:57,137::task::568::TaskManager.Task::(_updateState) Task=`3f4bb8f2-573c-49eb-a14c-ae2f109db210`::moving from state init -> state preparing >Thread-26734::INFO::2012-12-02 14:14:57,137::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26734::DEBUG::2012-12-02 14:14:57,137::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c568ed68-40c4-4dc7-8589-e5cadb56f354`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26734::DEBUG::2012-12-02 14:14:57,137::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26734::DEBUG::2012-12-02 14:14:57,138::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26734::DEBUG::2012-12-02 14:14:57,138::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c568ed68-40c4-4dc7-8589-e5cadb56f354`::Granted request >Thread-26734::DEBUG::2012-12-02 14:14:57,138::task::794::TaskManager.Task::(resourceAcquired) Task=`3f4bb8f2-573c-49eb-a14c-ae2f109db210`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26734::DEBUG::2012-12-02 14:14:57,138::task::957::TaskManager.Task::(_decref) Task=`3f4bb8f2-573c-49eb-a14c-ae2f109db210`::ref 1 aborting False >Thread-26734::INFO::2012-12-02 14:14:57,145::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26734::DEBUG::2012-12-02 14:14:57,145::task::1151::TaskManager.Task::(prepare) Task=`3f4bb8f2-573c-49eb-a14c-ae2f109db210`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26734::DEBUG::2012-12-02 14:14:57,145::task::568::TaskManager.Task::(_updateState) Task=`3f4bb8f2-573c-49eb-a14c-ae2f109db210`::moving from state preparing -> state finished >Thread-26734::DEBUG::2012-12-02 14:14:57,145::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26734::DEBUG::2012-12-02 14:14:57,145::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26734::DEBUG::2012-12-02 14:14:57,145::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26734::DEBUG::2012-12-02 14:14:57,145::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26734::DEBUG::2012-12-02 14:14:57,145::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26734::DEBUG::2012-12-02 14:14:57,146::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26734::DEBUG::2012-12-02 14:14:57,146::task::957::TaskManager.Task::(_decref) Task=`3f4bb8f2-573c-49eb-a14c-ae2f109db210`::ref 0 aborting False >Thread-26738::DEBUG::2012-12-02 14:15:05,116::task::568::TaskManager.Task::(_updateState) Task=`f67a43c0-80fb-46f3-bfa5-11910ef7ee43`::moving from state init -> state preparing >Thread-26738::INFO::2012-12-02 14:15:05,117::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26738::INFO::2012-12-02 14:15:05,117::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104820728302', 'lastCheck': 1354450495.662358, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00210404396057', 'lastCheck': 1354450500.753355, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00353598594666', 'lastCheck': 1354450503.6637919, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0079391002655', 'lastCheck': 1354450500.814019, 'code': 0, 'valid': True}} >Thread-26738::DEBUG::2012-12-02 14:15:05,117::task::1151::TaskManager.Task::(prepare) Task=`f67a43c0-80fb-46f3-bfa5-11910ef7ee43`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104820728302', 'lastCheck': 1354450495.662358, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00210404396057', 'lastCheck': 1354450500.753355, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00353598594666', 'lastCheck': 1354450503.6637919, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0079391002655', 'lastCheck': 1354450500.814019, 'code': 0, 'valid': True}} >Thread-26738::DEBUG::2012-12-02 14:15:05,117::task::568::TaskManager.Task::(_updateState) Task=`f67a43c0-80fb-46f3-bfa5-11910ef7ee43`::moving from state preparing -> state finished >Thread-26738::DEBUG::2012-12-02 14:15:05,117::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26738::DEBUG::2012-12-02 14:15:05,117::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26738::DEBUG::2012-12-02 14:15:05,117::task::957::TaskManager.Task::(_decref) Task=`f67a43c0-80fb-46f3-bfa5-11910ef7ee43`::ref 0 aborting False >Thread-26741::DEBUG::2012-12-02 14:15:07,245::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26741::DEBUG::2012-12-02 14:15:07,245::task::568::TaskManager.Task::(_updateState) Task=`8e84dfd8-2a30-4997-94dd-4f8bcda811d0`::moving from state init -> state preparing >Thread-26741::INFO::2012-12-02 14:15:07,245::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26741::INFO::2012-12-02 14:15:07,246::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26741::DEBUG::2012-12-02 14:15:07,246::task::1151::TaskManager.Task::(prepare) Task=`8e84dfd8-2a30-4997-94dd-4f8bcda811d0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26741::DEBUG::2012-12-02 14:15:07,246::task::568::TaskManager.Task::(_updateState) Task=`8e84dfd8-2a30-4997-94dd-4f8bcda811d0`::moving from state preparing -> state finished >Thread-26741::DEBUG::2012-12-02 14:15:07,246::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26741::DEBUG::2012-12-02 14:15:07,246::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26741::DEBUG::2012-12-02 14:15:07,246::task::957::TaskManager.Task::(_decref) Task=`8e84dfd8-2a30-4997-94dd-4f8bcda811d0`::ref 0 aborting False >Thread-26742::DEBUG::2012-12-02 14:15:07,249::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26742::DEBUG::2012-12-02 14:15:07,249::task::568::TaskManager.Task::(_updateState) Task=`459b2deb-2ad9-4e4b-85ca-1552ab66ef1d`::moving from state init -> state preparing >Thread-26742::INFO::2012-12-02 14:15:07,249::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26742::DEBUG::2012-12-02 14:15:07,249::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ccaa5f55-7d62-4291-8641-a86f4940428e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26742::DEBUG::2012-12-02 14:15:07,250::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26742::DEBUG::2012-12-02 14:15:07,250::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26742::DEBUG::2012-12-02 14:15:07,250::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ccaa5f55-7d62-4291-8641-a86f4940428e`::Granted request >Thread-26742::DEBUG::2012-12-02 14:15:07,250::task::794::TaskManager.Task::(resourceAcquired) Task=`459b2deb-2ad9-4e4b-85ca-1552ab66ef1d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26742::DEBUG::2012-12-02 14:15:07,250::task::957::TaskManager.Task::(_decref) Task=`459b2deb-2ad9-4e4b-85ca-1552ab66ef1d`::ref 1 aborting False >Thread-26742::INFO::2012-12-02 14:15:07,256::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26742::DEBUG::2012-12-02 14:15:07,256::task::1151::TaskManager.Task::(prepare) Task=`459b2deb-2ad9-4e4b-85ca-1552ab66ef1d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26742::DEBUG::2012-12-02 14:15:07,256::task::568::TaskManager.Task::(_updateState) Task=`459b2deb-2ad9-4e4b-85ca-1552ab66ef1d`::moving from state preparing -> state finished >Thread-26742::DEBUG::2012-12-02 14:15:07,256::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26742::DEBUG::2012-12-02 14:15:07,257::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26742::DEBUG::2012-12-02 14:15:07,257::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26742::DEBUG::2012-12-02 14:15:07,257::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26742::DEBUG::2012-12-02 14:15:07,257::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26742::DEBUG::2012-12-02 14:15:07,257::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26742::DEBUG::2012-12-02 14:15:07,257::task::957::TaskManager.Task::(_decref) Task=`459b2deb-2ad9-4e4b-85ca-1552ab66ef1d`::ref 0 aborting False >Thread-26746::DEBUG::2012-12-02 14:15:15,176::task::568::TaskManager.Task::(_updateState) Task=`e9d0c220-9d2a-4beb-af57-ebdc19510049`::moving from state init -> state preparing >Thread-26746::INFO::2012-12-02 14:15:15,176::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26746::INFO::2012-12-02 14:15:15,176::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00756907463074', 'lastCheck': 1354450505.6785991, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00365209579468', 'lastCheck': 1354450510.772059, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00336909294128', 'lastCheck': 1354450513.676487, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0081479549408', 'lastCheck': 1354450510.831192, 'code': 0, 'valid': True}} >Thread-26746::DEBUG::2012-12-02 14:15:15,176::task::1151::TaskManager.Task::(prepare) Task=`e9d0c220-9d2a-4beb-af57-ebdc19510049`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00756907463074', 'lastCheck': 1354450505.6785991, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00365209579468', 'lastCheck': 1354450510.772059, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00336909294128', 'lastCheck': 1354450513.676487, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0081479549408', 'lastCheck': 1354450510.831192, 'code': 0, 'valid': True}} >Thread-26746::DEBUG::2012-12-02 14:15:15,177::task::568::TaskManager.Task::(_updateState) Task=`e9d0c220-9d2a-4beb-af57-ebdc19510049`::moving from state preparing -> state finished >Thread-26746::DEBUG::2012-12-02 14:15:15,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26746::DEBUG::2012-12-02 14:15:15,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26746::DEBUG::2012-12-02 14:15:15,177::task::957::TaskManager.Task::(_decref) Task=`e9d0c220-9d2a-4beb-af57-ebdc19510049`::ref 0 aborting False >Thread-26749::DEBUG::2012-12-02 14:15:17,356::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26749::DEBUG::2012-12-02 14:15:17,357::task::568::TaskManager.Task::(_updateState) Task=`233cfe1d-1f27-4838-84f8-a2c8cb4cce71`::moving from state init -> state preparing >Thread-26749::INFO::2012-12-02 14:15:17,357::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26749::INFO::2012-12-02 14:15:17,357::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26749::DEBUG::2012-12-02 14:15:17,357::task::1151::TaskManager.Task::(prepare) Task=`233cfe1d-1f27-4838-84f8-a2c8cb4cce71`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26749::DEBUG::2012-12-02 14:15:17,357::task::568::TaskManager.Task::(_updateState) Task=`233cfe1d-1f27-4838-84f8-a2c8cb4cce71`::moving from state preparing -> state finished >Thread-26749::DEBUG::2012-12-02 14:15:17,357::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26749::DEBUG::2012-12-02 14:15:17,357::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26749::DEBUG::2012-12-02 14:15:17,358::task::957::TaskManager.Task::(_decref) Task=`233cfe1d-1f27-4838-84f8-a2c8cb4cce71`::ref 0 aborting False >Thread-26750::DEBUG::2012-12-02 14:15:17,360::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26750::DEBUG::2012-12-02 14:15:17,361::task::568::TaskManager.Task::(_updateState) Task=`d5c7242a-ca50-4ad1-a81f-89a7c21edc7d`::moving from state init -> state preparing >Thread-26750::INFO::2012-12-02 14:15:17,361::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26750::DEBUG::2012-12-02 14:15:17,361::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bba42f82-0430-4ddf-9b5e-5d4a666cf824`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26750::DEBUG::2012-12-02 14:15:17,361::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26750::DEBUG::2012-12-02 14:15:17,361::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26750::DEBUG::2012-12-02 14:15:17,361::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bba42f82-0430-4ddf-9b5e-5d4a666cf824`::Granted request >Thread-26750::DEBUG::2012-12-02 14:15:17,362::task::794::TaskManager.Task::(resourceAcquired) Task=`d5c7242a-ca50-4ad1-a81f-89a7c21edc7d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26750::DEBUG::2012-12-02 14:15:17,362::task::957::TaskManager.Task::(_decref) Task=`d5c7242a-ca50-4ad1-a81f-89a7c21edc7d`::ref 1 aborting False >Thread-26750::INFO::2012-12-02 14:15:17,367::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26750::DEBUG::2012-12-02 14:15:17,367::task::1151::TaskManager.Task::(prepare) Task=`d5c7242a-ca50-4ad1-a81f-89a7c21edc7d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26750::DEBUG::2012-12-02 14:15:17,367::task::568::TaskManager.Task::(_updateState) Task=`d5c7242a-ca50-4ad1-a81f-89a7c21edc7d`::moving from state preparing -> state finished >Thread-26750::DEBUG::2012-12-02 14:15:17,368::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26750::DEBUG::2012-12-02 14:15:17,368::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26750::DEBUG::2012-12-02 14:15:17,368::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26750::DEBUG::2012-12-02 14:15:17,368::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26750::DEBUG::2012-12-02 14:15:17,368::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26750::DEBUG::2012-12-02 14:15:17,368::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26750::DEBUG::2012-12-02 14:15:17,368::task::957::TaskManager.Task::(_decref) Task=`d5c7242a-ca50-4ad1-a81f-89a7c21edc7d`::ref 0 aborting False >Thread-26754::DEBUG::2012-12-02 14:15:25,247::task::568::TaskManager.Task::(_updateState) Task=`9ae700d2-2fb0-48e0-9f89-ca80f0082345`::moving from state init -> state preparing >Thread-26754::INFO::2012-12-02 14:15:25,248::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26754::INFO::2012-12-02 14:15:25,248::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00795316696167', 'lastCheck': 1354450515.6963339, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00286507606506', 'lastCheck': 1354450520.787751, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00201106071472', 'lastCheck': 1354450523.6877191, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00635099411011', 'lastCheck': 1354450520.8465569, 'code': 0, 'valid': True}} >Thread-26754::DEBUG::2012-12-02 14:15:25,248::task::1151::TaskManager.Task::(prepare) Task=`9ae700d2-2fb0-48e0-9f89-ca80f0082345`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00795316696167', 'lastCheck': 1354450515.6963339, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00286507606506', 'lastCheck': 1354450520.787751, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00201106071472', 'lastCheck': 1354450523.6877191, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00635099411011', 'lastCheck': 1354450520.8465569, 'code': 0, 'valid': True}} >Thread-26754::DEBUG::2012-12-02 14:15:25,248::task::568::TaskManager.Task::(_updateState) Task=`9ae700d2-2fb0-48e0-9f89-ca80f0082345`::moving from state preparing -> state finished >Thread-26754::DEBUG::2012-12-02 14:15:25,248::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26754::DEBUG::2012-12-02 14:15:25,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26754::DEBUG::2012-12-02 14:15:25,249::task::957::TaskManager.Task::(_decref) Task=`9ae700d2-2fb0-48e0-9f89-ca80f0082345`::ref 0 aborting False >Thread-26757::DEBUG::2012-12-02 14:15:27,511::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26757::DEBUG::2012-12-02 14:15:27,511::task::568::TaskManager.Task::(_updateState) Task=`fabdf53f-a72f-4d44-bee3-4c85e7126ff5`::moving from state init -> state preparing >Thread-26757::INFO::2012-12-02 14:15:27,511::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26757::INFO::2012-12-02 14:15:27,511::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26757::DEBUG::2012-12-02 14:15:27,512::task::1151::TaskManager.Task::(prepare) Task=`fabdf53f-a72f-4d44-bee3-4c85e7126ff5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26757::DEBUG::2012-12-02 14:15:27,512::task::568::TaskManager.Task::(_updateState) Task=`fabdf53f-a72f-4d44-bee3-4c85e7126ff5`::moving from state preparing -> state finished >Thread-26757::DEBUG::2012-12-02 14:15:27,512::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26757::DEBUG::2012-12-02 14:15:27,512::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26757::DEBUG::2012-12-02 14:15:27,512::task::957::TaskManager.Task::(_decref) Task=`fabdf53f-a72f-4d44-bee3-4c85e7126ff5`::ref 0 aborting False >Thread-26758::DEBUG::2012-12-02 14:15:27,515::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26758::DEBUG::2012-12-02 14:15:27,515::task::568::TaskManager.Task::(_updateState) Task=`9b3c52e7-0fc0-4841-9843-6fb271f64dad`::moving from state init -> state preparing >Thread-26758::INFO::2012-12-02 14:15:27,515::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26758::DEBUG::2012-12-02 14:15:27,515::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`45fc0b29-e17e-47a7-957d-323efacebd29`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26758::DEBUG::2012-12-02 14:15:27,516::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26758::DEBUG::2012-12-02 14:15:27,516::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26758::DEBUG::2012-12-02 14:15:27,516::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`45fc0b29-e17e-47a7-957d-323efacebd29`::Granted request >Thread-26758::DEBUG::2012-12-02 14:15:27,516::task::794::TaskManager.Task::(resourceAcquired) Task=`9b3c52e7-0fc0-4841-9843-6fb271f64dad`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26758::DEBUG::2012-12-02 14:15:27,516::task::957::TaskManager.Task::(_decref) Task=`9b3c52e7-0fc0-4841-9843-6fb271f64dad`::ref 1 aborting False >Thread-26758::INFO::2012-12-02 14:15:27,523::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26758::DEBUG::2012-12-02 14:15:27,523::task::1151::TaskManager.Task::(prepare) Task=`9b3c52e7-0fc0-4841-9843-6fb271f64dad`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26758::DEBUG::2012-12-02 14:15:27,523::task::568::TaskManager.Task::(_updateState) Task=`9b3c52e7-0fc0-4841-9843-6fb271f64dad`::moving from state preparing -> state finished >Thread-26758::DEBUG::2012-12-02 14:15:27,523::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26758::DEBUG::2012-12-02 14:15:27,523::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26758::DEBUG::2012-12-02 14:15:27,523::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26758::DEBUG::2012-12-02 14:15:27,524::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26758::DEBUG::2012-12-02 14:15:27,524::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26758::DEBUG::2012-12-02 14:15:27,524::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26758::DEBUG::2012-12-02 14:15:27,524::task::957::TaskManager.Task::(_decref) Task=`9b3c52e7-0fc0-4841-9843-6fb271f64dad`::ref 0 aborting False >Thread-26762::DEBUG::2012-12-02 14:15:35,370::task::568::TaskManager.Task::(_updateState) Task=`f9538579-265c-4766-a31f-77d26ad0dad6`::moving from state init -> state preparing >Thread-26762::INFO::2012-12-02 14:15:35,370::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26762::INFO::2012-12-02 14:15:35,370::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00948882102966', 'lastCheck': 1354450525.7153909, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00349617004395', 'lastCheck': 1354450530.80144, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00304198265076', 'lastCheck': 1354450533.7016721, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0070219039917', 'lastCheck': 1354450530.8623641, 'code': 0, 'valid': True}} >Thread-26762::DEBUG::2012-12-02 14:15:35,370::task::1151::TaskManager.Task::(prepare) Task=`f9538579-265c-4766-a31f-77d26ad0dad6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00948882102966', 'lastCheck': 1354450525.7153909, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00349617004395', 'lastCheck': 1354450530.80144, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00304198265076', 'lastCheck': 1354450533.7016721, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0070219039917', 'lastCheck': 1354450530.8623641, 'code': 0, 'valid': True}} >Thread-26762::DEBUG::2012-12-02 14:15:35,370::task::568::TaskManager.Task::(_updateState) Task=`f9538579-265c-4766-a31f-77d26ad0dad6`::moving from state preparing -> state finished >Thread-26762::DEBUG::2012-12-02 14:15:35,371::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26762::DEBUG::2012-12-02 14:15:35,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26762::DEBUG::2012-12-02 14:15:35,371::task::957::TaskManager.Task::(_decref) Task=`f9538579-265c-4766-a31f-77d26ad0dad6`::ref 0 aborting False >Thread-26765::DEBUG::2012-12-02 14:15:37,640::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26765::DEBUG::2012-12-02 14:15:37,640::task::568::TaskManager.Task::(_updateState) Task=`087b3d26-e94b-49e1-a82f-ba01b2674765`::moving from state init -> state preparing >Thread-26765::INFO::2012-12-02 14:15:37,641::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26765::INFO::2012-12-02 14:15:37,641::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26765::DEBUG::2012-12-02 14:15:37,641::task::1151::TaskManager.Task::(prepare) Task=`087b3d26-e94b-49e1-a82f-ba01b2674765`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26765::DEBUG::2012-12-02 14:15:37,641::task::568::TaskManager.Task::(_updateState) Task=`087b3d26-e94b-49e1-a82f-ba01b2674765`::moving from state preparing -> state finished >Thread-26765::DEBUG::2012-12-02 14:15:37,641::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26765::DEBUG::2012-12-02 14:15:37,641::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26765::DEBUG::2012-12-02 14:15:37,641::task::957::TaskManager.Task::(_decref) Task=`087b3d26-e94b-49e1-a82f-ba01b2674765`::ref 0 aborting False >Thread-26766::DEBUG::2012-12-02 14:15:37,644::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26766::DEBUG::2012-12-02 14:15:37,644::task::568::TaskManager.Task::(_updateState) Task=`24e9e463-35eb-4061-9148-c228206a3b5e`::moving from state init -> state preparing >Thread-26766::INFO::2012-12-02 14:15:37,644::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26766::DEBUG::2012-12-02 14:15:37,645::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`fadc028b-f525-4281-bd34-1776f6a35e27`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26766::DEBUG::2012-12-02 14:15:37,645::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26766::DEBUG::2012-12-02 14:15:37,645::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26766::DEBUG::2012-12-02 14:15:37,645::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`fadc028b-f525-4281-bd34-1776f6a35e27`::Granted request >Thread-26766::DEBUG::2012-12-02 14:15:37,645::task::794::TaskManager.Task::(resourceAcquired) Task=`24e9e463-35eb-4061-9148-c228206a3b5e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26766::DEBUG::2012-12-02 14:15:37,645::task::957::TaskManager.Task::(_decref) Task=`24e9e463-35eb-4061-9148-c228206a3b5e`::ref 1 aborting False >Thread-26766::INFO::2012-12-02 14:15:37,650::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26766::DEBUG::2012-12-02 14:15:37,650::task::1151::TaskManager.Task::(prepare) Task=`24e9e463-35eb-4061-9148-c228206a3b5e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26766::DEBUG::2012-12-02 14:15:37,650::task::568::TaskManager.Task::(_updateState) Task=`24e9e463-35eb-4061-9148-c228206a3b5e`::moving from state preparing -> state finished >Thread-26766::DEBUG::2012-12-02 14:15:37,650::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26766::DEBUG::2012-12-02 14:15:37,650::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26766::DEBUG::2012-12-02 14:15:37,650::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26766::DEBUG::2012-12-02 14:15:37,651::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26766::DEBUG::2012-12-02 14:15:37,651::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26766::DEBUG::2012-12-02 14:15:37,651::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26766::DEBUG::2012-12-02 14:15:37,651::task::957::TaskManager.Task::(_decref) Task=`24e9e463-35eb-4061-9148-c228206a3b5e`::ref 0 aborting False >Thread-26770::DEBUG::2012-12-02 14:15:45,506::task::568::TaskManager.Task::(_updateState) Task=`c8a371da-bebc-4f7c-b882-713307228198`::moving from state init -> state preparing >Thread-26770::INFO::2012-12-02 14:15:45,506::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26770::INFO::2012-12-02 14:15:45,506::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00861501693726', 'lastCheck': 1354450535.7353699, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00160098075867', 'lastCheck': 1354450540.8164029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00218415260315', 'lastCheck': 1354450543.7129619, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0085129737854', 'lastCheck': 1354450540.882549, 'code': 0, 'valid': True}} >Thread-26770::DEBUG::2012-12-02 14:15:45,506::task::1151::TaskManager.Task::(prepare) Task=`c8a371da-bebc-4f7c-b882-713307228198`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00861501693726', 'lastCheck': 1354450535.7353699, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00160098075867', 'lastCheck': 1354450540.8164029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00218415260315', 'lastCheck': 1354450543.7129619, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0085129737854', 'lastCheck': 1354450540.882549, 'code': 0, 'valid': True}} >Thread-26770::DEBUG::2012-12-02 14:15:45,506::task::568::TaskManager.Task::(_updateState) Task=`c8a371da-bebc-4f7c-b882-713307228198`::moving from state preparing -> state finished >Thread-26770::DEBUG::2012-12-02 14:15:45,507::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26770::DEBUG::2012-12-02 14:15:45,507::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26770::DEBUG::2012-12-02 14:15:45,507::task::957::TaskManager.Task::(_decref) Task=`c8a371da-bebc-4f7c-b882-713307228198`::ref 0 aborting False >Thread-26773::DEBUG::2012-12-02 14:15:47,753::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26773::DEBUG::2012-12-02 14:15:47,754::task::568::TaskManager.Task::(_updateState) Task=`ba52b406-5b30-4a6d-82d3-57288c1e97e4`::moving from state init -> state preparing >Thread-26773::INFO::2012-12-02 14:15:47,754::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26773::INFO::2012-12-02 14:15:47,754::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26773::DEBUG::2012-12-02 14:15:47,754::task::1151::TaskManager.Task::(prepare) Task=`ba52b406-5b30-4a6d-82d3-57288c1e97e4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26773::DEBUG::2012-12-02 14:15:47,754::task::568::TaskManager.Task::(_updateState) Task=`ba52b406-5b30-4a6d-82d3-57288c1e97e4`::moving from state preparing -> state finished >Thread-26773::DEBUG::2012-12-02 14:15:47,754::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26773::DEBUG::2012-12-02 14:15:47,754::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26773::DEBUG::2012-12-02 14:15:47,755::task::957::TaskManager.Task::(_decref) Task=`ba52b406-5b30-4a6d-82d3-57288c1e97e4`::ref 0 aborting False >Thread-26774::DEBUG::2012-12-02 14:15:47,757::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26774::DEBUG::2012-12-02 14:15:47,757::task::568::TaskManager.Task::(_updateState) Task=`0b913451-50a0-4d3f-a2b9-5a7542126d76`::moving from state init -> state preparing >Thread-26774::INFO::2012-12-02 14:15:47,757::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26774::DEBUG::2012-12-02 14:15:47,758::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3f8496d6-3591-40f2-94c9-5a3bc0c32319`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26774::DEBUG::2012-12-02 14:15:47,758::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26774::DEBUG::2012-12-02 14:15:47,758::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26774::DEBUG::2012-12-02 14:15:47,758::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3f8496d6-3591-40f2-94c9-5a3bc0c32319`::Granted request >Thread-26774::DEBUG::2012-12-02 14:15:47,758::task::794::TaskManager.Task::(resourceAcquired) Task=`0b913451-50a0-4d3f-a2b9-5a7542126d76`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26774::DEBUG::2012-12-02 14:15:47,759::task::957::TaskManager.Task::(_decref) Task=`0b913451-50a0-4d3f-a2b9-5a7542126d76`::ref 1 aborting False >Thread-26774::INFO::2012-12-02 14:15:47,765::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26774::DEBUG::2012-12-02 14:15:47,765::task::1151::TaskManager.Task::(prepare) Task=`0b913451-50a0-4d3f-a2b9-5a7542126d76`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26774::DEBUG::2012-12-02 14:15:47,765::task::568::TaskManager.Task::(_updateState) Task=`0b913451-50a0-4d3f-a2b9-5a7542126d76`::moving from state preparing -> state finished >Thread-26774::DEBUG::2012-12-02 14:15:47,765::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26774::DEBUG::2012-12-02 14:15:47,765::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26774::DEBUG::2012-12-02 14:15:47,766::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26774::DEBUG::2012-12-02 14:15:47,766::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26774::DEBUG::2012-12-02 14:15:47,766::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26774::DEBUG::2012-12-02 14:15:47,766::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26774::DEBUG::2012-12-02 14:15:47,766::task::957::TaskManager.Task::(_decref) Task=`0b913451-50a0-4d3f-a2b9-5a7542126d76`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:15:53,713::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:15:53,713::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:15:53,713::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:15:53,921::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:15:53,922::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:15:53,922::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:15:53,956::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:15:53,956::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:15:53,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:15:53,961::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:15:53,962::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:15:53,962::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-26839::DEBUG::2012-12-02 14:15:55,611::task::568::TaskManager.Task::(_updateState) Task=`16de643f-111a-430e-ac17-8d8f9d77e947`::moving from state init -> state preparing >Thread-26839::INFO::2012-12-02 14:15:55,611::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-26839::INFO::2012-12-02 14:15:55,611::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00788879394531', 'lastCheck': 1354450545.752527, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00278091430664', 'lastCheck': 1354450550.8307719, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00187492370605', 'lastCheck': 1354450553.9725921, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00757598876953', 'lastCheck': 1354450550.8982329, 'code': 0, 'valid': True}} >Thread-26839::DEBUG::2012-12-02 14:15:55,611::task::1151::TaskManager.Task::(prepare) Task=`16de643f-111a-430e-ac17-8d8f9d77e947`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00788879394531', 'lastCheck': 1354450545.752527, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00278091430664', 'lastCheck': 1354450550.8307719, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00187492370605', 'lastCheck': 1354450553.9725921, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00757598876953', 'lastCheck': 1354450550.8982329, 'code': 0, 'valid': True}} >Thread-26839::DEBUG::2012-12-02 14:15:55,612::task::568::TaskManager.Task::(_updateState) Task=`16de643f-111a-430e-ac17-8d8f9d77e947`::moving from state preparing -> state finished >Thread-26839::DEBUG::2012-12-02 14:15:55,612::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26839::DEBUG::2012-12-02 14:15:55,612::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26839::DEBUG::2012-12-02 14:15:55,612::task::957::TaskManager.Task::(_decref) Task=`16de643f-111a-430e-ac17-8d8f9d77e947`::ref 0 aborting False >Thread-11821::DEBUG::2012-12-02 14:15:55,752::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:15:55,753::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:15:55,753::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:15:55,958::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:15:55,959::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:15:55,960::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:15:55,994::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:15:55,994::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:15:56,017::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:15:56,019::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:15:56,019::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:15:56,020::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-26903::DEBUG::2012-12-02 14:15:57,923::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26903::DEBUG::2012-12-02 14:15:57,923::task::568::TaskManager.Task::(_updateState) Task=`b133c4c1-07ca-453e-9ef6-c30a40beae1d`::moving from state init -> state preparing >Thread-26903::INFO::2012-12-02 14:15:57,923::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26903::INFO::2012-12-02 14:15:57,924::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26903::DEBUG::2012-12-02 14:15:57,924::task::1151::TaskManager.Task::(prepare) Task=`b133c4c1-07ca-453e-9ef6-c30a40beae1d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-26903::DEBUG::2012-12-02 14:15:57,924::task::568::TaskManager.Task::(_updateState) Task=`b133c4c1-07ca-453e-9ef6-c30a40beae1d`::moving from state preparing -> state finished >Thread-26903::DEBUG::2012-12-02 14:15:57,924::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-26903::DEBUG::2012-12-02 14:15:57,924::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26903::DEBUG::2012-12-02 14:15:57,924::task::957::TaskManager.Task::(_decref) Task=`b133c4c1-07ca-453e-9ef6-c30a40beae1d`::ref 0 aborting False >Thread-26904::DEBUG::2012-12-02 14:15:57,927::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-26904::DEBUG::2012-12-02 14:15:57,927::task::568::TaskManager.Task::(_updateState) Task=`653b9b56-bbb6-4431-91df-7c924b183977`::moving from state init -> state preparing >Thread-26904::INFO::2012-12-02 14:15:57,927::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-26904::DEBUG::2012-12-02 14:15:57,927::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d8717868-3eb2-4319-aee2-0e9712c59eb7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-26904::DEBUG::2012-12-02 14:15:57,928::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-26904::DEBUG::2012-12-02 14:15:57,928::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-26904::DEBUG::2012-12-02 14:15:57,928::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d8717868-3eb2-4319-aee2-0e9712c59eb7`::Granted request >Thread-26904::DEBUG::2012-12-02 14:15:57,928::task::794::TaskManager.Task::(resourceAcquired) Task=`653b9b56-bbb6-4431-91df-7c924b183977`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-26904::DEBUG::2012-12-02 14:15:57,928::task::957::TaskManager.Task::(_decref) Task=`653b9b56-bbb6-4431-91df-7c924b183977`::ref 1 aborting False >Thread-26904::INFO::2012-12-02 14:15:57,935::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26904::DEBUG::2012-12-02 14:15:57,935::task::1151::TaskManager.Task::(prepare) Task=`653b9b56-bbb6-4431-91df-7c924b183977`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-26904::DEBUG::2012-12-02 14:15:57,935::task::568::TaskManager.Task::(_updateState) Task=`653b9b56-bbb6-4431-91df-7c924b183977`::moving from state preparing -> state finished >Thread-26904::DEBUG::2012-12-02 14:15:57,935::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-26904::DEBUG::2012-12-02 14:15:57,935::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-26904::DEBUG::2012-12-02 14:15:57,935::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-26904::DEBUG::2012-12-02 14:15:57,936::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-26904::DEBUG::2012-12-02 14:15:57,936::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-26904::DEBUG::2012-12-02 14:15:57,936::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-26904::DEBUG::2012-12-02 14:15:57,936::task::957::TaskManager.Task::(_decref) Task=`653b9b56-bbb6-4431-91df-7c924b183977`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:16:00,830::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:16:00,831::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:16:00,831::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:16:01,041::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:16:01,042::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:16:01,043::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:16:01,078::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:16:01,078::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:16:01,081::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:16:01,084::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:16:01,084::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:16:01,085::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:16:01,085::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:16:01,085::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:16:01,088::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:16:01,291::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:16:01,292::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:16:01,292::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:16:01,326::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:16:01,327::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:16:01,335::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:16:01,338::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:16:01,338::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:16:01,338::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-27030::DEBUG::2012-12-02 14:16:05,690::task::568::TaskManager.Task::(_updateState) Task=`7e2216fc-6c9c-49c2-8eae-558d6ba3c9cf`::moving from state init -> state preparing >Thread-27030::INFO::2012-12-02 14:16:05,690::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27030::INFO::2012-12-02 14:16:05,690::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00332593917847', 'lastCheck': 1354450556.031517, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00218391418457', 'lastCheck': 1354450561.346632, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00182795524597', 'lastCheck': 1354450563.982789, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00666213035583', 'lastCheck': 1354450561.3550611, 'code': 0, 'valid': True}} >Thread-27030::DEBUG::2012-12-02 14:16:05,690::task::1151::TaskManager.Task::(prepare) Task=`7e2216fc-6c9c-49c2-8eae-558d6ba3c9cf`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00332593917847', 'lastCheck': 1354450556.031517, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00218391418457', 'lastCheck': 1354450561.346632, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00182795524597', 'lastCheck': 1354450563.982789, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00666213035583', 'lastCheck': 1354450561.3550611, 'code': 0, 'valid': True}} >Thread-27030::DEBUG::2012-12-02 14:16:05,690::task::568::TaskManager.Task::(_updateState) Task=`7e2216fc-6c9c-49c2-8eae-558d6ba3c9cf`::moving from state preparing -> state finished >Thread-27030::DEBUG::2012-12-02 14:16:05,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27030::DEBUG::2012-12-02 14:16:05,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27030::DEBUG::2012-12-02 14:16:05,691::task::957::TaskManager.Task::(_decref) Task=`7e2216fc-6c9c-49c2-8eae-558d6ba3c9cf`::ref 0 aborting False >Thread-27033::DEBUG::2012-12-02 14:16:08,068::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27033::DEBUG::2012-12-02 14:16:08,068::task::568::TaskManager.Task::(_updateState) Task=`bd533da6-035b-4587-9745-de015ec0f439`::moving from state init -> state preparing >Thread-27033::INFO::2012-12-02 14:16:08,068::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27033::INFO::2012-12-02 14:16:08,069::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27033::DEBUG::2012-12-02 14:16:08,069::task::1151::TaskManager.Task::(prepare) Task=`bd533da6-035b-4587-9745-de015ec0f439`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27033::DEBUG::2012-12-02 14:16:08,069::task::568::TaskManager.Task::(_updateState) Task=`bd533da6-035b-4587-9745-de015ec0f439`::moving from state preparing -> state finished >Thread-27033::DEBUG::2012-12-02 14:16:08,069::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27033::DEBUG::2012-12-02 14:16:08,069::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27033::DEBUG::2012-12-02 14:16:08,069::task::957::TaskManager.Task::(_decref) Task=`bd533da6-035b-4587-9745-de015ec0f439`::ref 0 aborting False >Thread-27034::DEBUG::2012-12-02 14:16:08,072::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27034::DEBUG::2012-12-02 14:16:08,072::task::568::TaskManager.Task::(_updateState) Task=`dcdaee83-2a3d-47e5-a264-34b7efb001f8`::moving from state init -> state preparing >Thread-27034::INFO::2012-12-02 14:16:08,072::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27034::DEBUG::2012-12-02 14:16:08,072::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`da0afba3-144a-4dda-82af-6e0635cc2be0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27034::DEBUG::2012-12-02 14:16:08,073::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27034::DEBUG::2012-12-02 14:16:08,073::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27034::DEBUG::2012-12-02 14:16:08,073::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`da0afba3-144a-4dda-82af-6e0635cc2be0`::Granted request >Thread-27034::DEBUG::2012-12-02 14:16:08,073::task::794::TaskManager.Task::(resourceAcquired) Task=`dcdaee83-2a3d-47e5-a264-34b7efb001f8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27034::DEBUG::2012-12-02 14:16:08,073::task::957::TaskManager.Task::(_decref) Task=`dcdaee83-2a3d-47e5-a264-34b7efb001f8`::ref 1 aborting False >Thread-27034::INFO::2012-12-02 14:16:08,079::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27034::DEBUG::2012-12-02 14:16:08,079::task::1151::TaskManager.Task::(prepare) Task=`dcdaee83-2a3d-47e5-a264-34b7efb001f8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27034::DEBUG::2012-12-02 14:16:08,079::task::568::TaskManager.Task::(_updateState) Task=`dcdaee83-2a3d-47e5-a264-34b7efb001f8`::moving from state preparing -> state finished >Thread-27034::DEBUG::2012-12-02 14:16:08,079::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27034::DEBUG::2012-12-02 14:16:08,079::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27034::DEBUG::2012-12-02 14:16:08,079::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27034::DEBUG::2012-12-02 14:16:08,079::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27034::DEBUG::2012-12-02 14:16:08,079::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27034::DEBUG::2012-12-02 14:16:08,079::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27034::DEBUG::2012-12-02 14:16:08,079::task::957::TaskManager.Task::(_decref) Task=`dcdaee83-2a3d-47e5-a264-34b7efb001f8`::ref 0 aborting False >Thread-27038::DEBUG::2012-12-02 14:16:15,785::task::568::TaskManager.Task::(_updateState) Task=`18db56b6-434d-4df9-b2a8-19e0acf926cd`::moving from state init -> state preparing >Thread-27038::INFO::2012-12-02 14:16:15,785::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27038::INFO::2012-12-02 14:16:15,785::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00362801551819', 'lastCheck': 1354450566.0436089, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00484299659729', 'lastCheck': 1354450571.365279, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00260210037231', 'lastCheck': 1354450573.9939289, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0065381526947', 'lastCheck': 1354450571.3735399, 'code': 0, 'valid': True}} >Thread-27038::DEBUG::2012-12-02 14:16:15,785::task::1151::TaskManager.Task::(prepare) Task=`18db56b6-434d-4df9-b2a8-19e0acf926cd`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00362801551819', 'lastCheck': 1354450566.0436089, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00484299659729', 'lastCheck': 1354450571.365279, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00260210037231', 'lastCheck': 1354450573.9939289, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0065381526947', 'lastCheck': 1354450571.3735399, 'code': 0, 'valid': True}} >Thread-27038::DEBUG::2012-12-02 14:16:15,785::task::568::TaskManager.Task::(_updateState) Task=`18db56b6-434d-4df9-b2a8-19e0acf926cd`::moving from state preparing -> state finished >Thread-27038::DEBUG::2012-12-02 14:16:15,786::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27038::DEBUG::2012-12-02 14:16:15,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27038::DEBUG::2012-12-02 14:16:15,786::task::957::TaskManager.Task::(_decref) Task=`18db56b6-434d-4df9-b2a8-19e0acf926cd`::ref 0 aborting False >Thread-27041::DEBUG::2012-12-02 14:16:18,212::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27041::DEBUG::2012-12-02 14:16:18,213::task::568::TaskManager.Task::(_updateState) Task=`2af9f7f9-eb5a-4926-ab3c-18dbfbf10c1c`::moving from state init -> state preparing >Thread-27041::INFO::2012-12-02 14:16:18,213::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27041::INFO::2012-12-02 14:16:18,213::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27041::DEBUG::2012-12-02 14:16:18,213::task::1151::TaskManager.Task::(prepare) Task=`2af9f7f9-eb5a-4926-ab3c-18dbfbf10c1c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27041::DEBUG::2012-12-02 14:16:18,213::task::568::TaskManager.Task::(_updateState) Task=`2af9f7f9-eb5a-4926-ab3c-18dbfbf10c1c`::moving from state preparing -> state finished >Thread-27041::DEBUG::2012-12-02 14:16:18,213::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27041::DEBUG::2012-12-02 14:16:18,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27041::DEBUG::2012-12-02 14:16:18,214::task::957::TaskManager.Task::(_decref) Task=`2af9f7f9-eb5a-4926-ab3c-18dbfbf10c1c`::ref 0 aborting False >Thread-27042::DEBUG::2012-12-02 14:16:18,216::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27042::DEBUG::2012-12-02 14:16:18,216::task::568::TaskManager.Task::(_updateState) Task=`bfaf61c9-2325-44c2-81f6-0da9ba52b1f7`::moving from state init -> state preparing >Thread-27042::INFO::2012-12-02 14:16:18,217::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27042::DEBUG::2012-12-02 14:16:18,217::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a0d1678-d92c-44bb-8947-9581aed36e01`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27042::DEBUG::2012-12-02 14:16:18,217::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27042::DEBUG::2012-12-02 14:16:18,217::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27042::DEBUG::2012-12-02 14:16:18,217::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a0d1678-d92c-44bb-8947-9581aed36e01`::Granted request >Thread-27042::DEBUG::2012-12-02 14:16:18,218::task::794::TaskManager.Task::(resourceAcquired) Task=`bfaf61c9-2325-44c2-81f6-0da9ba52b1f7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27042::DEBUG::2012-12-02 14:16:18,218::task::957::TaskManager.Task::(_decref) Task=`bfaf61c9-2325-44c2-81f6-0da9ba52b1f7`::ref 1 aborting False >Thread-27042::INFO::2012-12-02 14:16:18,224::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27042::DEBUG::2012-12-02 14:16:18,224::task::1151::TaskManager.Task::(prepare) Task=`bfaf61c9-2325-44c2-81f6-0da9ba52b1f7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27042::DEBUG::2012-12-02 14:16:18,224::task::568::TaskManager.Task::(_updateState) Task=`bfaf61c9-2325-44c2-81f6-0da9ba52b1f7`::moving from state preparing -> state finished >Thread-27042::DEBUG::2012-12-02 14:16:18,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27042::DEBUG::2012-12-02 14:16:18,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27042::DEBUG::2012-12-02 14:16:18,225::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27042::DEBUG::2012-12-02 14:16:18,225::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27042::DEBUG::2012-12-02 14:16:18,225::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27042::DEBUG::2012-12-02 14:16:18,225::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27042::DEBUG::2012-12-02 14:16:18,225::task::957::TaskManager.Task::(_decref) Task=`bfaf61c9-2325-44c2-81f6-0da9ba52b1f7`::ref 0 aborting False >Thread-27046::DEBUG::2012-12-02 14:16:25,889::task::568::TaskManager.Task::(_updateState) Task=`20e900a5-c2a1-4eb2-9119-d3a4adb2fba9`::moving from state init -> state preparing >Thread-27046::INFO::2012-12-02 14:16:25,889::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27046::INFO::2012-12-02 14:16:25,889::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0119519233704', 'lastCheck': 1354450576.0654211, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00333499908447', 'lastCheck': 1354450581.384795, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00272703170776', 'lastCheck': 1354450584.0038891, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00795912742615', 'lastCheck': 1354450581.392278, 'code': 0, 'valid': True}} >Thread-27046::DEBUG::2012-12-02 14:16:25,889::task::1151::TaskManager.Task::(prepare) Task=`20e900a5-c2a1-4eb2-9119-d3a4adb2fba9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0119519233704', 'lastCheck': 1354450576.0654211, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00333499908447', 'lastCheck': 1354450581.384795, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00272703170776', 'lastCheck': 1354450584.0038891, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00795912742615', 'lastCheck': 1354450581.392278, 'code': 0, 'valid': True}} >Thread-27046::DEBUG::2012-12-02 14:16:25,889::task::568::TaskManager.Task::(_updateState) Task=`20e900a5-c2a1-4eb2-9119-d3a4adb2fba9`::moving from state preparing -> state finished >Thread-27046::DEBUG::2012-12-02 14:16:25,889::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27046::DEBUG::2012-12-02 14:16:25,890::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27046::DEBUG::2012-12-02 14:16:25,890::task::957::TaskManager.Task::(_decref) Task=`20e900a5-c2a1-4eb2-9119-d3a4adb2fba9`::ref 0 aborting False >Thread-27049::DEBUG::2012-12-02 14:16:28,333::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27049::DEBUG::2012-12-02 14:16:28,334::task::568::TaskManager.Task::(_updateState) Task=`b40299ab-eec4-43a3-9efb-7ed2b7ab9e5a`::moving from state init -> state preparing >Thread-27049::INFO::2012-12-02 14:16:28,334::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27049::INFO::2012-12-02 14:16:28,334::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27049::DEBUG::2012-12-02 14:16:28,334::task::1151::TaskManager.Task::(prepare) Task=`b40299ab-eec4-43a3-9efb-7ed2b7ab9e5a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27049::DEBUG::2012-12-02 14:16:28,334::task::568::TaskManager.Task::(_updateState) Task=`b40299ab-eec4-43a3-9efb-7ed2b7ab9e5a`::moving from state preparing -> state finished >Thread-27049::DEBUG::2012-12-02 14:16:28,335::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27049::DEBUG::2012-12-02 14:16:28,335::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27049::DEBUG::2012-12-02 14:16:28,335::task::957::TaskManager.Task::(_decref) Task=`b40299ab-eec4-43a3-9efb-7ed2b7ab9e5a`::ref 0 aborting False >Thread-27050::DEBUG::2012-12-02 14:16:28,337::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27050::DEBUG::2012-12-02 14:16:28,338::task::568::TaskManager.Task::(_updateState) Task=`a14df9b6-539c-4a9a-9d27-f5fe872e9bd9`::moving from state init -> state preparing >Thread-27050::INFO::2012-12-02 14:16:28,338::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27050::DEBUG::2012-12-02 14:16:28,338::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e55c4a1d-12f3-4d71-b5aa-f6d2bfc3db8a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27050::DEBUG::2012-12-02 14:16:28,338::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27050::DEBUG::2012-12-02 14:16:28,338::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27050::DEBUG::2012-12-02 14:16:28,339::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e55c4a1d-12f3-4d71-b5aa-f6d2bfc3db8a`::Granted request >Thread-27050::DEBUG::2012-12-02 14:16:28,339::task::794::TaskManager.Task::(resourceAcquired) Task=`a14df9b6-539c-4a9a-9d27-f5fe872e9bd9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27050::DEBUG::2012-12-02 14:16:28,339::task::957::TaskManager.Task::(_decref) Task=`a14df9b6-539c-4a9a-9d27-f5fe872e9bd9`::ref 1 aborting False >Thread-27050::INFO::2012-12-02 14:16:28,345::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27050::DEBUG::2012-12-02 14:16:28,345::task::1151::TaskManager.Task::(prepare) Task=`a14df9b6-539c-4a9a-9d27-f5fe872e9bd9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27050::DEBUG::2012-12-02 14:16:28,346::task::568::TaskManager.Task::(_updateState) Task=`a14df9b6-539c-4a9a-9d27-f5fe872e9bd9`::moving from state preparing -> state finished >Thread-27050::DEBUG::2012-12-02 14:16:28,346::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27050::DEBUG::2012-12-02 14:16:28,346::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27050::DEBUG::2012-12-02 14:16:28,346::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27050::DEBUG::2012-12-02 14:16:28,346::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27050::DEBUG::2012-12-02 14:16:28,346::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27050::DEBUG::2012-12-02 14:16:28,346::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27050::DEBUG::2012-12-02 14:16:28,346::task::957::TaskManager.Task::(_decref) Task=`a14df9b6-539c-4a9a-9d27-f5fe872e9bd9`::ref 0 aborting False >Thread-27054::DEBUG::2012-12-02 14:16:35,984::task::568::TaskManager.Task::(_updateState) Task=`6caa8044-b273-4b6e-836c-03d6432c7db9`::moving from state init -> state preparing >Thread-27054::INFO::2012-12-02 14:16:35,984::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27054::INFO::2012-12-02 14:16:35,985::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00344491004944', 'lastCheck': 1354450586.0773649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00426697731018', 'lastCheck': 1354450591.4005549, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00202894210815', 'lastCheck': 1354450594.0791121, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00678086280823', 'lastCheck': 1354450591.4081869, 'code': 0, 'valid': True}} >Thread-27054::DEBUG::2012-12-02 14:16:35,985::task::1151::TaskManager.Task::(prepare) Task=`6caa8044-b273-4b6e-836c-03d6432c7db9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00344491004944', 'lastCheck': 1354450586.0773649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00426697731018', 'lastCheck': 1354450591.4005549, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00202894210815', 'lastCheck': 1354450594.0791121, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00678086280823', 'lastCheck': 1354450591.4081869, 'code': 0, 'valid': True}} >Thread-27054::DEBUG::2012-12-02 14:16:35,985::task::568::TaskManager.Task::(_updateState) Task=`6caa8044-b273-4b6e-836c-03d6432c7db9`::moving from state preparing -> state finished >Thread-27054::DEBUG::2012-12-02 14:16:35,985::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27054::DEBUG::2012-12-02 14:16:35,985::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27054::DEBUG::2012-12-02 14:16:35,985::task::957::TaskManager.Task::(_decref) Task=`6caa8044-b273-4b6e-836c-03d6432c7db9`::ref 0 aborting False >Thread-27057::DEBUG::2012-12-02 14:16:38,471::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27057::DEBUG::2012-12-02 14:16:38,471::task::568::TaskManager.Task::(_updateState) Task=`c5e12cd6-3ab1-447d-a085-a3ed85aa68d7`::moving from state init -> state preparing >Thread-27057::INFO::2012-12-02 14:16:38,471::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27057::INFO::2012-12-02 14:16:38,471::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27057::DEBUG::2012-12-02 14:16:38,471::task::1151::TaskManager.Task::(prepare) Task=`c5e12cd6-3ab1-447d-a085-a3ed85aa68d7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27057::DEBUG::2012-12-02 14:16:38,472::task::568::TaskManager.Task::(_updateState) Task=`c5e12cd6-3ab1-447d-a085-a3ed85aa68d7`::moving from state preparing -> state finished >Thread-27057::DEBUG::2012-12-02 14:16:38,472::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27057::DEBUG::2012-12-02 14:16:38,472::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27057::DEBUG::2012-12-02 14:16:38,472::task::957::TaskManager.Task::(_decref) Task=`c5e12cd6-3ab1-447d-a085-a3ed85aa68d7`::ref 0 aborting False >Thread-27058::DEBUG::2012-12-02 14:16:38,474::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27058::DEBUG::2012-12-02 14:16:38,475::task::568::TaskManager.Task::(_updateState) Task=`ca048dc3-6965-4901-a33e-a39092b5dc9a`::moving from state init -> state preparing >Thread-27058::INFO::2012-12-02 14:16:38,475::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27058::DEBUG::2012-12-02 14:16:38,475::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`692a9bcc-a2e1-47ea-8581-2060161060e0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27058::DEBUG::2012-12-02 14:16:38,475::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27058::DEBUG::2012-12-02 14:16:38,475::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27058::DEBUG::2012-12-02 14:16:38,476::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`692a9bcc-a2e1-47ea-8581-2060161060e0`::Granted request >Thread-27058::DEBUG::2012-12-02 14:16:38,476::task::794::TaskManager.Task::(resourceAcquired) Task=`ca048dc3-6965-4901-a33e-a39092b5dc9a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27058::DEBUG::2012-12-02 14:16:38,476::task::957::TaskManager.Task::(_decref) Task=`ca048dc3-6965-4901-a33e-a39092b5dc9a`::ref 1 aborting False >Thread-27058::INFO::2012-12-02 14:16:38,482::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27058::DEBUG::2012-12-02 14:16:38,482::task::1151::TaskManager.Task::(prepare) Task=`ca048dc3-6965-4901-a33e-a39092b5dc9a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27058::DEBUG::2012-12-02 14:16:38,483::task::568::TaskManager.Task::(_updateState) Task=`ca048dc3-6965-4901-a33e-a39092b5dc9a`::moving from state preparing -> state finished >Thread-27058::DEBUG::2012-12-02 14:16:38,483::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27058::DEBUG::2012-12-02 14:16:38,483::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27058::DEBUG::2012-12-02 14:16:38,483::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27058::DEBUG::2012-12-02 14:16:38,483::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27058::DEBUG::2012-12-02 14:16:38,483::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27058::DEBUG::2012-12-02 14:16:38,484::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27058::DEBUG::2012-12-02 14:16:38,484::task::957::TaskManager.Task::(_decref) Task=`ca048dc3-6965-4901-a33e-a39092b5dc9a`::ref 0 aborting False >Thread-27062::DEBUG::2012-12-02 14:16:46,140::task::568::TaskManager.Task::(_updateState) Task=`8e181cae-2389-4341-8b6b-76c56be682ab`::moving from state init -> state preparing >Thread-27062::INFO::2012-12-02 14:16:46,140::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27062::INFO::2012-12-02 14:16:46,140::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00950813293457', 'lastCheck': 1354450606.1147771, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00136804580688', 'lastCheck': 1354450601.4150591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00230598449707', 'lastCheck': 1354450604.0902641, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00396800041199', 'lastCheck': 1354450601.4219799, 'code': 0, 'valid': True}} >Thread-27062::DEBUG::2012-12-02 14:16:46,140::task::1151::TaskManager.Task::(prepare) Task=`8e181cae-2389-4341-8b6b-76c56be682ab`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00950813293457', 'lastCheck': 1354450606.1147771, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00136804580688', 'lastCheck': 1354450601.4150591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00230598449707', 'lastCheck': 1354450604.0902641, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00396800041199', 'lastCheck': 1354450601.4219799, 'code': 0, 'valid': True}} >Thread-27062::DEBUG::2012-12-02 14:16:46,141::task::568::TaskManager.Task::(_updateState) Task=`8e181cae-2389-4341-8b6b-76c56be682ab`::moving from state preparing -> state finished >Thread-27062::DEBUG::2012-12-02 14:16:46,141::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27062::DEBUG::2012-12-02 14:16:46,141::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27062::DEBUG::2012-12-02 14:16:46,141::task::957::TaskManager.Task::(_decref) Task=`8e181cae-2389-4341-8b6b-76c56be682ab`::ref 0 aborting False >Thread-27065::DEBUG::2012-12-02 14:16:48,575::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27065::DEBUG::2012-12-02 14:16:48,576::task::568::TaskManager.Task::(_updateState) Task=`6ec08a43-db2c-4edf-8af5-a6912c6fca65`::moving from state init -> state preparing >Thread-27065::INFO::2012-12-02 14:16:48,576::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27065::INFO::2012-12-02 14:16:48,576::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27065::DEBUG::2012-12-02 14:16:48,576::task::1151::TaskManager.Task::(prepare) Task=`6ec08a43-db2c-4edf-8af5-a6912c6fca65`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27065::DEBUG::2012-12-02 14:16:48,576::task::568::TaskManager.Task::(_updateState) Task=`6ec08a43-db2c-4edf-8af5-a6912c6fca65`::moving from state preparing -> state finished >Thread-27065::DEBUG::2012-12-02 14:16:48,577::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27065::DEBUG::2012-12-02 14:16:48,577::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27065::DEBUG::2012-12-02 14:16:48,577::task::957::TaskManager.Task::(_decref) Task=`6ec08a43-db2c-4edf-8af5-a6912c6fca65`::ref 0 aborting False >Thread-27066::DEBUG::2012-12-02 14:16:48,579::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27066::DEBUG::2012-12-02 14:16:48,580::task::568::TaskManager.Task::(_updateState) Task=`a2cb18d1-9cc5-4030-8724-6276cf85c268`::moving from state init -> state preparing >Thread-27066::INFO::2012-12-02 14:16:48,580::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27066::DEBUG::2012-12-02 14:16:48,580::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e14e93c-66d4-41b3-a3c2-6cecb8703888`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27066::DEBUG::2012-12-02 14:16:48,580::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27066::DEBUG::2012-12-02 14:16:48,580::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27066::DEBUG::2012-12-02 14:16:48,580::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e14e93c-66d4-41b3-a3c2-6cecb8703888`::Granted request >Thread-27066::DEBUG::2012-12-02 14:16:48,581::task::794::TaskManager.Task::(resourceAcquired) Task=`a2cb18d1-9cc5-4030-8724-6276cf85c268`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27066::DEBUG::2012-12-02 14:16:48,581::task::957::TaskManager.Task::(_decref) Task=`a2cb18d1-9cc5-4030-8724-6276cf85c268`::ref 1 aborting False >Thread-27066::INFO::2012-12-02 14:16:48,587::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27066::DEBUG::2012-12-02 14:16:48,587::task::1151::TaskManager.Task::(prepare) Task=`a2cb18d1-9cc5-4030-8724-6276cf85c268`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27066::DEBUG::2012-12-02 14:16:48,587::task::568::TaskManager.Task::(_updateState) Task=`a2cb18d1-9cc5-4030-8724-6276cf85c268`::moving from state preparing -> state finished >Thread-27066::DEBUG::2012-12-02 14:16:48,587::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27066::DEBUG::2012-12-02 14:16:48,587::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27066::DEBUG::2012-12-02 14:16:48,587::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27066::DEBUG::2012-12-02 14:16:48,588::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27066::DEBUG::2012-12-02 14:16:48,588::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27066::DEBUG::2012-12-02 14:16:48,588::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27066::DEBUG::2012-12-02 14:16:48,588::task::957::TaskManager.Task::(_decref) Task=`a2cb18d1-9cc5-4030-8724-6276cf85c268`::ref 0 aborting False >Thread-27070::DEBUG::2012-12-02 14:16:56,242::task::568::TaskManager.Task::(_updateState) Task=`8f91f72b-c2be-4b98-a9dc-18ef933c7c33`::moving from state init -> state preparing >Thread-27070::INFO::2012-12-02 14:16:56,242::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27070::INFO::2012-12-02 14:16:56,243::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0140588283539', 'lastCheck': 1354450616.1387391, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00265789031982', 'lastCheck': 1354450611.432318, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00200510025024', 'lastCheck': 1354450614.102726, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00696802139282', 'lastCheck': 1354450611.44087, 'code': 0, 'valid': True}} >Thread-27070::DEBUG::2012-12-02 14:16:56,243::task::1151::TaskManager.Task::(prepare) Task=`8f91f72b-c2be-4b98-a9dc-18ef933c7c33`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0140588283539', 'lastCheck': 1354450616.1387391, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00265789031982', 'lastCheck': 1354450611.432318, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00200510025024', 'lastCheck': 1354450614.102726, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00696802139282', 'lastCheck': 1354450611.44087, 'code': 0, 'valid': True}} >Thread-27070::DEBUG::2012-12-02 14:16:56,243::task::568::TaskManager.Task::(_updateState) Task=`8f91f72b-c2be-4b98-a9dc-18ef933c7c33`::moving from state preparing -> state finished >Thread-27070::DEBUG::2012-12-02 14:16:56,243::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27070::DEBUG::2012-12-02 14:16:56,243::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27070::DEBUG::2012-12-02 14:16:56,243::task::957::TaskManager.Task::(_decref) Task=`8f91f72b-c2be-4b98-a9dc-18ef933c7c33`::ref 0 aborting False >Thread-27073::DEBUG::2012-12-02 14:16:58,679::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27073::DEBUG::2012-12-02 14:16:58,679::task::568::TaskManager.Task::(_updateState) Task=`3238977b-bacf-462a-9966-2335acd73660`::moving from state init -> state preparing >Thread-27073::INFO::2012-12-02 14:16:58,680::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27073::INFO::2012-12-02 14:16:58,680::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27073::DEBUG::2012-12-02 14:16:58,680::task::1151::TaskManager.Task::(prepare) Task=`3238977b-bacf-462a-9966-2335acd73660`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27073::DEBUG::2012-12-02 14:16:58,680::task::568::TaskManager.Task::(_updateState) Task=`3238977b-bacf-462a-9966-2335acd73660`::moving from state preparing -> state finished >Thread-27073::DEBUG::2012-12-02 14:16:58,680::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27073::DEBUG::2012-12-02 14:16:58,680::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27073::DEBUG::2012-12-02 14:16:58,680::task::957::TaskManager.Task::(_decref) Task=`3238977b-bacf-462a-9966-2335acd73660`::ref 0 aborting False >Thread-27074::DEBUG::2012-12-02 14:16:58,683::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27074::DEBUG::2012-12-02 14:16:58,683::task::568::TaskManager.Task::(_updateState) Task=`be25dda2-59aa-41a8-9372-67b8703d225b`::moving from state init -> state preparing >Thread-27074::INFO::2012-12-02 14:16:58,683::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27074::DEBUG::2012-12-02 14:16:58,684::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6f23ce31-ddc7-4201-83fd-d507c6603df2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27074::DEBUG::2012-12-02 14:16:58,684::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27074::DEBUG::2012-12-02 14:16:58,684::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27074::DEBUG::2012-12-02 14:16:58,684::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6f23ce31-ddc7-4201-83fd-d507c6603df2`::Granted request >Thread-27074::DEBUG::2012-12-02 14:16:58,684::task::794::TaskManager.Task::(resourceAcquired) Task=`be25dda2-59aa-41a8-9372-67b8703d225b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27074::DEBUG::2012-12-02 14:16:58,685::task::957::TaskManager.Task::(_decref) Task=`be25dda2-59aa-41a8-9372-67b8703d225b`::ref 1 aborting False >Thread-27074::INFO::2012-12-02 14:16:58,689::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27074::DEBUG::2012-12-02 14:16:58,689::task::1151::TaskManager.Task::(prepare) Task=`be25dda2-59aa-41a8-9372-67b8703d225b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27074::DEBUG::2012-12-02 14:16:58,689::task::568::TaskManager.Task::(_updateState) Task=`be25dda2-59aa-41a8-9372-67b8703d225b`::moving from state preparing -> state finished >Thread-27074::DEBUG::2012-12-02 14:16:58,689::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27074::DEBUG::2012-12-02 14:16:58,689::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27074::DEBUG::2012-12-02 14:16:58,690::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27074::DEBUG::2012-12-02 14:16:58,690::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27074::DEBUG::2012-12-02 14:16:58,690::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27074::DEBUG::2012-12-02 14:16:58,690::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27074::DEBUG::2012-12-02 14:16:58,690::task::957::TaskManager.Task::(_decref) Task=`be25dda2-59aa-41a8-9372-67b8703d225b`::ref 0 aborting False >Thread-27078::DEBUG::2012-12-02 14:17:06,340::task::568::TaskManager.Task::(_updateState) Task=`62381612-c26f-4aa4-9963-7acab95eea80`::moving from state init -> state preparing >Thread-27078::INFO::2012-12-02 14:17:06,340::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27078::INFO::2012-12-02 14:17:06,340::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00809097290039', 'lastCheck': 1354450626.1562409, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00211095809937', 'lastCheck': 1354450621.449513, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00208210945129', 'lastCheck': 1354450624.1146181, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812101364136', 'lastCheck': 1354450621.459409, 'code': 0, 'valid': True}} >Thread-27078::DEBUG::2012-12-02 14:17:06,341::task::1151::TaskManager.Task::(prepare) Task=`62381612-c26f-4aa4-9963-7acab95eea80`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00809097290039', 'lastCheck': 1354450626.1562409, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00211095809937', 'lastCheck': 1354450621.449513, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00208210945129', 'lastCheck': 1354450624.1146181, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812101364136', 'lastCheck': 1354450621.459409, 'code': 0, 'valid': True}} >Thread-27078::DEBUG::2012-12-02 14:17:06,341::task::568::TaskManager.Task::(_updateState) Task=`62381612-c26f-4aa4-9963-7acab95eea80`::moving from state preparing -> state finished >Thread-27078::DEBUG::2012-12-02 14:17:06,341::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27078::DEBUG::2012-12-02 14:17:06,341::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27078::DEBUG::2012-12-02 14:17:06,341::task::957::TaskManager.Task::(_decref) Task=`62381612-c26f-4aa4-9963-7acab95eea80`::ref 0 aborting False >Thread-27081::DEBUG::2012-12-02 14:17:08,792::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27081::DEBUG::2012-12-02 14:17:08,792::task::568::TaskManager.Task::(_updateState) Task=`9f791af6-f83a-4d8a-ade4-92fec972bf58`::moving from state init -> state preparing >Thread-27081::INFO::2012-12-02 14:17:08,792::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27081::INFO::2012-12-02 14:17:08,792::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27081::DEBUG::2012-12-02 14:17:08,792::task::1151::TaskManager.Task::(prepare) Task=`9f791af6-f83a-4d8a-ade4-92fec972bf58`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27081::DEBUG::2012-12-02 14:17:08,793::task::568::TaskManager.Task::(_updateState) Task=`9f791af6-f83a-4d8a-ade4-92fec972bf58`::moving from state preparing -> state finished >Thread-27081::DEBUG::2012-12-02 14:17:08,793::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27081::DEBUG::2012-12-02 14:17:08,793::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27081::DEBUG::2012-12-02 14:17:08,793::task::957::TaskManager.Task::(_decref) Task=`9f791af6-f83a-4d8a-ade4-92fec972bf58`::ref 0 aborting False >Thread-27082::DEBUG::2012-12-02 14:17:08,795::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27082::DEBUG::2012-12-02 14:17:08,796::task::568::TaskManager.Task::(_updateState) Task=`ec474c7c-eab6-46ac-a92e-b91b901f08ac`::moving from state init -> state preparing >Thread-27082::INFO::2012-12-02 14:17:08,796::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27082::DEBUG::2012-12-02 14:17:08,796::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`99e2bbb7-1a44-404b-b29f-ac29a0119543`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27082::DEBUG::2012-12-02 14:17:08,796::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27082::DEBUG::2012-12-02 14:17:08,796::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27082::DEBUG::2012-12-02 14:17:08,797::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`99e2bbb7-1a44-404b-b29f-ac29a0119543`::Granted request >Thread-27082::DEBUG::2012-12-02 14:17:08,797::task::794::TaskManager.Task::(resourceAcquired) Task=`ec474c7c-eab6-46ac-a92e-b91b901f08ac`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27082::DEBUG::2012-12-02 14:17:08,797::task::957::TaskManager.Task::(_decref) Task=`ec474c7c-eab6-46ac-a92e-b91b901f08ac`::ref 1 aborting False >Thread-27082::INFO::2012-12-02 14:17:08,803::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27082::DEBUG::2012-12-02 14:17:08,804::task::1151::TaskManager.Task::(prepare) Task=`ec474c7c-eab6-46ac-a92e-b91b901f08ac`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27082::DEBUG::2012-12-02 14:17:08,804::task::568::TaskManager.Task::(_updateState) Task=`ec474c7c-eab6-46ac-a92e-b91b901f08ac`::moving from state preparing -> state finished >Thread-27082::DEBUG::2012-12-02 14:17:08,804::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27082::DEBUG::2012-12-02 14:17:08,804::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27082::DEBUG::2012-12-02 14:17:08,804::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27082::DEBUG::2012-12-02 14:17:08,804::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27082::DEBUG::2012-12-02 14:17:08,805::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27082::DEBUG::2012-12-02 14:17:08,805::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27082::DEBUG::2012-12-02 14:17:08,805::task::957::TaskManager.Task::(_decref) Task=`ec474c7c-eab6-46ac-a92e-b91b901f08ac`::ref 0 aborting False >Thread-27086::DEBUG::2012-12-02 14:17:16,446::task::568::TaskManager.Task::(_updateState) Task=`682d0361-173b-4de1-bfe1-cf6541c1df28`::moving from state init -> state preparing >Thread-27086::INFO::2012-12-02 14:17:16,446::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27086::INFO::2012-12-02 14:17:16,446::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0080189704895', 'lastCheck': 1354450636.173435, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00345396995544', 'lastCheck': 1354450631.4674129, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00244307518005', 'lastCheck': 1354450634.129514, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00727105140686', 'lastCheck': 1354450631.477016, 'code': 0, 'valid': True}} >Thread-27086::DEBUG::2012-12-02 14:17:16,446::task::1151::TaskManager.Task::(prepare) Task=`682d0361-173b-4de1-bfe1-cf6541c1df28`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0080189704895', 'lastCheck': 1354450636.173435, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00345396995544', 'lastCheck': 1354450631.4674129, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00244307518005', 'lastCheck': 1354450634.129514, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00727105140686', 'lastCheck': 1354450631.477016, 'code': 0, 'valid': True}} >Thread-27086::DEBUG::2012-12-02 14:17:16,446::task::568::TaskManager.Task::(_updateState) Task=`682d0361-173b-4de1-bfe1-cf6541c1df28`::moving from state preparing -> state finished >Thread-27086::DEBUG::2012-12-02 14:17:16,447::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27086::DEBUG::2012-12-02 14:17:16,447::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27086::DEBUG::2012-12-02 14:17:16,447::task::957::TaskManager.Task::(_decref) Task=`682d0361-173b-4de1-bfe1-cf6541c1df28`::ref 0 aborting False >Thread-27089::DEBUG::2012-12-02 14:17:18,963::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27089::DEBUG::2012-12-02 14:17:18,964::task::568::TaskManager.Task::(_updateState) Task=`1cfae35b-f871-4e13-8d50-96bc07915bb2`::moving from state init -> state preparing >Thread-27089::INFO::2012-12-02 14:17:18,964::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27089::INFO::2012-12-02 14:17:18,964::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27089::DEBUG::2012-12-02 14:17:18,964::task::1151::TaskManager.Task::(prepare) Task=`1cfae35b-f871-4e13-8d50-96bc07915bb2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27089::DEBUG::2012-12-02 14:17:18,964::task::568::TaskManager.Task::(_updateState) Task=`1cfae35b-f871-4e13-8d50-96bc07915bb2`::moving from state preparing -> state finished >Thread-27089::DEBUG::2012-12-02 14:17:18,964::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27089::DEBUG::2012-12-02 14:17:18,964::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27089::DEBUG::2012-12-02 14:17:18,965::task::957::TaskManager.Task::(_decref) Task=`1cfae35b-f871-4e13-8d50-96bc07915bb2`::ref 0 aborting False >Thread-27090::DEBUG::2012-12-02 14:17:18,967::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27090::DEBUG::2012-12-02 14:17:18,967::task::568::TaskManager.Task::(_updateState) Task=`3b8129f4-366e-4f41-b76a-0b3e925443f5`::moving from state init -> state preparing >Thread-27090::INFO::2012-12-02 14:17:18,968::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27090::DEBUG::2012-12-02 14:17:18,968::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`507043e0-b6aa-4331-bdff-1ce8a31c168e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27090::DEBUG::2012-12-02 14:17:18,968::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27090::DEBUG::2012-12-02 14:17:18,968::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27090::DEBUG::2012-12-02 14:17:18,968::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`507043e0-b6aa-4331-bdff-1ce8a31c168e`::Granted request >Thread-27090::DEBUG::2012-12-02 14:17:18,969::task::794::TaskManager.Task::(resourceAcquired) Task=`3b8129f4-366e-4f41-b76a-0b3e925443f5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27090::DEBUG::2012-12-02 14:17:18,969::task::957::TaskManager.Task::(_decref) Task=`3b8129f4-366e-4f41-b76a-0b3e925443f5`::ref 1 aborting False >Thread-27090::INFO::2012-12-02 14:17:18,975::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27090::DEBUG::2012-12-02 14:17:18,975::task::1151::TaskManager.Task::(prepare) Task=`3b8129f4-366e-4f41-b76a-0b3e925443f5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27090::DEBUG::2012-12-02 14:17:18,975::task::568::TaskManager.Task::(_updateState) Task=`3b8129f4-366e-4f41-b76a-0b3e925443f5`::moving from state preparing -> state finished >Thread-27090::DEBUG::2012-12-02 14:17:18,975::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27090::DEBUG::2012-12-02 14:17:18,976::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27090::DEBUG::2012-12-02 14:17:18,976::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27090::DEBUG::2012-12-02 14:17:18,976::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27090::DEBUG::2012-12-02 14:17:18,976::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27090::DEBUG::2012-12-02 14:17:18,976::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27090::DEBUG::2012-12-02 14:17:18,976::task::957::TaskManager.Task::(_decref) Task=`3b8129f4-366e-4f41-b76a-0b3e925443f5`::ref 0 aborting False >Thread-27094::DEBUG::2012-12-02 14:17:26,526::task::568::TaskManager.Task::(_updateState) Task=`5ec82815-8f5f-4c2c-a867-ab87d1236360`::moving from state init -> state preparing >Thread-27094::INFO::2012-12-02 14:17:26,526::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27094::INFO::2012-12-02 14:17:26,526::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00860095024109', 'lastCheck': 1354450646.1905489, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00187206268311', 'lastCheck': 1354450641.4858611, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00382399559021', 'lastCheck': 1354450644.142442, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00696802139282', 'lastCheck': 1354450641.4939251, 'code': 0, 'valid': True}} >Thread-27094::DEBUG::2012-12-02 14:17:26,526::task::1151::TaskManager.Task::(prepare) Task=`5ec82815-8f5f-4c2c-a867-ab87d1236360`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00860095024109', 'lastCheck': 1354450646.1905489, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00187206268311', 'lastCheck': 1354450641.4858611, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00382399559021', 'lastCheck': 1354450644.142442, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00696802139282', 'lastCheck': 1354450641.4939251, 'code': 0, 'valid': True}} >Thread-27094::DEBUG::2012-12-02 14:17:26,526::task::568::TaskManager.Task::(_updateState) Task=`5ec82815-8f5f-4c2c-a867-ab87d1236360`::moving from state preparing -> state finished >Thread-27094::DEBUG::2012-12-02 14:17:26,527::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27094::DEBUG::2012-12-02 14:17:26,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27094::DEBUG::2012-12-02 14:17:26,527::task::957::TaskManager.Task::(_decref) Task=`5ec82815-8f5f-4c2c-a867-ab87d1236360`::ref 0 aborting False >Thread-27097::DEBUG::2012-12-02 14:17:29,075::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27097::DEBUG::2012-12-02 14:17:29,075::task::568::TaskManager.Task::(_updateState) Task=`0793479a-c3e1-48f6-a129-89001b44fea2`::moving from state init -> state preparing >Thread-27097::INFO::2012-12-02 14:17:29,075::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27097::INFO::2012-12-02 14:17:29,076::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27097::DEBUG::2012-12-02 14:17:29,076::task::1151::TaskManager.Task::(prepare) Task=`0793479a-c3e1-48f6-a129-89001b44fea2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27097::DEBUG::2012-12-02 14:17:29,076::task::568::TaskManager.Task::(_updateState) Task=`0793479a-c3e1-48f6-a129-89001b44fea2`::moving from state preparing -> state finished >Thread-27097::DEBUG::2012-12-02 14:17:29,076::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27097::DEBUG::2012-12-02 14:17:29,076::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27097::DEBUG::2012-12-02 14:17:29,076::task::957::TaskManager.Task::(_decref) Task=`0793479a-c3e1-48f6-a129-89001b44fea2`::ref 0 aborting False >Thread-27098::DEBUG::2012-12-02 14:17:29,079::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27098::DEBUG::2012-12-02 14:17:29,079::task::568::TaskManager.Task::(_updateState) Task=`f6735bfe-6bd4-499e-9927-6b2e9ba08305`::moving from state init -> state preparing >Thread-27098::INFO::2012-12-02 14:17:29,079::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27098::DEBUG::2012-12-02 14:17:29,080::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0ef9ff6e-838d-40db-b7d7-32b58a3cc6d4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27098::DEBUG::2012-12-02 14:17:29,080::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27098::DEBUG::2012-12-02 14:17:29,080::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27098::DEBUG::2012-12-02 14:17:29,080::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0ef9ff6e-838d-40db-b7d7-32b58a3cc6d4`::Granted request >Thread-27098::DEBUG::2012-12-02 14:17:29,081::task::794::TaskManager.Task::(resourceAcquired) Task=`f6735bfe-6bd4-499e-9927-6b2e9ba08305`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27098::DEBUG::2012-12-02 14:17:29,081::task::957::TaskManager.Task::(_decref) Task=`f6735bfe-6bd4-499e-9927-6b2e9ba08305`::ref 1 aborting False >Thread-27098::INFO::2012-12-02 14:17:29,087::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27098::DEBUG::2012-12-02 14:17:29,087::task::1151::TaskManager.Task::(prepare) Task=`f6735bfe-6bd4-499e-9927-6b2e9ba08305`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27098::DEBUG::2012-12-02 14:17:29,087::task::568::TaskManager.Task::(_updateState) Task=`f6735bfe-6bd4-499e-9927-6b2e9ba08305`::moving from state preparing -> state finished >Thread-27098::DEBUG::2012-12-02 14:17:29,087::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27098::DEBUG::2012-12-02 14:17:29,087::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27098::DEBUG::2012-12-02 14:17:29,087::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27098::DEBUG::2012-12-02 14:17:29,087::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27098::DEBUG::2012-12-02 14:17:29,087::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27098::DEBUG::2012-12-02 14:17:29,087::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27098::DEBUG::2012-12-02 14:17:29,088::task::957::TaskManager.Task::(_decref) Task=`f6735bfe-6bd4-499e-9927-6b2e9ba08305`::ref 0 aborting False >Thread-27102::DEBUG::2012-12-02 14:17:36,585::task::568::TaskManager.Task::(_updateState) Task=`ec0eb2a7-c54f-47db-a378-5620c0539b73`::moving from state init -> state preparing >Thread-27102::INFO::2012-12-02 14:17:36,586::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27102::INFO::2012-12-02 14:17:36,586::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00789785385132', 'lastCheck': 1354450656.2104571, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00389695167542', 'lastCheck': 1354450651.501034, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00345516204834', 'lastCheck': 1354450654.157115, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00718212127686', 'lastCheck': 1354450651.51019, 'code': 0, 'valid': True}} >Thread-27102::DEBUG::2012-12-02 14:17:36,586::task::1151::TaskManager.Task::(prepare) Task=`ec0eb2a7-c54f-47db-a378-5620c0539b73`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00789785385132', 'lastCheck': 1354450656.2104571, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00389695167542', 'lastCheck': 1354450651.501034, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00345516204834', 'lastCheck': 1354450654.157115, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00718212127686', 'lastCheck': 1354450651.51019, 'code': 0, 'valid': True}} >Thread-27102::DEBUG::2012-12-02 14:17:36,586::task::568::TaskManager.Task::(_updateState) Task=`ec0eb2a7-c54f-47db-a378-5620c0539b73`::moving from state preparing -> state finished >Thread-27102::DEBUG::2012-12-02 14:17:36,586::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27102::DEBUG::2012-12-02 14:17:36,586::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27102::DEBUG::2012-12-02 14:17:36,587::task::957::TaskManager.Task::(_decref) Task=`ec0eb2a7-c54f-47db-a378-5620c0539b73`::ref 0 aborting False >Thread-27105::DEBUG::2012-12-02 14:17:39,180::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27105::DEBUG::2012-12-02 14:17:39,180::task::568::TaskManager.Task::(_updateState) Task=`f40c25a7-a778-4370-b720-e554de6cba83`::moving from state init -> state preparing >Thread-27105::INFO::2012-12-02 14:17:39,180::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27105::INFO::2012-12-02 14:17:39,180::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27105::DEBUG::2012-12-02 14:17:39,180::task::1151::TaskManager.Task::(prepare) Task=`f40c25a7-a778-4370-b720-e554de6cba83`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27105::DEBUG::2012-12-02 14:17:39,181::task::568::TaskManager.Task::(_updateState) Task=`f40c25a7-a778-4370-b720-e554de6cba83`::moving from state preparing -> state finished >Thread-27105::DEBUG::2012-12-02 14:17:39,181::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27105::DEBUG::2012-12-02 14:17:39,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27105::DEBUG::2012-12-02 14:17:39,181::task::957::TaskManager.Task::(_decref) Task=`f40c25a7-a778-4370-b720-e554de6cba83`::ref 0 aborting False >Thread-27106::DEBUG::2012-12-02 14:17:39,183::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27106::DEBUG::2012-12-02 14:17:39,184::task::568::TaskManager.Task::(_updateState) Task=`a2909b59-7b1c-4eb4-ae9e-494dcf088400`::moving from state init -> state preparing >Thread-27106::INFO::2012-12-02 14:17:39,184::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27106::DEBUG::2012-12-02 14:17:39,184::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cfb1af94-3a2d-4ffd-b5c2-9d9b4e480d75`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27106::DEBUG::2012-12-02 14:17:39,184::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27106::DEBUG::2012-12-02 14:17:39,184::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27106::DEBUG::2012-12-02 14:17:39,185::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cfb1af94-3a2d-4ffd-b5c2-9d9b4e480d75`::Granted request >Thread-27106::DEBUG::2012-12-02 14:17:39,185::task::794::TaskManager.Task::(resourceAcquired) Task=`a2909b59-7b1c-4eb4-ae9e-494dcf088400`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27106::DEBUG::2012-12-02 14:17:39,185::task::957::TaskManager.Task::(_decref) Task=`a2909b59-7b1c-4eb4-ae9e-494dcf088400`::ref 1 aborting False >Thread-27106::INFO::2012-12-02 14:17:39,190::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27106::DEBUG::2012-12-02 14:17:39,190::task::1151::TaskManager.Task::(prepare) Task=`a2909b59-7b1c-4eb4-ae9e-494dcf088400`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27106::DEBUG::2012-12-02 14:17:39,190::task::568::TaskManager.Task::(_updateState) Task=`a2909b59-7b1c-4eb4-ae9e-494dcf088400`::moving from state preparing -> state finished >Thread-27106::DEBUG::2012-12-02 14:17:39,190::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27106::DEBUG::2012-12-02 14:17:39,190::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27106::DEBUG::2012-12-02 14:17:39,190::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27106::DEBUG::2012-12-02 14:17:39,190::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27106::DEBUG::2012-12-02 14:17:39,190::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27106::DEBUG::2012-12-02 14:17:39,190::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27106::DEBUG::2012-12-02 14:17:39,191::task::957::TaskManager.Task::(_decref) Task=`a2909b59-7b1c-4eb4-ae9e-494dcf088400`::ref 0 aborting False >Thread-27110::DEBUG::2012-12-02 14:17:46,694::task::568::TaskManager.Task::(_updateState) Task=`4b810583-1508-4cf2-a078-3e4b01e3d223`::moving from state init -> state preparing >Thread-27110::INFO::2012-12-02 14:17:46,694::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27110::INFO::2012-12-02 14:17:46,694::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00793290138245', 'lastCheck': 1354450666.227556, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00284099578857', 'lastCheck': 1354450661.5177491, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00227117538452', 'lastCheck': 1354450664.1684201, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812816619873', 'lastCheck': 1354450661.529649, 'code': 0, 'valid': True}} >Thread-27110::DEBUG::2012-12-02 14:17:46,694::task::1151::TaskManager.Task::(prepare) Task=`4b810583-1508-4cf2-a078-3e4b01e3d223`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00793290138245', 'lastCheck': 1354450666.227556, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00284099578857', 'lastCheck': 1354450661.5177491, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00227117538452', 'lastCheck': 1354450664.1684201, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812816619873', 'lastCheck': 1354450661.529649, 'code': 0, 'valid': True}} >Thread-27110::DEBUG::2012-12-02 14:17:46,694::task::568::TaskManager.Task::(_updateState) Task=`4b810583-1508-4cf2-a078-3e4b01e3d223`::moving from state preparing -> state finished >Thread-27110::DEBUG::2012-12-02 14:17:46,694::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27110::DEBUG::2012-12-02 14:17:46,695::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27110::DEBUG::2012-12-02 14:17:46,695::task::957::TaskManager.Task::(_decref) Task=`4b810583-1508-4cf2-a078-3e4b01e3d223`::ref 0 aborting False >Thread-27113::DEBUG::2012-12-02 14:17:49,309::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27113::DEBUG::2012-12-02 14:17:49,310::task::568::TaskManager.Task::(_updateState) Task=`febd72d2-f781-48ff-bc9d-2cb89272e2b8`::moving from state init -> state preparing >Thread-27113::INFO::2012-12-02 14:17:49,310::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27113::INFO::2012-12-02 14:17:49,310::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27113::DEBUG::2012-12-02 14:17:49,310::task::1151::TaskManager.Task::(prepare) Task=`febd72d2-f781-48ff-bc9d-2cb89272e2b8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27113::DEBUG::2012-12-02 14:17:49,311::task::568::TaskManager.Task::(_updateState) Task=`febd72d2-f781-48ff-bc9d-2cb89272e2b8`::moving from state preparing -> state finished >Thread-27113::DEBUG::2012-12-02 14:17:49,311::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27113::DEBUG::2012-12-02 14:17:49,311::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27113::DEBUG::2012-12-02 14:17:49,311::task::957::TaskManager.Task::(_decref) Task=`febd72d2-f781-48ff-bc9d-2cb89272e2b8`::ref 0 aborting False >Thread-27114::DEBUG::2012-12-02 14:17:49,313::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27114::DEBUG::2012-12-02 14:17:49,314::task::568::TaskManager.Task::(_updateState) Task=`5056870b-4b71-4e3c-9467-f390c5bdf05f`::moving from state init -> state preparing >Thread-27114::INFO::2012-12-02 14:17:49,314::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27114::DEBUG::2012-12-02 14:17:49,314::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`90b64531-1b78-432c-b917-80bfa3e5d596`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27114::DEBUG::2012-12-02 14:17:49,314::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27114::DEBUG::2012-12-02 14:17:49,314::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27114::DEBUG::2012-12-02 14:17:49,315::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`90b64531-1b78-432c-b917-80bfa3e5d596`::Granted request >Thread-27114::DEBUG::2012-12-02 14:17:49,315::task::794::TaskManager.Task::(resourceAcquired) Task=`5056870b-4b71-4e3c-9467-f390c5bdf05f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27114::DEBUG::2012-12-02 14:17:49,315::task::957::TaskManager.Task::(_decref) Task=`5056870b-4b71-4e3c-9467-f390c5bdf05f`::ref 1 aborting False >Thread-27114::INFO::2012-12-02 14:17:49,320::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27114::DEBUG::2012-12-02 14:17:49,320::task::1151::TaskManager.Task::(prepare) Task=`5056870b-4b71-4e3c-9467-f390c5bdf05f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27114::DEBUG::2012-12-02 14:17:49,320::task::568::TaskManager.Task::(_updateState) Task=`5056870b-4b71-4e3c-9467-f390c5bdf05f`::moving from state preparing -> state finished >Thread-27114::DEBUG::2012-12-02 14:17:49,320::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27114::DEBUG::2012-12-02 14:17:49,320::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27114::DEBUG::2012-12-02 14:17:49,320::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27114::DEBUG::2012-12-02 14:17:49,320::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27114::DEBUG::2012-12-02 14:17:49,320::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27114::DEBUG::2012-12-02 14:17:49,320::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27114::DEBUG::2012-12-02 14:17:49,321::task::957::TaskManager.Task::(_decref) Task=`5056870b-4b71-4e3c-9467-f390c5bdf05f`::ref 0 aborting False >Thread-27118::DEBUG::2012-12-02 14:17:56,938::task::568::TaskManager.Task::(_updateState) Task=`32d3b298-186e-4bf9-808a-f657396f282e`::moving from state init -> state preparing >Thread-27118::INFO::2012-12-02 14:17:56,938::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27118::INFO::2012-12-02 14:17:56,938::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00702595710754', 'lastCheck': 1354450676.244175, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0018298625946', 'lastCheck': 1354450671.5294271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00429391860962', 'lastCheck': 1354450674.182385, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00709509849548', 'lastCheck': 1354450671.545413, 'code': 0, 'valid': True}} >Thread-27118::DEBUG::2012-12-02 14:17:56,939::task::1151::TaskManager.Task::(prepare) Task=`32d3b298-186e-4bf9-808a-f657396f282e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00702595710754', 'lastCheck': 1354450676.244175, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0018298625946', 'lastCheck': 1354450671.5294271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00429391860962', 'lastCheck': 1354450674.182385, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00709509849548', 'lastCheck': 1354450671.545413, 'code': 0, 'valid': True}} >Thread-27118::DEBUG::2012-12-02 14:17:56,939::task::568::TaskManager.Task::(_updateState) Task=`32d3b298-186e-4bf9-808a-f657396f282e`::moving from state preparing -> state finished >Thread-27118::DEBUG::2012-12-02 14:17:56,939::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27118::DEBUG::2012-12-02 14:17:56,939::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27118::DEBUG::2012-12-02 14:17:56,939::task::957::TaskManager.Task::(_decref) Task=`32d3b298-186e-4bf9-808a-f657396f282e`::ref 0 aborting False >Thread-27121::DEBUG::2012-12-02 14:17:59,569::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27121::DEBUG::2012-12-02 14:17:59,570::task::568::TaskManager.Task::(_updateState) Task=`7f45341b-e726-4ddc-afd6-38352cd1f9a1`::moving from state init -> state preparing >Thread-27121::INFO::2012-12-02 14:17:59,570::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27121::INFO::2012-12-02 14:17:59,570::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27121::DEBUG::2012-12-02 14:17:59,570::task::1151::TaskManager.Task::(prepare) Task=`7f45341b-e726-4ddc-afd6-38352cd1f9a1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27121::DEBUG::2012-12-02 14:17:59,570::task::568::TaskManager.Task::(_updateState) Task=`7f45341b-e726-4ddc-afd6-38352cd1f9a1`::moving from state preparing -> state finished >Thread-27121::DEBUG::2012-12-02 14:17:59,571::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27121::DEBUG::2012-12-02 14:17:59,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27121::DEBUG::2012-12-02 14:17:59,571::task::957::TaskManager.Task::(_decref) Task=`7f45341b-e726-4ddc-afd6-38352cd1f9a1`::ref 0 aborting False >Thread-27122::DEBUG::2012-12-02 14:17:59,573::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27122::DEBUG::2012-12-02 14:17:59,574::task::568::TaskManager.Task::(_updateState) Task=`73cbc381-949d-4681-80b8-dccd9cb4297f`::moving from state init -> state preparing >Thread-27122::INFO::2012-12-02 14:17:59,574::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27122::DEBUG::2012-12-02 14:17:59,574::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c3a74ea5-2ad1-4f5e-9f07-f6dddcfdf44b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27122::DEBUG::2012-12-02 14:17:59,574::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27122::DEBUG::2012-12-02 14:17:59,574::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27122::DEBUG::2012-12-02 14:17:59,574::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c3a74ea5-2ad1-4f5e-9f07-f6dddcfdf44b`::Granted request >Thread-27122::DEBUG::2012-12-02 14:17:59,575::task::794::TaskManager.Task::(resourceAcquired) Task=`73cbc381-949d-4681-80b8-dccd9cb4297f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27122::DEBUG::2012-12-02 14:17:59,575::task::957::TaskManager.Task::(_decref) Task=`73cbc381-949d-4681-80b8-dccd9cb4297f`::ref 1 aborting False >Thread-27122::INFO::2012-12-02 14:17:59,580::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27122::DEBUG::2012-12-02 14:17:59,580::task::1151::TaskManager.Task::(prepare) Task=`73cbc381-949d-4681-80b8-dccd9cb4297f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27122::DEBUG::2012-12-02 14:17:59,580::task::568::TaskManager.Task::(_updateState) Task=`73cbc381-949d-4681-80b8-dccd9cb4297f`::moving from state preparing -> state finished >Thread-27122::DEBUG::2012-12-02 14:17:59,580::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27122::DEBUG::2012-12-02 14:17:59,580::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27122::DEBUG::2012-12-02 14:17:59,580::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27122::DEBUG::2012-12-02 14:17:59,580::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27122::DEBUG::2012-12-02 14:17:59,580::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27122::DEBUG::2012-12-02 14:17:59,580::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27122::DEBUG::2012-12-02 14:17:59,580::task::957::TaskManager.Task::(_decref) Task=`73cbc381-949d-4681-80b8-dccd9cb4297f`::ref 0 aborting False >Thread-27126::DEBUG::2012-12-02 14:18:07,048::task::568::TaskManager.Task::(_updateState) Task=`146cb670-6bcb-4fc7-b641-fb3c64bff36e`::moving from state init -> state preparing >Thread-27126::INFO::2012-12-02 14:18:07,049::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27126::INFO::2012-12-02 14:18:07,049::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00915288925171', 'lastCheck': 1354450686.2625201, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00216913223267', 'lastCheck': 1354450681.5407591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0036289691925', 'lastCheck': 1354450684.195297, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00749683380127', 'lastCheck': 1354450681.561785, 'code': 0, 'valid': True}} >Thread-27126::DEBUG::2012-12-02 14:18:07,049::task::1151::TaskManager.Task::(prepare) Task=`146cb670-6bcb-4fc7-b641-fb3c64bff36e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00915288925171', 'lastCheck': 1354450686.2625201, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00216913223267', 'lastCheck': 1354450681.5407591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0036289691925', 'lastCheck': 1354450684.195297, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00749683380127', 'lastCheck': 1354450681.561785, 'code': 0, 'valid': True}} >Thread-27126::DEBUG::2012-12-02 14:18:07,049::task::568::TaskManager.Task::(_updateState) Task=`146cb670-6bcb-4fc7-b641-fb3c64bff36e`::moving from state preparing -> state finished >Thread-27126::DEBUG::2012-12-02 14:18:07,049::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27126::DEBUG::2012-12-02 14:18:07,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27126::DEBUG::2012-12-02 14:18:07,049::task::957::TaskManager.Task::(_decref) Task=`146cb670-6bcb-4fc7-b641-fb3c64bff36e`::ref 0 aborting False >Thread-27129::DEBUG::2012-12-02 14:18:09,708::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27129::DEBUG::2012-12-02 14:18:09,708::task::568::TaskManager.Task::(_updateState) Task=`570acf26-89df-490c-97a7-c60ccfe94215`::moving from state init -> state preparing >Thread-27129::INFO::2012-12-02 14:18:09,708::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27129::INFO::2012-12-02 14:18:09,708::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27129::DEBUG::2012-12-02 14:18:09,708::task::1151::TaskManager.Task::(prepare) Task=`570acf26-89df-490c-97a7-c60ccfe94215`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27129::DEBUG::2012-12-02 14:18:09,709::task::568::TaskManager.Task::(_updateState) Task=`570acf26-89df-490c-97a7-c60ccfe94215`::moving from state preparing -> state finished >Thread-27129::DEBUG::2012-12-02 14:18:09,709::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27129::DEBUG::2012-12-02 14:18:09,709::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27129::DEBUG::2012-12-02 14:18:09,709::task::957::TaskManager.Task::(_decref) Task=`570acf26-89df-490c-97a7-c60ccfe94215`::ref 0 aborting False >Thread-27130::DEBUG::2012-12-02 14:18:09,712::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27130::DEBUG::2012-12-02 14:18:09,712::task::568::TaskManager.Task::(_updateState) Task=`bb005b29-0c6b-436c-8c6a-60a4a95498b1`::moving from state init -> state preparing >Thread-27130::INFO::2012-12-02 14:18:09,712::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27130::DEBUG::2012-12-02 14:18:09,712::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`09cd4036-bf6d-4cdd-9010-1d0adfa8efe3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27130::DEBUG::2012-12-02 14:18:09,712::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27130::DEBUG::2012-12-02 14:18:09,713::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27130::DEBUG::2012-12-02 14:18:09,713::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`09cd4036-bf6d-4cdd-9010-1d0adfa8efe3`::Granted request >Thread-27130::DEBUG::2012-12-02 14:18:09,713::task::794::TaskManager.Task::(resourceAcquired) Task=`bb005b29-0c6b-436c-8c6a-60a4a95498b1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27130::DEBUG::2012-12-02 14:18:09,713::task::957::TaskManager.Task::(_decref) Task=`bb005b29-0c6b-436c-8c6a-60a4a95498b1`::ref 1 aborting False >Thread-27130::INFO::2012-12-02 14:18:09,718::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27130::DEBUG::2012-12-02 14:18:09,718::task::1151::TaskManager.Task::(prepare) Task=`bb005b29-0c6b-436c-8c6a-60a4a95498b1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27130::DEBUG::2012-12-02 14:18:09,718::task::568::TaskManager.Task::(_updateState) Task=`bb005b29-0c6b-436c-8c6a-60a4a95498b1`::moving from state preparing -> state finished >Thread-27130::DEBUG::2012-12-02 14:18:09,718::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27130::DEBUG::2012-12-02 14:18:09,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27130::DEBUG::2012-12-02 14:18:09,719::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27130::DEBUG::2012-12-02 14:18:09,719::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27130::DEBUG::2012-12-02 14:18:09,719::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27130::DEBUG::2012-12-02 14:18:09,719::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27130::DEBUG::2012-12-02 14:18:09,719::task::957::TaskManager.Task::(_decref) Task=`bb005b29-0c6b-436c-8c6a-60a4a95498b1`::ref 0 aborting False >Thread-27134::DEBUG::2012-12-02 14:18:17,169::task::568::TaskManager.Task::(_updateState) Task=`87d01566-84c3-419c-8a04-4c3d47ceb705`::moving from state init -> state preparing >Thread-27134::INFO::2012-12-02 14:18:17,169::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27134::INFO::2012-12-02 14:18:17,169::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00745296478271', 'lastCheck': 1354450696.2784431, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00309085845947', 'lastCheck': 1354450691.5539751, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00325298309326', 'lastCheck': 1354450694.2113979, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00723099708557', 'lastCheck': 1354450691.577759, 'code': 0, 'valid': True}} >Thread-27134::DEBUG::2012-12-02 14:18:17,169::task::1151::TaskManager.Task::(prepare) Task=`87d01566-84c3-419c-8a04-4c3d47ceb705`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00745296478271', 'lastCheck': 1354450696.2784431, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00309085845947', 'lastCheck': 1354450691.5539751, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00325298309326', 'lastCheck': 1354450694.2113979, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00723099708557', 'lastCheck': 1354450691.577759, 'code': 0, 'valid': True}} >Thread-27134::DEBUG::2012-12-02 14:18:17,169::task::568::TaskManager.Task::(_updateState) Task=`87d01566-84c3-419c-8a04-4c3d47ceb705`::moving from state preparing -> state finished >Thread-27134::DEBUG::2012-12-02 14:18:17,170::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27134::DEBUG::2012-12-02 14:18:17,170::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27134::DEBUG::2012-12-02 14:18:17,170::task::957::TaskManager.Task::(_decref) Task=`87d01566-84c3-419c-8a04-4c3d47ceb705`::ref 0 aborting False >Thread-27137::DEBUG::2012-12-02 14:18:19,855::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27137::DEBUG::2012-12-02 14:18:19,855::task::568::TaskManager.Task::(_updateState) Task=`36b3facb-2b3d-4c23-9652-912ae875fc18`::moving from state init -> state preparing >Thread-27137::INFO::2012-12-02 14:18:19,855::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27137::INFO::2012-12-02 14:18:19,855::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27137::DEBUG::2012-12-02 14:18:19,855::task::1151::TaskManager.Task::(prepare) Task=`36b3facb-2b3d-4c23-9652-912ae875fc18`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27137::DEBUG::2012-12-02 14:18:19,856::task::568::TaskManager.Task::(_updateState) Task=`36b3facb-2b3d-4c23-9652-912ae875fc18`::moving from state preparing -> state finished >Thread-27137::DEBUG::2012-12-02 14:18:19,856::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27137::DEBUG::2012-12-02 14:18:19,856::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27137::DEBUG::2012-12-02 14:18:19,856::task::957::TaskManager.Task::(_decref) Task=`36b3facb-2b3d-4c23-9652-912ae875fc18`::ref 0 aborting False >Thread-27138::DEBUG::2012-12-02 14:18:19,859::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27138::DEBUG::2012-12-02 14:18:19,859::task::568::TaskManager.Task::(_updateState) Task=`18cafd64-7dba-4149-8c80-fbbf2e4f96db`::moving from state init -> state preparing >Thread-27138::INFO::2012-12-02 14:18:19,859::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27138::DEBUG::2012-12-02 14:18:19,859::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`af884782-c8a9-4541-9241-1d71160034df`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27138::DEBUG::2012-12-02 14:18:19,859::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27138::DEBUG::2012-12-02 14:18:19,860::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27138::DEBUG::2012-12-02 14:18:19,860::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`af884782-c8a9-4541-9241-1d71160034df`::Granted request >Thread-27138::DEBUG::2012-12-02 14:18:19,860::task::794::TaskManager.Task::(resourceAcquired) Task=`18cafd64-7dba-4149-8c80-fbbf2e4f96db`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27138::DEBUG::2012-12-02 14:18:19,860::task::957::TaskManager.Task::(_decref) Task=`18cafd64-7dba-4149-8c80-fbbf2e4f96db`::ref 1 aborting False >Thread-27138::INFO::2012-12-02 14:18:19,866::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27138::DEBUG::2012-12-02 14:18:19,866::task::1151::TaskManager.Task::(prepare) Task=`18cafd64-7dba-4149-8c80-fbbf2e4f96db`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27138::DEBUG::2012-12-02 14:18:19,866::task::568::TaskManager.Task::(_updateState) Task=`18cafd64-7dba-4149-8c80-fbbf2e4f96db`::moving from state preparing -> state finished >Thread-27138::DEBUG::2012-12-02 14:18:19,866::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27138::DEBUG::2012-12-02 14:18:19,867::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27138::DEBUG::2012-12-02 14:18:19,867::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27138::DEBUG::2012-12-02 14:18:19,867::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27138::DEBUG::2012-12-02 14:18:19,867::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27138::DEBUG::2012-12-02 14:18:19,867::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27138::DEBUG::2012-12-02 14:18:19,867::task::957::TaskManager.Task::(_decref) Task=`18cafd64-7dba-4149-8c80-fbbf2e4f96db`::ref 0 aborting False >Thread-27142::DEBUG::2012-12-02 14:18:27,310::task::568::TaskManager.Task::(_updateState) Task=`67f3333a-8a32-4661-96de-c6959737480c`::moving from state init -> state preparing >Thread-27142::INFO::2012-12-02 14:18:27,310::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27142::INFO::2012-12-02 14:18:27,310::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00755310058594', 'lastCheck': 1354450706.2954559, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00179100036621', 'lastCheck': 1354450701.5661311, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00194096565247', 'lastCheck': 1354450704.222326, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00711297988892', 'lastCheck': 1354450701.5938089, 'code': 0, 'valid': True}} >Thread-27142::DEBUG::2012-12-02 14:18:27,310::task::1151::TaskManager.Task::(prepare) Task=`67f3333a-8a32-4661-96de-c6959737480c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00755310058594', 'lastCheck': 1354450706.2954559, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00179100036621', 'lastCheck': 1354450701.5661311, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00194096565247', 'lastCheck': 1354450704.222326, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00711297988892', 'lastCheck': 1354450701.5938089, 'code': 0, 'valid': True}} >Thread-27142::DEBUG::2012-12-02 14:18:27,311::task::568::TaskManager.Task::(_updateState) Task=`67f3333a-8a32-4661-96de-c6959737480c`::moving from state preparing -> state finished >Thread-27142::DEBUG::2012-12-02 14:18:27,311::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27142::DEBUG::2012-12-02 14:18:27,311::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27142::DEBUG::2012-12-02 14:18:27,311::task::957::TaskManager.Task::(_decref) Task=`67f3333a-8a32-4661-96de-c6959737480c`::ref 0 aborting False >Thread-27145::DEBUG::2012-12-02 14:18:29,991::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27145::DEBUG::2012-12-02 14:18:29,991::task::568::TaskManager.Task::(_updateState) Task=`396998e9-b508-478b-9b4c-75ddb9450d61`::moving from state init -> state preparing >Thread-27145::INFO::2012-12-02 14:18:29,991::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27145::INFO::2012-12-02 14:18:29,991::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27145::DEBUG::2012-12-02 14:18:29,992::task::1151::TaskManager.Task::(prepare) Task=`396998e9-b508-478b-9b4c-75ddb9450d61`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27145::DEBUG::2012-12-02 14:18:29,992::task::568::TaskManager.Task::(_updateState) Task=`396998e9-b508-478b-9b4c-75ddb9450d61`::moving from state preparing -> state finished >Thread-27145::DEBUG::2012-12-02 14:18:29,992::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27145::DEBUG::2012-12-02 14:18:29,992::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27145::DEBUG::2012-12-02 14:18:29,992::task::957::TaskManager.Task::(_decref) Task=`396998e9-b508-478b-9b4c-75ddb9450d61`::ref 0 aborting False >Thread-27146::DEBUG::2012-12-02 14:18:29,995::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27146::DEBUG::2012-12-02 14:18:29,995::task::568::TaskManager.Task::(_updateState) Task=`a2bedff6-c77b-41e0-a89d-f0d23eb80e41`::moving from state init -> state preparing >Thread-27146::INFO::2012-12-02 14:18:29,995::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27146::DEBUG::2012-12-02 14:18:29,996::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`dc57aa13-9efb-40bc-a78b-c26bc7173e59`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27146::DEBUG::2012-12-02 14:18:29,996::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27146::DEBUG::2012-12-02 14:18:29,996::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27146::DEBUG::2012-12-02 14:18:29,996::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`dc57aa13-9efb-40bc-a78b-c26bc7173e59`::Granted request >Thread-27146::DEBUG::2012-12-02 14:18:29,996::task::794::TaskManager.Task::(resourceAcquired) Task=`a2bedff6-c77b-41e0-a89d-f0d23eb80e41`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27146::DEBUG::2012-12-02 14:18:29,996::task::957::TaskManager.Task::(_decref) Task=`a2bedff6-c77b-41e0-a89d-f0d23eb80e41`::ref 1 aborting False >Thread-27146::INFO::2012-12-02 14:18:30,003::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27146::DEBUG::2012-12-02 14:18:30,003::task::1151::TaskManager.Task::(prepare) Task=`a2bedff6-c77b-41e0-a89d-f0d23eb80e41`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27146::DEBUG::2012-12-02 14:18:30,003::task::568::TaskManager.Task::(_updateState) Task=`a2bedff6-c77b-41e0-a89d-f0d23eb80e41`::moving from state preparing -> state finished >Thread-27146::DEBUG::2012-12-02 14:18:30,003::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27146::DEBUG::2012-12-02 14:18:30,003::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27146::DEBUG::2012-12-02 14:18:30,004::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27146::DEBUG::2012-12-02 14:18:30,004::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27146::DEBUG::2012-12-02 14:18:30,004::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27146::DEBUG::2012-12-02 14:18:30,004::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27146::DEBUG::2012-12-02 14:18:30,004::task::957::TaskManager.Task::(_decref) Task=`a2bedff6-c77b-41e0-a89d-f0d23eb80e41`::ref 0 aborting False >Thread-27150::DEBUG::2012-12-02 14:18:37,421::task::568::TaskManager.Task::(_updateState) Task=`f60c445c-80bd-403b-bb07-bfb27bfc3ca3`::moving from state init -> state preparing >Thread-27150::INFO::2012-12-02 14:18:37,421::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27150::INFO::2012-12-02 14:18:37,421::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00398707389832', 'lastCheck': 1354450716.3089991, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00334310531616', 'lastCheck': 1354450711.57932, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00480818748474', 'lastCheck': 1354450714.2355771, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00716114044189', 'lastCheck': 1354450711.6098599, 'code': 0, 'valid': True}} >Thread-27150::DEBUG::2012-12-02 14:18:37,421::task::1151::TaskManager.Task::(prepare) Task=`f60c445c-80bd-403b-bb07-bfb27bfc3ca3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00398707389832', 'lastCheck': 1354450716.3089991, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00334310531616', 'lastCheck': 1354450711.57932, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00480818748474', 'lastCheck': 1354450714.2355771, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00716114044189', 'lastCheck': 1354450711.6098599, 'code': 0, 'valid': True}} >Thread-27150::DEBUG::2012-12-02 14:18:37,422::task::568::TaskManager.Task::(_updateState) Task=`f60c445c-80bd-403b-bb07-bfb27bfc3ca3`::moving from state preparing -> state finished >Thread-27150::DEBUG::2012-12-02 14:18:37,422::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27150::DEBUG::2012-12-02 14:18:37,422::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27150::DEBUG::2012-12-02 14:18:37,422::task::957::TaskManager.Task::(_decref) Task=`f60c445c-80bd-403b-bb07-bfb27bfc3ca3`::ref 0 aborting False >Thread-27153::DEBUG::2012-12-02 14:18:40,103::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27153::DEBUG::2012-12-02 14:18:40,104::task::568::TaskManager.Task::(_updateState) Task=`41c4ac62-808a-416d-82c6-7cc76008464e`::moving from state init -> state preparing >Thread-27153::INFO::2012-12-02 14:18:40,104::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27153::INFO::2012-12-02 14:18:40,104::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27153::DEBUG::2012-12-02 14:18:40,104::task::1151::TaskManager.Task::(prepare) Task=`41c4ac62-808a-416d-82c6-7cc76008464e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27153::DEBUG::2012-12-02 14:18:40,104::task::568::TaskManager.Task::(_updateState) Task=`41c4ac62-808a-416d-82c6-7cc76008464e`::moving from state preparing -> state finished >Thread-27153::DEBUG::2012-12-02 14:18:40,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27153::DEBUG::2012-12-02 14:18:40,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27153::DEBUG::2012-12-02 14:18:40,105::task::957::TaskManager.Task::(_decref) Task=`41c4ac62-808a-416d-82c6-7cc76008464e`::ref 0 aborting False >Thread-27154::DEBUG::2012-12-02 14:18:40,107::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27154::DEBUG::2012-12-02 14:18:40,108::task::568::TaskManager.Task::(_updateState) Task=`288a8884-5fb2-498f-931b-3e39aae5aa62`::moving from state init -> state preparing >Thread-27154::INFO::2012-12-02 14:18:40,108::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27154::DEBUG::2012-12-02 14:18:40,108::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f39dda33-9070-404c-99aa-cd7a8b3cb247`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27154::DEBUG::2012-12-02 14:18:40,108::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27154::DEBUG::2012-12-02 14:18:40,109::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27154::DEBUG::2012-12-02 14:18:40,109::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f39dda33-9070-404c-99aa-cd7a8b3cb247`::Granted request >Thread-27154::DEBUG::2012-12-02 14:18:40,109::task::794::TaskManager.Task::(resourceAcquired) Task=`288a8884-5fb2-498f-931b-3e39aae5aa62`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27154::DEBUG::2012-12-02 14:18:40,109::task::957::TaskManager.Task::(_decref) Task=`288a8884-5fb2-498f-931b-3e39aae5aa62`::ref 1 aborting False >Thread-27154::INFO::2012-12-02 14:18:40,115::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27154::DEBUG::2012-12-02 14:18:40,116::task::1151::TaskManager.Task::(prepare) Task=`288a8884-5fb2-498f-931b-3e39aae5aa62`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27154::DEBUG::2012-12-02 14:18:40,116::task::568::TaskManager.Task::(_updateState) Task=`288a8884-5fb2-498f-931b-3e39aae5aa62`::moving from state preparing -> state finished >Thread-27154::DEBUG::2012-12-02 14:18:40,116::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27154::DEBUG::2012-12-02 14:18:40,116::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27154::DEBUG::2012-12-02 14:18:40,116::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27154::DEBUG::2012-12-02 14:18:40,116::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27154::DEBUG::2012-12-02 14:18:40,116::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27154::DEBUG::2012-12-02 14:18:40,116::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27154::DEBUG::2012-12-02 14:18:40,116::task::957::TaskManager.Task::(_decref) Task=`288a8884-5fb2-498f-931b-3e39aae5aa62`::ref 0 aborting False >Thread-27158::DEBUG::2012-12-02 14:18:47,515::task::568::TaskManager.Task::(_updateState) Task=`47a23de8-e137-4dff-946d-68be71c7f6eb`::moving from state init -> state preparing >Thread-27158::INFO::2012-12-02 14:18:47,515::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27158::INFO::2012-12-02 14:18:47,515::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00912094116211', 'lastCheck': 1354450726.327563, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00211191177368', 'lastCheck': 1354450721.5929501, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00241899490356', 'lastCheck': 1354450724.2492509, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0074770450592', 'lastCheck': 1354450721.6280279, 'code': 0, 'valid': True}} >Thread-27158::DEBUG::2012-12-02 14:18:47,515::task::1151::TaskManager.Task::(prepare) Task=`47a23de8-e137-4dff-946d-68be71c7f6eb`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00912094116211', 'lastCheck': 1354450726.327563, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00211191177368', 'lastCheck': 1354450721.5929501, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00241899490356', 'lastCheck': 1354450724.2492509, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0074770450592', 'lastCheck': 1354450721.6280279, 'code': 0, 'valid': True}} >Thread-27158::DEBUG::2012-12-02 14:18:47,516::task::568::TaskManager.Task::(_updateState) Task=`47a23de8-e137-4dff-946d-68be71c7f6eb`::moving from state preparing -> state finished >Thread-27158::DEBUG::2012-12-02 14:18:47,516::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27158::DEBUG::2012-12-02 14:18:47,516::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27158::DEBUG::2012-12-02 14:18:47,516::task::957::TaskManager.Task::(_decref) Task=`47a23de8-e137-4dff-946d-68be71c7f6eb`::ref 0 aborting False >Thread-27161::DEBUG::2012-12-02 14:18:50,258::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27161::DEBUG::2012-12-02 14:18:50,259::task::568::TaskManager.Task::(_updateState) Task=`f4b70a3f-f643-4793-b7da-e68c2f25facc`::moving from state init -> state preparing >Thread-27161::INFO::2012-12-02 14:18:50,259::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27161::INFO::2012-12-02 14:18:50,259::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27161::DEBUG::2012-12-02 14:18:50,259::task::1151::TaskManager.Task::(prepare) Task=`f4b70a3f-f643-4793-b7da-e68c2f25facc`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27161::DEBUG::2012-12-02 14:18:50,259::task::568::TaskManager.Task::(_updateState) Task=`f4b70a3f-f643-4793-b7da-e68c2f25facc`::moving from state preparing -> state finished >Thread-27161::DEBUG::2012-12-02 14:18:50,259::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27161::DEBUG::2012-12-02 14:18:50,260::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27161::DEBUG::2012-12-02 14:18:50,260::task::957::TaskManager.Task::(_decref) Task=`f4b70a3f-f643-4793-b7da-e68c2f25facc`::ref 0 aborting False >Thread-27162::DEBUG::2012-12-02 14:18:50,262::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27162::DEBUG::2012-12-02 14:18:50,263::task::568::TaskManager.Task::(_updateState) Task=`6b658fee-9a77-46ca-b0f1-6af58f14a5c0`::moving from state init -> state preparing >Thread-27162::INFO::2012-12-02 14:18:50,263::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27162::DEBUG::2012-12-02 14:18:50,263::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3a66d9f7-674e-4d31-b42e-69d9a1ac9bc4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27162::DEBUG::2012-12-02 14:18:50,263::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27162::DEBUG::2012-12-02 14:18:50,263::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27162::DEBUG::2012-12-02 14:18:50,264::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3a66d9f7-674e-4d31-b42e-69d9a1ac9bc4`::Granted request >Thread-27162::DEBUG::2012-12-02 14:18:50,264::task::794::TaskManager.Task::(resourceAcquired) Task=`6b658fee-9a77-46ca-b0f1-6af58f14a5c0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27162::DEBUG::2012-12-02 14:18:50,264::task::957::TaskManager.Task::(_decref) Task=`6b658fee-9a77-46ca-b0f1-6af58f14a5c0`::ref 1 aborting False >Thread-27162::INFO::2012-12-02 14:18:50,269::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27162::DEBUG::2012-12-02 14:18:50,269::task::1151::TaskManager.Task::(prepare) Task=`6b658fee-9a77-46ca-b0f1-6af58f14a5c0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27162::DEBUG::2012-12-02 14:18:50,269::task::568::TaskManager.Task::(_updateState) Task=`6b658fee-9a77-46ca-b0f1-6af58f14a5c0`::moving from state preparing -> state finished >Thread-27162::DEBUG::2012-12-02 14:18:50,269::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27162::DEBUG::2012-12-02 14:18:50,269::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27162::DEBUG::2012-12-02 14:18:50,269::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27162::DEBUG::2012-12-02 14:18:50,269::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27162::DEBUG::2012-12-02 14:18:50,269::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27162::DEBUG::2012-12-02 14:18:50,269::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27162::DEBUG::2012-12-02 14:18:50,270::task::957::TaskManager.Task::(_decref) Task=`6b658fee-9a77-46ca-b0f1-6af58f14a5c0`::ref 0 aborting False >Thread-27166::DEBUG::2012-12-02 14:18:57,629::task::568::TaskManager.Task::(_updateState) Task=`f75b75bb-c90c-4865-ad8b-49c144c6aee0`::moving from state init -> state preparing >Thread-27166::INFO::2012-12-02 14:18:57,629::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27166::INFO::2012-12-02 14:18:57,629::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00750494003296', 'lastCheck': 1354450736.343951, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180697441101', 'lastCheck': 1354450731.6032419, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00186705589294', 'lastCheck': 1354450734.2606449, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00751209259033', 'lastCheck': 1354450731.6441209, 'code': 0, 'valid': True}} >Thread-27166::DEBUG::2012-12-02 14:18:57,630::task::1151::TaskManager.Task::(prepare) Task=`f75b75bb-c90c-4865-ad8b-49c144c6aee0`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00750494003296', 'lastCheck': 1354450736.343951, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180697441101', 'lastCheck': 1354450731.6032419, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00186705589294', 'lastCheck': 1354450734.2606449, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00751209259033', 'lastCheck': 1354450731.6441209, 'code': 0, 'valid': True}} >Thread-27166::DEBUG::2012-12-02 14:18:57,630::task::568::TaskManager.Task::(_updateState) Task=`f75b75bb-c90c-4865-ad8b-49c144c6aee0`::moving from state preparing -> state finished >Thread-27166::DEBUG::2012-12-02 14:18:57,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27166::DEBUG::2012-12-02 14:18:57,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27166::DEBUG::2012-12-02 14:18:57,630::task::957::TaskManager.Task::(_decref) Task=`f75b75bb-c90c-4865-ad8b-49c144c6aee0`::ref 0 aborting False >Thread-27169::DEBUG::2012-12-02 14:19:00,388::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27169::DEBUG::2012-12-02 14:19:00,388::task::568::TaskManager.Task::(_updateState) Task=`d2180aae-969d-4b9e-9e4b-3c025baa781e`::moving from state init -> state preparing >Thread-27169::INFO::2012-12-02 14:19:00,389::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27169::INFO::2012-12-02 14:19:00,389::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27169::DEBUG::2012-12-02 14:19:00,389::task::1151::TaskManager.Task::(prepare) Task=`d2180aae-969d-4b9e-9e4b-3c025baa781e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27169::DEBUG::2012-12-02 14:19:00,389::task::568::TaskManager.Task::(_updateState) Task=`d2180aae-969d-4b9e-9e4b-3c025baa781e`::moving from state preparing -> state finished >Thread-27169::DEBUG::2012-12-02 14:19:00,389::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27169::DEBUG::2012-12-02 14:19:00,389::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27169::DEBUG::2012-12-02 14:19:00,390::task::957::TaskManager.Task::(_decref) Task=`d2180aae-969d-4b9e-9e4b-3c025baa781e`::ref 0 aborting False >Thread-27170::DEBUG::2012-12-02 14:19:00,392::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27170::DEBUG::2012-12-02 14:19:00,392::task::568::TaskManager.Task::(_updateState) Task=`95c5a98b-f6f1-4c69-b2f3-22ac5669d249`::moving from state init -> state preparing >Thread-27170::INFO::2012-12-02 14:19:00,393::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27170::DEBUG::2012-12-02 14:19:00,393::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c481f4b8-238d-4b7f-bd4d-d2a19a4cf60a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27170::DEBUG::2012-12-02 14:19:00,393::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27170::DEBUG::2012-12-02 14:19:00,393::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27170::DEBUG::2012-12-02 14:19:00,393::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c481f4b8-238d-4b7f-bd4d-d2a19a4cf60a`::Granted request >Thread-27170::DEBUG::2012-12-02 14:19:00,394::task::794::TaskManager.Task::(resourceAcquired) Task=`95c5a98b-f6f1-4c69-b2f3-22ac5669d249`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27170::DEBUG::2012-12-02 14:19:00,394::task::957::TaskManager.Task::(_decref) Task=`95c5a98b-f6f1-4c69-b2f3-22ac5669d249`::ref 1 aborting False >Thread-27170::INFO::2012-12-02 14:19:00,398::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27170::DEBUG::2012-12-02 14:19:00,399::task::1151::TaskManager.Task::(prepare) Task=`95c5a98b-f6f1-4c69-b2f3-22ac5669d249`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27170::DEBUG::2012-12-02 14:19:00,399::task::568::TaskManager.Task::(_updateState) Task=`95c5a98b-f6f1-4c69-b2f3-22ac5669d249`::moving from state preparing -> state finished >Thread-27170::DEBUG::2012-12-02 14:19:00,399::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27170::DEBUG::2012-12-02 14:19:00,399::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27170::DEBUG::2012-12-02 14:19:00,399::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27170::DEBUG::2012-12-02 14:19:00,399::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27170::DEBUG::2012-12-02 14:19:00,399::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27170::DEBUG::2012-12-02 14:19:00,399::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27170::DEBUG::2012-12-02 14:19:00,399::task::957::TaskManager.Task::(_decref) Task=`95c5a98b-f6f1-4c69-b2f3-22ac5669d249`::ref 0 aborting False >Thread-27174::DEBUG::2012-12-02 14:19:07,734::task::568::TaskManager.Task::(_updateState) Task=`de5fe47b-5ddd-45a5-92af-0517b32b34e1`::moving from state init -> state preparing >Thread-27174::INFO::2012-12-02 14:19:07,734::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27174::INFO::2012-12-02 14:19:07,735::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00829219818115', 'lastCheck': 1354450746.361594, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00197887420654', 'lastCheck': 1354450741.613719, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177311897278', 'lastCheck': 1354450744.2723999, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00775384902954', 'lastCheck': 1354450741.660521, 'code': 0, 'valid': True}} >Thread-27174::DEBUG::2012-12-02 14:19:07,735::task::1151::TaskManager.Task::(prepare) Task=`de5fe47b-5ddd-45a5-92af-0517b32b34e1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00829219818115', 'lastCheck': 1354450746.361594, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00197887420654', 'lastCheck': 1354450741.613719, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177311897278', 'lastCheck': 1354450744.2723999, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00775384902954', 'lastCheck': 1354450741.660521, 'code': 0, 'valid': True}} >Thread-27174::DEBUG::2012-12-02 14:19:07,735::task::568::TaskManager.Task::(_updateState) Task=`de5fe47b-5ddd-45a5-92af-0517b32b34e1`::moving from state preparing -> state finished >Thread-27174::DEBUG::2012-12-02 14:19:07,735::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27174::DEBUG::2012-12-02 14:19:07,735::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27174::DEBUG::2012-12-02 14:19:07,735::task::957::TaskManager.Task::(_decref) Task=`de5fe47b-5ddd-45a5-92af-0517b32b34e1`::ref 0 aborting False >Thread-27177::DEBUG::2012-12-02 14:19:10,526::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27177::DEBUG::2012-12-02 14:19:10,526::task::568::TaskManager.Task::(_updateState) Task=`8e0fc5ef-4f78-47c1-a757-8983a4fce395`::moving from state init -> state preparing >Thread-27177::INFO::2012-12-02 14:19:10,527::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27177::INFO::2012-12-02 14:19:10,527::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27177::DEBUG::2012-12-02 14:19:10,527::task::1151::TaskManager.Task::(prepare) Task=`8e0fc5ef-4f78-47c1-a757-8983a4fce395`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27177::DEBUG::2012-12-02 14:19:10,527::task::568::TaskManager.Task::(_updateState) Task=`8e0fc5ef-4f78-47c1-a757-8983a4fce395`::moving from state preparing -> state finished >Thread-27177::DEBUG::2012-12-02 14:19:10,527::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27177::DEBUG::2012-12-02 14:19:10,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27177::DEBUG::2012-12-02 14:19:10,527::task::957::TaskManager.Task::(_decref) Task=`8e0fc5ef-4f78-47c1-a757-8983a4fce395`::ref 0 aborting False >Thread-27178::DEBUG::2012-12-02 14:19:10,530::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27178::DEBUG::2012-12-02 14:19:10,530::task::568::TaskManager.Task::(_updateState) Task=`a8bb76df-e7b7-4f3e-adc0-d13b3e06ae65`::moving from state init -> state preparing >Thread-27178::INFO::2012-12-02 14:19:10,530::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27178::DEBUG::2012-12-02 14:19:10,531::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`860d5d3c-5c55-4b42-b09a-256970d0be83`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27178::DEBUG::2012-12-02 14:19:10,531::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27178::DEBUG::2012-12-02 14:19:10,531::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27178::DEBUG::2012-12-02 14:19:10,531::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`860d5d3c-5c55-4b42-b09a-256970d0be83`::Granted request >Thread-27178::DEBUG::2012-12-02 14:19:10,531::task::794::TaskManager.Task::(resourceAcquired) Task=`a8bb76df-e7b7-4f3e-adc0-d13b3e06ae65`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27178::DEBUG::2012-12-02 14:19:10,532::task::957::TaskManager.Task::(_decref) Task=`a8bb76df-e7b7-4f3e-adc0-d13b3e06ae65`::ref 1 aborting False >Thread-27178::INFO::2012-12-02 14:19:10,537::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27178::DEBUG::2012-12-02 14:19:10,537::task::1151::TaskManager.Task::(prepare) Task=`a8bb76df-e7b7-4f3e-adc0-d13b3e06ae65`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953885392896'}}} >Thread-27178::DEBUG::2012-12-02 14:19:10,538::task::568::TaskManager.Task::(_updateState) Task=`a8bb76df-e7b7-4f3e-adc0-d13b3e06ae65`::moving from state preparing -> state finished >Thread-27178::DEBUG::2012-12-02 14:19:10,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27178::DEBUG::2012-12-02 14:19:10,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27178::DEBUG::2012-12-02 14:19:10,538::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27178::DEBUG::2012-12-02 14:19:10,538::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27178::DEBUG::2012-12-02 14:19:10,538::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27178::DEBUG::2012-12-02 14:19:10,538::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27178::DEBUG::2012-12-02 14:19:10,538::task::957::TaskManager.Task::(_decref) Task=`a8bb76df-e7b7-4f3e-adc0-d13b3e06ae65`::ref 0 aborting False >Thread-27182::DEBUG::2012-12-02 14:19:17,830::task::568::TaskManager.Task::(_updateState) Task=`8e6ac045-660d-4f49-9c9c-c839c08466f3`::moving from state init -> state preparing >Thread-27182::INFO::2012-12-02 14:19:17,830::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27182::INFO::2012-12-02 14:19:17,830::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00793600082397', 'lastCheck': 1354450756.3791299, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00462985038757', 'lastCheck': 1354450751.6346059, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00316905975342', 'lastCheck': 1354450754.2851419, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00339603424072', 'lastCheck': 1354450751.6721661, 'code': 0, 'valid': True}} >Thread-27182::DEBUG::2012-12-02 14:19:17,831::task::1151::TaskManager.Task::(prepare) Task=`8e6ac045-660d-4f49-9c9c-c839c08466f3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00793600082397', 'lastCheck': 1354450756.3791299, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00462985038757', 'lastCheck': 1354450751.6346059, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00316905975342', 'lastCheck': 1354450754.2851419, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00339603424072', 'lastCheck': 1354450751.6721661, 'code': 0, 'valid': True}} >Thread-27182::DEBUG::2012-12-02 14:19:17,831::task::568::TaskManager.Task::(_updateState) Task=`8e6ac045-660d-4f49-9c9c-c839c08466f3`::moving from state preparing -> state finished >Thread-27182::DEBUG::2012-12-02 14:19:17,831::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27182::DEBUG::2012-12-02 14:19:17,831::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27182::DEBUG::2012-12-02 14:19:17,831::task::957::TaskManager.Task::(_decref) Task=`8e6ac045-660d-4f49-9c9c-c839c08466f3`::ref 0 aborting False >Thread-27185::DEBUG::2012-12-02 14:19:20,664::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27185::DEBUG::2012-12-02 14:19:20,665::task::568::TaskManager.Task::(_updateState) Task=`46a5971c-aeda-40d8-868e-ee585c623672`::moving from state init -> state preparing >Thread-27185::INFO::2012-12-02 14:19:20,665::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27185::INFO::2012-12-02 14:19:20,665::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27185::DEBUG::2012-12-02 14:19:20,665::task::1151::TaskManager.Task::(prepare) Task=`46a5971c-aeda-40d8-868e-ee585c623672`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27185::DEBUG::2012-12-02 14:19:20,665::task::568::TaskManager.Task::(_updateState) Task=`46a5971c-aeda-40d8-868e-ee585c623672`::moving from state preparing -> state finished >Thread-27185::DEBUG::2012-12-02 14:19:20,665::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27185::DEBUG::2012-12-02 14:19:20,665::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27185::DEBUG::2012-12-02 14:19:20,665::task::957::TaskManager.Task::(_decref) Task=`46a5971c-aeda-40d8-868e-ee585c623672`::ref 0 aborting False >Thread-27186::DEBUG::2012-12-02 14:19:20,668::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27186::DEBUG::2012-12-02 14:19:20,668::task::568::TaskManager.Task::(_updateState) Task=`a56be200-3749-4b83-ad5d-a48c803e5eb2`::moving from state init -> state preparing >Thread-27186::INFO::2012-12-02 14:19:20,668::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27186::DEBUG::2012-12-02 14:19:20,668::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c7eb1bb7-56c7-4627-a1a6-53238276f3a1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27186::DEBUG::2012-12-02 14:19:20,668::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27186::DEBUG::2012-12-02 14:19:20,669::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27186::DEBUG::2012-12-02 14:19:20,669::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c7eb1bb7-56c7-4627-a1a6-53238276f3a1`::Granted request >Thread-27186::DEBUG::2012-12-02 14:19:20,669::task::794::TaskManager.Task::(resourceAcquired) Task=`a56be200-3749-4b83-ad5d-a48c803e5eb2`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27186::DEBUG::2012-12-02 14:19:20,669::task::957::TaskManager.Task::(_decref) Task=`a56be200-3749-4b83-ad5d-a48c803e5eb2`::ref 1 aborting False >Thread-27186::INFO::2012-12-02 14:19:20,673::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27186::DEBUG::2012-12-02 14:19:20,673::task::1151::TaskManager.Task::(prepare) Task=`a56be200-3749-4b83-ad5d-a48c803e5eb2`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27186::DEBUG::2012-12-02 14:19:20,674::task::568::TaskManager.Task::(_updateState) Task=`a56be200-3749-4b83-ad5d-a48c803e5eb2`::moving from state preparing -> state finished >Thread-27186::DEBUG::2012-12-02 14:19:20,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27186::DEBUG::2012-12-02 14:19:20,674::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27186::DEBUG::2012-12-02 14:19:20,674::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27186::DEBUG::2012-12-02 14:19:20,674::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27186::DEBUG::2012-12-02 14:19:20,674::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27186::DEBUG::2012-12-02 14:19:20,674::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27186::DEBUG::2012-12-02 14:19:20,674::task::957::TaskManager.Task::(_decref) Task=`a56be200-3749-4b83-ad5d-a48c803e5eb2`::ref 0 aborting False >Thread-27190::DEBUG::2012-12-02 14:19:27,945::task::568::TaskManager.Task::(_updateState) Task=`875e0564-841f-4ada-8fa8-3b18683d60f7`::moving from state init -> state preparing >Thread-27190::INFO::2012-12-02 14:19:27,945::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27190::INFO::2012-12-02 14:19:27,946::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00343108177185', 'lastCheck': 1354450766.390902, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00171494483948', 'lastCheck': 1354450761.6487491, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00339198112488', 'lastCheck': 1354450764.2997661, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00629782676697', 'lastCheck': 1354450761.6862161, 'code': 0, 'valid': True}} >Thread-27190::DEBUG::2012-12-02 14:19:27,946::task::1151::TaskManager.Task::(prepare) Task=`875e0564-841f-4ada-8fa8-3b18683d60f7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00343108177185', 'lastCheck': 1354450766.390902, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00171494483948', 'lastCheck': 1354450761.6487491, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00339198112488', 'lastCheck': 1354450764.2997661, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00629782676697', 'lastCheck': 1354450761.6862161, 'code': 0, 'valid': True}} >Thread-27190::DEBUG::2012-12-02 14:19:27,946::task::568::TaskManager.Task::(_updateState) Task=`875e0564-841f-4ada-8fa8-3b18683d60f7`::moving from state preparing -> state finished >Thread-27190::DEBUG::2012-12-02 14:19:27,946::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27190::DEBUG::2012-12-02 14:19:27,946::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27190::DEBUG::2012-12-02 14:19:27,947::task::957::TaskManager.Task::(_decref) Task=`875e0564-841f-4ada-8fa8-3b18683d60f7`::ref 0 aborting False >Thread-27193::DEBUG::2012-12-02 14:19:30,795::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27193::DEBUG::2012-12-02 14:19:30,795::task::568::TaskManager.Task::(_updateState) Task=`da24c14b-9502-4f5e-a265-180da5dfd737`::moving from state init -> state preparing >Thread-27193::INFO::2012-12-02 14:19:30,795::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27193::INFO::2012-12-02 14:19:30,796::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27193::DEBUG::2012-12-02 14:19:30,796::task::1151::TaskManager.Task::(prepare) Task=`da24c14b-9502-4f5e-a265-180da5dfd737`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27193::DEBUG::2012-12-02 14:19:30,796::task::568::TaskManager.Task::(_updateState) Task=`da24c14b-9502-4f5e-a265-180da5dfd737`::moving from state preparing -> state finished >Thread-27193::DEBUG::2012-12-02 14:19:30,796::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27193::DEBUG::2012-12-02 14:19:30,796::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27193::DEBUG::2012-12-02 14:19:30,796::task::957::TaskManager.Task::(_decref) Task=`da24c14b-9502-4f5e-a265-180da5dfd737`::ref 0 aborting False >Thread-27194::DEBUG::2012-12-02 14:19:30,799::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27194::DEBUG::2012-12-02 14:19:30,799::task::568::TaskManager.Task::(_updateState) Task=`e1fb9095-030c-4a98-a3a0-5ef00c778770`::moving from state init -> state preparing >Thread-27194::INFO::2012-12-02 14:19:30,799::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27194::DEBUG::2012-12-02 14:19:30,800::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`297ce98d-c68f-41d0-83c8-afe26679cbdc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27194::DEBUG::2012-12-02 14:19:30,800::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27194::DEBUG::2012-12-02 14:19:30,800::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27194::DEBUG::2012-12-02 14:19:30,800::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`297ce98d-c68f-41d0-83c8-afe26679cbdc`::Granted request >Thread-27194::DEBUG::2012-12-02 14:19:30,800::task::794::TaskManager.Task::(resourceAcquired) Task=`e1fb9095-030c-4a98-a3a0-5ef00c778770`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27194::DEBUG::2012-12-02 14:19:30,800::task::957::TaskManager.Task::(_decref) Task=`e1fb9095-030c-4a98-a3a0-5ef00c778770`::ref 1 aborting False >Thread-27194::INFO::2012-12-02 14:19:30,806::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27194::DEBUG::2012-12-02 14:19:30,806::task::1151::TaskManager.Task::(prepare) Task=`e1fb9095-030c-4a98-a3a0-5ef00c778770`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27194::DEBUG::2012-12-02 14:19:30,807::task::568::TaskManager.Task::(_updateState) Task=`e1fb9095-030c-4a98-a3a0-5ef00c778770`::moving from state preparing -> state finished >Thread-27194::DEBUG::2012-12-02 14:19:30,807::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27194::DEBUG::2012-12-02 14:19:30,807::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27194::DEBUG::2012-12-02 14:19:30,807::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27194::DEBUG::2012-12-02 14:19:30,807::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27194::DEBUG::2012-12-02 14:19:30,807::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27194::DEBUG::2012-12-02 14:19:30,807::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27194::DEBUG::2012-12-02 14:19:30,807::task::957::TaskManager.Task::(_decref) Task=`e1fb9095-030c-4a98-a3a0-5ef00c778770`::ref 0 aborting False >Thread-27198::DEBUG::2012-12-02 14:19:38,058::task::568::TaskManager.Task::(_updateState) Task=`0d905818-b8df-4099-8b8c-f1d5fef19176`::moving from state init -> state preparing >Thread-27198::INFO::2012-12-02 14:19:38,058::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27198::INFO::2012-12-02 14:19:38,059::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00742101669312', 'lastCheck': 1354450776.409637, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00260996818542', 'lastCheck': 1354450771.6592529, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287199020386', 'lastCheck': 1354450774.31567, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00743508338928', 'lastCheck': 1354450771.702275, 'code': 0, 'valid': True}} >Thread-27198::DEBUG::2012-12-02 14:19:38,059::task::1151::TaskManager.Task::(prepare) Task=`0d905818-b8df-4099-8b8c-f1d5fef19176`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00742101669312', 'lastCheck': 1354450776.409637, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00260996818542', 'lastCheck': 1354450771.6592529, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287199020386', 'lastCheck': 1354450774.31567, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00743508338928', 'lastCheck': 1354450771.702275, 'code': 0, 'valid': True}} >Thread-27198::DEBUG::2012-12-02 14:19:38,059::task::568::TaskManager.Task::(_updateState) Task=`0d905818-b8df-4099-8b8c-f1d5fef19176`::moving from state preparing -> state finished >Thread-27198::DEBUG::2012-12-02 14:19:38,059::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27198::DEBUG::2012-12-02 14:19:38,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27198::DEBUG::2012-12-02 14:19:38,059::task::957::TaskManager.Task::(_decref) Task=`0d905818-b8df-4099-8b8c-f1d5fef19176`::ref 0 aborting False >Thread-27201::DEBUG::2012-12-02 14:19:40,932::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27201::DEBUG::2012-12-02 14:19:40,933::task::568::TaskManager.Task::(_updateState) Task=`e8030d59-1b88-434a-8bcd-af42d6c16e41`::moving from state init -> state preparing >Thread-27201::INFO::2012-12-02 14:19:40,933::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27201::INFO::2012-12-02 14:19:40,933::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27201::DEBUG::2012-12-02 14:19:40,933::task::1151::TaskManager.Task::(prepare) Task=`e8030d59-1b88-434a-8bcd-af42d6c16e41`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27201::DEBUG::2012-12-02 14:19:40,933::task::568::TaskManager.Task::(_updateState) Task=`e8030d59-1b88-434a-8bcd-af42d6c16e41`::moving from state preparing -> state finished >Thread-27201::DEBUG::2012-12-02 14:19:40,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27201::DEBUG::2012-12-02 14:19:40,934::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27201::DEBUG::2012-12-02 14:19:40,934::task::957::TaskManager.Task::(_decref) Task=`e8030d59-1b88-434a-8bcd-af42d6c16e41`::ref 0 aborting False >Thread-27202::DEBUG::2012-12-02 14:19:40,936::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27202::DEBUG::2012-12-02 14:19:40,936::task::568::TaskManager.Task::(_updateState) Task=`62eb6288-074a-4595-89c8-2b03302d6128`::moving from state init -> state preparing >Thread-27202::INFO::2012-12-02 14:19:40,937::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27202::DEBUG::2012-12-02 14:19:40,937::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`13843b73-72bf-4035-bf0b-ed2be04a14bc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27202::DEBUG::2012-12-02 14:19:40,937::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27202::DEBUG::2012-12-02 14:19:40,937::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27202::DEBUG::2012-12-02 14:19:40,937::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`13843b73-72bf-4035-bf0b-ed2be04a14bc`::Granted request >Thread-27202::DEBUG::2012-12-02 14:19:40,937::task::794::TaskManager.Task::(resourceAcquired) Task=`62eb6288-074a-4595-89c8-2b03302d6128`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27202::DEBUG::2012-12-02 14:19:40,937::task::957::TaskManager.Task::(_decref) Task=`62eb6288-074a-4595-89c8-2b03302d6128`::ref 1 aborting False >Thread-27202::INFO::2012-12-02 14:19:40,942::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27202::DEBUG::2012-12-02 14:19:40,942::task::1151::TaskManager.Task::(prepare) Task=`62eb6288-074a-4595-89c8-2b03302d6128`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27202::DEBUG::2012-12-02 14:19:40,942::task::568::TaskManager.Task::(_updateState) Task=`62eb6288-074a-4595-89c8-2b03302d6128`::moving from state preparing -> state finished >Thread-27202::DEBUG::2012-12-02 14:19:40,942::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27202::DEBUG::2012-12-02 14:19:40,942::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27202::DEBUG::2012-12-02 14:19:40,942::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27202::DEBUG::2012-12-02 14:19:40,942::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27202::DEBUG::2012-12-02 14:19:40,942::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27202::DEBUG::2012-12-02 14:19:40,942::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27202::DEBUG::2012-12-02 14:19:40,942::task::957::TaskManager.Task::(_decref) Task=`62eb6288-074a-4595-89c8-2b03302d6128`::ref 0 aborting False >Thread-27206::DEBUG::2012-12-02 14:19:48,187::task::568::TaskManager.Task::(_updateState) Task=`ea9986f3-a577-4d97-97b3-5d738cbf783d`::moving from state init -> state preparing >Thread-27206::INFO::2012-12-02 14:19:48,187::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27206::INFO::2012-12-02 14:19:48,187::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00713205337524', 'lastCheck': 1354450786.4264281, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0020899772644', 'lastCheck': 1354450781.6745, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00190210342407', 'lastCheck': 1354450784.3272221, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0080771446228', 'lastCheck': 1354450781.720974, 'code': 0, 'valid': True}} >Thread-27206::DEBUG::2012-12-02 14:19:48,188::task::1151::TaskManager.Task::(prepare) Task=`ea9986f3-a577-4d97-97b3-5d738cbf783d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00713205337524', 'lastCheck': 1354450786.4264281, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0020899772644', 'lastCheck': 1354450781.6745, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00190210342407', 'lastCheck': 1354450784.3272221, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0080771446228', 'lastCheck': 1354450781.720974, 'code': 0, 'valid': True}} >Thread-27206::DEBUG::2012-12-02 14:19:48,188::task::568::TaskManager.Task::(_updateState) Task=`ea9986f3-a577-4d97-97b3-5d738cbf783d`::moving from state preparing -> state finished >Thread-27206::DEBUG::2012-12-02 14:19:48,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27206::DEBUG::2012-12-02 14:19:48,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27206::DEBUG::2012-12-02 14:19:48,188::task::957::TaskManager.Task::(_decref) Task=`ea9986f3-a577-4d97-97b3-5d738cbf783d`::ref 0 aborting False >Thread-27209::DEBUG::2012-12-02 14:19:51,061::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27209::DEBUG::2012-12-02 14:19:51,061::task::568::TaskManager.Task::(_updateState) Task=`63d6f21b-ce48-44f9-9b33-23f9aa7ed65e`::moving from state init -> state preparing >Thread-27209::INFO::2012-12-02 14:19:51,061::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27209::INFO::2012-12-02 14:19:51,061::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27209::DEBUG::2012-12-02 14:19:51,061::task::1151::TaskManager.Task::(prepare) Task=`63d6f21b-ce48-44f9-9b33-23f9aa7ed65e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27209::DEBUG::2012-12-02 14:19:51,061::task::568::TaskManager.Task::(_updateState) Task=`63d6f21b-ce48-44f9-9b33-23f9aa7ed65e`::moving from state preparing -> state finished >Thread-27209::DEBUG::2012-12-02 14:19:51,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27209::DEBUG::2012-12-02 14:19:51,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27209::DEBUG::2012-12-02 14:19:51,062::task::957::TaskManager.Task::(_decref) Task=`63d6f21b-ce48-44f9-9b33-23f9aa7ed65e`::ref 0 aborting False >Thread-27210::DEBUG::2012-12-02 14:19:51,064::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27210::DEBUG::2012-12-02 14:19:51,065::task::568::TaskManager.Task::(_updateState) Task=`c89e1c39-046b-4e05-a531-7a53d3659dfb`::moving from state init -> state preparing >Thread-27210::INFO::2012-12-02 14:19:51,065::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27210::DEBUG::2012-12-02 14:19:51,065::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`eb1eec98-3b92-4055-9a0f-f9c5bc212901`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27210::DEBUG::2012-12-02 14:19:51,065::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27210::DEBUG::2012-12-02 14:19:51,065::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27210::DEBUG::2012-12-02 14:19:51,065::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`eb1eec98-3b92-4055-9a0f-f9c5bc212901`::Granted request >Thread-27210::DEBUG::2012-12-02 14:19:51,066::task::794::TaskManager.Task::(resourceAcquired) Task=`c89e1c39-046b-4e05-a531-7a53d3659dfb`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27210::DEBUG::2012-12-02 14:19:51,066::task::957::TaskManager.Task::(_decref) Task=`c89e1c39-046b-4e05-a531-7a53d3659dfb`::ref 1 aborting False >Thread-27210::INFO::2012-12-02 14:19:51,070::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27210::DEBUG::2012-12-02 14:19:51,070::task::1151::TaskManager.Task::(prepare) Task=`c89e1c39-046b-4e05-a531-7a53d3659dfb`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27210::DEBUG::2012-12-02 14:19:51,071::task::568::TaskManager.Task::(_updateState) Task=`c89e1c39-046b-4e05-a531-7a53d3659dfb`::moving from state preparing -> state finished >Thread-27210::DEBUG::2012-12-02 14:19:51,071::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27210::DEBUG::2012-12-02 14:19:51,071::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27210::DEBUG::2012-12-02 14:19:51,071::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27210::DEBUG::2012-12-02 14:19:51,071::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27210::DEBUG::2012-12-02 14:19:51,071::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27210::DEBUG::2012-12-02 14:19:51,071::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27210::DEBUG::2012-12-02 14:19:51,071::task::957::TaskManager.Task::(_decref) Task=`c89e1c39-046b-4e05-a531-7a53d3659dfb`::ref 0 aborting False >Thread-27214::DEBUG::2012-12-02 14:19:58,283::task::568::TaskManager.Task::(_updateState) Task=`28146a3d-060e-49fd-ac3b-8ca7758f6665`::moving from state init -> state preparing >Thread-27214::INFO::2012-12-02 14:19:58,283::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27214::INFO::2012-12-02 14:19:58,283::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00924897193909', 'lastCheck': 1354450796.4448831, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0024778842926', 'lastCheck': 1354450791.686954, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178790092468', 'lastCheck': 1354450794.33833, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00687885284424', 'lastCheck': 1354450791.737009, 'code': 0, 'valid': True}} >Thread-27214::DEBUG::2012-12-02 14:19:58,283::task::1151::TaskManager.Task::(prepare) Task=`28146a3d-060e-49fd-ac3b-8ca7758f6665`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00924897193909', 'lastCheck': 1354450796.4448831, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0024778842926', 'lastCheck': 1354450791.686954, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178790092468', 'lastCheck': 1354450794.33833, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00687885284424', 'lastCheck': 1354450791.737009, 'code': 0, 'valid': True}} >Thread-27214::DEBUG::2012-12-02 14:19:58,283::task::568::TaskManager.Task::(_updateState) Task=`28146a3d-060e-49fd-ac3b-8ca7758f6665`::moving from state preparing -> state finished >Thread-27214::DEBUG::2012-12-02 14:19:58,284::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27214::DEBUG::2012-12-02 14:19:58,284::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27214::DEBUG::2012-12-02 14:19:58,284::task::957::TaskManager.Task::(_decref) Task=`28146a3d-060e-49fd-ac3b-8ca7758f6665`::ref 0 aborting False >Thread-27217::DEBUG::2012-12-02 14:20:01,191::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27217::DEBUG::2012-12-02 14:20:01,191::task::568::TaskManager.Task::(_updateState) Task=`dc5bcc5e-2e29-410e-b7be-2b61b0faff7e`::moving from state init -> state preparing >Thread-27217::INFO::2012-12-02 14:20:01,191::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27217::INFO::2012-12-02 14:20:01,191::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27217::DEBUG::2012-12-02 14:20:01,191::task::1151::TaskManager.Task::(prepare) Task=`dc5bcc5e-2e29-410e-b7be-2b61b0faff7e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27217::DEBUG::2012-12-02 14:20:01,192::task::568::TaskManager.Task::(_updateState) Task=`dc5bcc5e-2e29-410e-b7be-2b61b0faff7e`::moving from state preparing -> state finished >Thread-27217::DEBUG::2012-12-02 14:20:01,192::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27217::DEBUG::2012-12-02 14:20:01,192::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27217::DEBUG::2012-12-02 14:20:01,192::task::957::TaskManager.Task::(_decref) Task=`dc5bcc5e-2e29-410e-b7be-2b61b0faff7e`::ref 0 aborting False >Thread-27218::DEBUG::2012-12-02 14:20:01,194::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27218::DEBUG::2012-12-02 14:20:01,195::task::568::TaskManager.Task::(_updateState) Task=`31e49c27-64b3-43bf-a146-b14ec6c88044`::moving from state init -> state preparing >Thread-27218::INFO::2012-12-02 14:20:01,195::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27218::DEBUG::2012-12-02 14:20:01,195::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7ea9f5a0-ae33-4707-b908-28874df527fe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27218::DEBUG::2012-12-02 14:20:01,195::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27218::DEBUG::2012-12-02 14:20:01,196::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27218::DEBUG::2012-12-02 14:20:01,196::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7ea9f5a0-ae33-4707-b908-28874df527fe`::Granted request >Thread-27218::DEBUG::2012-12-02 14:20:01,196::task::794::TaskManager.Task::(resourceAcquired) Task=`31e49c27-64b3-43bf-a146-b14ec6c88044`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27218::DEBUG::2012-12-02 14:20:01,196::task::957::TaskManager.Task::(_decref) Task=`31e49c27-64b3-43bf-a146-b14ec6c88044`::ref 1 aborting False >Thread-27218::INFO::2012-12-02 14:20:01,203::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27218::DEBUG::2012-12-02 14:20:01,203::task::1151::TaskManager.Task::(prepare) Task=`31e49c27-64b3-43bf-a146-b14ec6c88044`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27218::DEBUG::2012-12-02 14:20:01,203::task::568::TaskManager.Task::(_updateState) Task=`31e49c27-64b3-43bf-a146-b14ec6c88044`::moving from state preparing -> state finished >Thread-27218::DEBUG::2012-12-02 14:20:01,203::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27218::DEBUG::2012-12-02 14:20:01,203::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27218::DEBUG::2012-12-02 14:20:01,203::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27218::DEBUG::2012-12-02 14:20:01,203::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27218::DEBUG::2012-12-02 14:20:01,204::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27218::DEBUG::2012-12-02 14:20:01,204::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27218::DEBUG::2012-12-02 14:20:01,204::task::957::TaskManager.Task::(_decref) Task=`31e49c27-64b3-43bf-a146-b14ec6c88044`::ref 0 aborting False >Thread-27222::DEBUG::2012-12-02 14:20:08,387::task::568::TaskManager.Task::(_updateState) Task=`de0bb23b-f795-4ccc-9a39-88d2cebc6a8c`::moving from state init -> state preparing >Thread-27222::INFO::2012-12-02 14:20:08,388::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27222::INFO::2012-12-02 14:20:08,388::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00710606575012', 'lastCheck': 1354450806.4608669, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00227499008179', 'lastCheck': 1354450801.6996901, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163888931274', 'lastCheck': 1354450804.3489671, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842499732971', 'lastCheck': 1354450801.7545829, 'code': 0, 'valid': True}} >Thread-27222::DEBUG::2012-12-02 14:20:08,388::task::1151::TaskManager.Task::(prepare) Task=`de0bb23b-f795-4ccc-9a39-88d2cebc6a8c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00710606575012', 'lastCheck': 1354450806.4608669, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00227499008179', 'lastCheck': 1354450801.6996901, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163888931274', 'lastCheck': 1354450804.3489671, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842499732971', 'lastCheck': 1354450801.7545829, 'code': 0, 'valid': True}} >Thread-27222::DEBUG::2012-12-02 14:20:08,388::task::568::TaskManager.Task::(_updateState) Task=`de0bb23b-f795-4ccc-9a39-88d2cebc6a8c`::moving from state preparing -> state finished >Thread-27222::DEBUG::2012-12-02 14:20:08,388::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27222::DEBUG::2012-12-02 14:20:08,388::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27222::DEBUG::2012-12-02 14:20:08,388::task::957::TaskManager.Task::(_decref) Task=`de0bb23b-f795-4ccc-9a39-88d2cebc6a8c`::ref 0 aborting False >Thread-27225::DEBUG::2012-12-02 14:20:11,361::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27225::DEBUG::2012-12-02 14:20:11,362::task::568::TaskManager.Task::(_updateState) Task=`ce2ecab0-7416-4717-929e-ab9c27c3a618`::moving from state init -> state preparing >Thread-27225::INFO::2012-12-02 14:20:11,362::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27225::INFO::2012-12-02 14:20:11,362::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27225::DEBUG::2012-12-02 14:20:11,362::task::1151::TaskManager.Task::(prepare) Task=`ce2ecab0-7416-4717-929e-ab9c27c3a618`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27225::DEBUG::2012-12-02 14:20:11,362::task::568::TaskManager.Task::(_updateState) Task=`ce2ecab0-7416-4717-929e-ab9c27c3a618`::moving from state preparing -> state finished >Thread-27225::DEBUG::2012-12-02 14:20:11,362::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27225::DEBUG::2012-12-02 14:20:11,363::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27225::DEBUG::2012-12-02 14:20:11,363::task::957::TaskManager.Task::(_decref) Task=`ce2ecab0-7416-4717-929e-ab9c27c3a618`::ref 0 aborting False >Thread-27226::DEBUG::2012-12-02 14:20:11,365::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27226::DEBUG::2012-12-02 14:20:11,365::task::568::TaskManager.Task::(_updateState) Task=`e45882ca-48fb-47ae-a741-f2f2fed40313`::moving from state init -> state preparing >Thread-27226::INFO::2012-12-02 14:20:11,366::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27226::DEBUG::2012-12-02 14:20:11,366::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e35ed58-4354-4d01-b7ca-ce05c2e30f60`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27226::DEBUG::2012-12-02 14:20:11,366::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27226::DEBUG::2012-12-02 14:20:11,366::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27226::DEBUG::2012-12-02 14:20:11,366::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e35ed58-4354-4d01-b7ca-ce05c2e30f60`::Granted request >Thread-27226::DEBUG::2012-12-02 14:20:11,366::task::794::TaskManager.Task::(resourceAcquired) Task=`e45882ca-48fb-47ae-a741-f2f2fed40313`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27226::DEBUG::2012-12-02 14:20:11,367::task::957::TaskManager.Task::(_decref) Task=`e45882ca-48fb-47ae-a741-f2f2fed40313`::ref 1 aborting False >Thread-27226::INFO::2012-12-02 14:20:11,371::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27226::DEBUG::2012-12-02 14:20:11,371::task::1151::TaskManager.Task::(prepare) Task=`e45882ca-48fb-47ae-a741-f2f2fed40313`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27226::DEBUG::2012-12-02 14:20:11,371::task::568::TaskManager.Task::(_updateState) Task=`e45882ca-48fb-47ae-a741-f2f2fed40313`::moving from state preparing -> state finished >Thread-27226::DEBUG::2012-12-02 14:20:11,371::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27226::DEBUG::2012-12-02 14:20:11,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27226::DEBUG::2012-12-02 14:20:11,371::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27226::DEBUG::2012-12-02 14:20:11,372::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27226::DEBUG::2012-12-02 14:20:11,372::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27226::DEBUG::2012-12-02 14:20:11,372::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27226::DEBUG::2012-12-02 14:20:11,372::task::957::TaskManager.Task::(_decref) Task=`e45882ca-48fb-47ae-a741-f2f2fed40313`::ref 0 aborting False >Thread-27230::DEBUG::2012-12-02 14:20:18,593::task::568::TaskManager.Task::(_updateState) Task=`12213f7f-1aab-4337-a8fe-2c7ac196a2fb`::moving from state init -> state preparing >Thread-27230::INFO::2012-12-02 14:20:18,593::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27230::INFO::2012-12-02 14:20:18,594::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833415985107', 'lastCheck': 1354450816.4785869, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00391387939453', 'lastCheck': 1354450811.714361, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00276112556458', 'lastCheck': 1354450814.3622999, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00922608375549', 'lastCheck': 1354450811.7726071, 'code': 0, 'valid': True}} >Thread-27230::DEBUG::2012-12-02 14:20:18,594::task::1151::TaskManager.Task::(prepare) Task=`12213f7f-1aab-4337-a8fe-2c7ac196a2fb`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833415985107', 'lastCheck': 1354450816.4785869, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00391387939453', 'lastCheck': 1354450811.714361, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00276112556458', 'lastCheck': 1354450814.3622999, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00922608375549', 'lastCheck': 1354450811.7726071, 'code': 0, 'valid': True}} >Thread-27230::DEBUG::2012-12-02 14:20:18,594::task::568::TaskManager.Task::(_updateState) Task=`12213f7f-1aab-4337-a8fe-2c7ac196a2fb`::moving from state preparing -> state finished >Thread-27230::DEBUG::2012-12-02 14:20:18,594::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27230::DEBUG::2012-12-02 14:20:18,594::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27230::DEBUG::2012-12-02 14:20:18,594::task::957::TaskManager.Task::(_decref) Task=`12213f7f-1aab-4337-a8fe-2c7ac196a2fb`::ref 0 aborting False >Thread-27233::DEBUG::2012-12-02 14:20:21,492::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27233::DEBUG::2012-12-02 14:20:21,493::task::568::TaskManager.Task::(_updateState) Task=`ed3f2650-aa59-4380-b3a4-59849b7f89e2`::moving from state init -> state preparing >Thread-27233::INFO::2012-12-02 14:20:21,493::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27233::INFO::2012-12-02 14:20:21,493::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27233::DEBUG::2012-12-02 14:20:21,493::task::1151::TaskManager.Task::(prepare) Task=`ed3f2650-aa59-4380-b3a4-59849b7f89e2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27233::DEBUG::2012-12-02 14:20:21,493::task::568::TaskManager.Task::(_updateState) Task=`ed3f2650-aa59-4380-b3a4-59849b7f89e2`::moving from state preparing -> state finished >Thread-27233::DEBUG::2012-12-02 14:20:21,493::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27233::DEBUG::2012-12-02 14:20:21,494::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27233::DEBUG::2012-12-02 14:20:21,494::task::957::TaskManager.Task::(_decref) Task=`ed3f2650-aa59-4380-b3a4-59849b7f89e2`::ref 0 aborting False >Thread-27234::DEBUG::2012-12-02 14:20:21,496::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27234::DEBUG::2012-12-02 14:20:21,496::task::568::TaskManager.Task::(_updateState) Task=`a782d9e9-1922-41d3-8a79-10910e8d5d7c`::moving from state init -> state preparing >Thread-27234::INFO::2012-12-02 14:20:21,497::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27234::DEBUG::2012-12-02 14:20:21,497::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a5d27319-2ce4-42d3-a13e-8c1b1b3a7d41`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27234::DEBUG::2012-12-02 14:20:21,497::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27234::DEBUG::2012-12-02 14:20:21,497::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27234::DEBUG::2012-12-02 14:20:21,497::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a5d27319-2ce4-42d3-a13e-8c1b1b3a7d41`::Granted request >Thread-27234::DEBUG::2012-12-02 14:20:21,498::task::794::TaskManager.Task::(resourceAcquired) Task=`a782d9e9-1922-41d3-8a79-10910e8d5d7c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27234::DEBUG::2012-12-02 14:20:21,498::task::957::TaskManager.Task::(_decref) Task=`a782d9e9-1922-41d3-8a79-10910e8d5d7c`::ref 1 aborting False >Thread-27234::INFO::2012-12-02 14:20:21,504::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27234::DEBUG::2012-12-02 14:20:21,504::task::1151::TaskManager.Task::(prepare) Task=`a782d9e9-1922-41d3-8a79-10910e8d5d7c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27234::DEBUG::2012-12-02 14:20:21,505::task::568::TaskManager.Task::(_updateState) Task=`a782d9e9-1922-41d3-8a79-10910e8d5d7c`::moving from state preparing -> state finished >Thread-27234::DEBUG::2012-12-02 14:20:21,505::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27234::DEBUG::2012-12-02 14:20:21,505::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27234::DEBUG::2012-12-02 14:20:21,505::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27234::DEBUG::2012-12-02 14:20:21,505::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27234::DEBUG::2012-12-02 14:20:21,505::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27234::DEBUG::2012-12-02 14:20:21,505::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27234::DEBUG::2012-12-02 14:20:21,505::task::957::TaskManager.Task::(_decref) Task=`a782d9e9-1922-41d3-8a79-10910e8d5d7c`::ref 0 aborting False >Thread-27238::DEBUG::2012-12-02 14:20:28,688::task::568::TaskManager.Task::(_updateState) Task=`e3a4abc3-b150-4e95-bfcb-62994f6e80c1`::moving from state init -> state preparing >Thread-27238::INFO::2012-12-02 14:20:28,688::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27238::INFO::2012-12-02 14:20:28,688::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833010673523', 'lastCheck': 1354450826.4959741, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00205397605896', 'lastCheck': 1354450821.728852, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0015549659729', 'lastCheck': 1354450824.371969, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00709009170532', 'lastCheck': 1354450821.7884009, 'code': 0, 'valid': True}} >Thread-27238::DEBUG::2012-12-02 14:20:28,688::task::1151::TaskManager.Task::(prepare) Task=`e3a4abc3-b150-4e95-bfcb-62994f6e80c1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833010673523', 'lastCheck': 1354450826.4959741, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00205397605896', 'lastCheck': 1354450821.728852, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0015549659729', 'lastCheck': 1354450824.371969, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00709009170532', 'lastCheck': 1354450821.7884009, 'code': 0, 'valid': True}} >Thread-27238::DEBUG::2012-12-02 14:20:28,688::task::568::TaskManager.Task::(_updateState) Task=`e3a4abc3-b150-4e95-bfcb-62994f6e80c1`::moving from state preparing -> state finished >Thread-27238::DEBUG::2012-12-02 14:20:28,689::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27238::DEBUG::2012-12-02 14:20:28,689::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27238::DEBUG::2012-12-02 14:20:28,689::task::957::TaskManager.Task::(_decref) Task=`e3a4abc3-b150-4e95-bfcb-62994f6e80c1`::ref 0 aborting False >Thread-27241::DEBUG::2012-12-02 14:20:31,596::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27241::DEBUG::2012-12-02 14:20:31,596::task::568::TaskManager.Task::(_updateState) Task=`fb84cd5f-d0f5-4007-a8cc-f9b4c4c0bcc9`::moving from state init -> state preparing >Thread-27241::INFO::2012-12-02 14:20:31,597::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27241::INFO::2012-12-02 14:20:31,597::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27241::DEBUG::2012-12-02 14:20:31,597::task::1151::TaskManager.Task::(prepare) Task=`fb84cd5f-d0f5-4007-a8cc-f9b4c4c0bcc9`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27241::DEBUG::2012-12-02 14:20:31,597::task::568::TaskManager.Task::(_updateState) Task=`fb84cd5f-d0f5-4007-a8cc-f9b4c4c0bcc9`::moving from state preparing -> state finished >Thread-27241::DEBUG::2012-12-02 14:20:31,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27241::DEBUG::2012-12-02 14:20:31,597::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27241::DEBUG::2012-12-02 14:20:31,597::task::957::TaskManager.Task::(_decref) Task=`fb84cd5f-d0f5-4007-a8cc-f9b4c4c0bcc9`::ref 0 aborting False >Thread-27242::DEBUG::2012-12-02 14:20:31,600::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27242::DEBUG::2012-12-02 14:20:31,600::task::568::TaskManager.Task::(_updateState) Task=`ad61ed0b-31a0-469c-95f0-3cdd030ead9c`::moving from state init -> state preparing >Thread-27242::INFO::2012-12-02 14:20:31,600::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27242::DEBUG::2012-12-02 14:20:31,601::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2ce65395-2d2d-4c3b-85b7-b8c93b8255e0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27242::DEBUG::2012-12-02 14:20:31,601::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27242::DEBUG::2012-12-02 14:20:31,601::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27242::DEBUG::2012-12-02 14:20:31,601::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2ce65395-2d2d-4c3b-85b7-b8c93b8255e0`::Granted request >Thread-27242::DEBUG::2012-12-02 14:20:31,601::task::794::TaskManager.Task::(resourceAcquired) Task=`ad61ed0b-31a0-469c-95f0-3cdd030ead9c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27242::DEBUG::2012-12-02 14:20:31,601::task::957::TaskManager.Task::(_decref) Task=`ad61ed0b-31a0-469c-95f0-3cdd030ead9c`::ref 1 aborting False >Thread-27242::INFO::2012-12-02 14:20:31,607::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27242::DEBUG::2012-12-02 14:20:31,607::task::1151::TaskManager.Task::(prepare) Task=`ad61ed0b-31a0-469c-95f0-3cdd030ead9c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27242::DEBUG::2012-12-02 14:20:31,608::task::568::TaskManager.Task::(_updateState) Task=`ad61ed0b-31a0-469c-95f0-3cdd030ead9c`::moving from state preparing -> state finished >Thread-27242::DEBUG::2012-12-02 14:20:31,608::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27242::DEBUG::2012-12-02 14:20:31,608::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27242::DEBUG::2012-12-02 14:20:31,608::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27242::DEBUG::2012-12-02 14:20:31,608::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27242::DEBUG::2012-12-02 14:20:31,608::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27242::DEBUG::2012-12-02 14:20:31,608::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27242::DEBUG::2012-12-02 14:20:31,608::task::957::TaskManager.Task::(_decref) Task=`ad61ed0b-31a0-469c-95f0-3cdd030ead9c`::ref 0 aborting False >Thread-27246::DEBUG::2012-12-02 14:20:38,769::task::568::TaskManager.Task::(_updateState) Task=`e5080e29-1f31-4e3b-999f-c044bb640200`::moving from state init -> state preparing >Thread-27246::INFO::2012-12-02 14:20:38,769::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27246::INFO::2012-12-02 14:20:38,770::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00823616981506', 'lastCheck': 1354450836.5161941, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00386714935303', 'lastCheck': 1354450831.7430129, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00195407867432', 'lastCheck': 1354450834.38183, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00718808174133', 'lastCheck': 1354450831.804328, 'code': 0, 'valid': True}} >Thread-27246::DEBUG::2012-12-02 14:20:38,770::task::1151::TaskManager.Task::(prepare) Task=`e5080e29-1f31-4e3b-999f-c044bb640200`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00823616981506', 'lastCheck': 1354450836.5161941, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00386714935303', 'lastCheck': 1354450831.7430129, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00195407867432', 'lastCheck': 1354450834.38183, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00718808174133', 'lastCheck': 1354450831.804328, 'code': 0, 'valid': True}} >Thread-27246::DEBUG::2012-12-02 14:20:38,770::task::568::TaskManager.Task::(_updateState) Task=`e5080e29-1f31-4e3b-999f-c044bb640200`::moving from state preparing -> state finished >Thread-27246::DEBUG::2012-12-02 14:20:38,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27246::DEBUG::2012-12-02 14:20:38,770::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27246::DEBUG::2012-12-02 14:20:38,770::task::957::TaskManager.Task::(_decref) Task=`e5080e29-1f31-4e3b-999f-c044bb640200`::ref 0 aborting False >Thread-27249::DEBUG::2012-12-02 14:20:41,760::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27249::DEBUG::2012-12-02 14:20:41,760::task::568::TaskManager.Task::(_updateState) Task=`190c25b5-27c2-42c1-9cd9-8ac329d2c51b`::moving from state init -> state preparing >Thread-27249::INFO::2012-12-02 14:20:41,760::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27249::INFO::2012-12-02 14:20:41,761::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27249::DEBUG::2012-12-02 14:20:41,761::task::1151::TaskManager.Task::(prepare) Task=`190c25b5-27c2-42c1-9cd9-8ac329d2c51b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27249::DEBUG::2012-12-02 14:20:41,761::task::568::TaskManager.Task::(_updateState) Task=`190c25b5-27c2-42c1-9cd9-8ac329d2c51b`::moving from state preparing -> state finished >Thread-27249::DEBUG::2012-12-02 14:20:41,761::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27249::DEBUG::2012-12-02 14:20:41,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27249::DEBUG::2012-12-02 14:20:41,761::task::957::TaskManager.Task::(_decref) Task=`190c25b5-27c2-42c1-9cd9-8ac329d2c51b`::ref 0 aborting False >Thread-27250::DEBUG::2012-12-02 14:20:41,764::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27250::DEBUG::2012-12-02 14:20:41,764::task::568::TaskManager.Task::(_updateState) Task=`52c51517-65b4-4274-980e-aaeaff6cd257`::moving from state init -> state preparing >Thread-27250::INFO::2012-12-02 14:20:41,764::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27250::DEBUG::2012-12-02 14:20:41,765::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`13bbf23e-ab89-41df-9b05-89e02936e3c1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27250::DEBUG::2012-12-02 14:20:41,765::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27250::DEBUG::2012-12-02 14:20:41,765::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27250::DEBUG::2012-12-02 14:20:41,765::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`13bbf23e-ab89-41df-9b05-89e02936e3c1`::Granted request >Thread-27250::DEBUG::2012-12-02 14:20:41,765::task::794::TaskManager.Task::(resourceAcquired) Task=`52c51517-65b4-4274-980e-aaeaff6cd257`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27250::DEBUG::2012-12-02 14:20:41,765::task::957::TaskManager.Task::(_decref) Task=`52c51517-65b4-4274-980e-aaeaff6cd257`::ref 1 aborting False >Thread-27250::INFO::2012-12-02 14:20:41,770::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27250::DEBUG::2012-12-02 14:20:41,770::task::1151::TaskManager.Task::(prepare) Task=`52c51517-65b4-4274-980e-aaeaff6cd257`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27250::DEBUG::2012-12-02 14:20:41,770::task::568::TaskManager.Task::(_updateState) Task=`52c51517-65b4-4274-980e-aaeaff6cd257`::moving from state preparing -> state finished >Thread-27250::DEBUG::2012-12-02 14:20:41,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27250::DEBUG::2012-12-02 14:20:41,770::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27250::DEBUG::2012-12-02 14:20:41,770::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27250::DEBUG::2012-12-02 14:20:41,770::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27250::DEBUG::2012-12-02 14:20:41,770::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27250::DEBUG::2012-12-02 14:20:41,770::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27250::DEBUG::2012-12-02 14:20:41,771::task::957::TaskManager.Task::(_decref) Task=`52c51517-65b4-4274-980e-aaeaff6cd257`::ref 0 aborting False >Thread-27254::DEBUG::2012-12-02 14:20:48,833::task::568::TaskManager.Task::(_updateState) Task=`d8e585fd-1d28-4a40-9254-7b7e1278a055`::moving from state init -> state preparing >Thread-27254::INFO::2012-12-02 14:20:48,834::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27254::INFO::2012-12-02 14:20:48,834::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00735902786255', 'lastCheck': 1354450846.532428, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00217413902283', 'lastCheck': 1354450841.757849, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183892250061', 'lastCheck': 1354450844.391767, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00740504264832', 'lastCheck': 1354450841.8223269, 'code': 0, 'valid': True}} >Thread-27254::DEBUG::2012-12-02 14:20:48,834::task::1151::TaskManager.Task::(prepare) Task=`d8e585fd-1d28-4a40-9254-7b7e1278a055`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00735902786255', 'lastCheck': 1354450846.532428, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00217413902283', 'lastCheck': 1354450841.757849, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183892250061', 'lastCheck': 1354450844.391767, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00740504264832', 'lastCheck': 1354450841.8223269, 'code': 0, 'valid': True}} >Thread-27254::DEBUG::2012-12-02 14:20:48,834::task::568::TaskManager.Task::(_updateState) Task=`d8e585fd-1d28-4a40-9254-7b7e1278a055`::moving from state preparing -> state finished >Thread-27254::DEBUG::2012-12-02 14:20:48,834::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27254::DEBUG::2012-12-02 14:20:48,834::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27254::DEBUG::2012-12-02 14:20:48,834::task::957::TaskManager.Task::(_decref) Task=`d8e585fd-1d28-4a40-9254-7b7e1278a055`::ref 0 aborting False >Thread-27257::DEBUG::2012-12-02 14:20:51,890::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27257::DEBUG::2012-12-02 14:20:51,890::task::568::TaskManager.Task::(_updateState) Task=`318167ec-a82d-45ff-a21c-9d9ec503e5bf`::moving from state init -> state preparing >Thread-27257::INFO::2012-12-02 14:20:51,891::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27257::INFO::2012-12-02 14:20:51,891::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27257::DEBUG::2012-12-02 14:20:51,891::task::1151::TaskManager.Task::(prepare) Task=`318167ec-a82d-45ff-a21c-9d9ec503e5bf`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27257::DEBUG::2012-12-02 14:20:51,891::task::568::TaskManager.Task::(_updateState) Task=`318167ec-a82d-45ff-a21c-9d9ec503e5bf`::moving from state preparing -> state finished >Thread-27257::DEBUG::2012-12-02 14:20:51,891::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27257::DEBUG::2012-12-02 14:20:51,891::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27257::DEBUG::2012-12-02 14:20:51,892::task::957::TaskManager.Task::(_decref) Task=`318167ec-a82d-45ff-a21c-9d9ec503e5bf`::ref 0 aborting False >Thread-27258::DEBUG::2012-12-02 14:20:51,894::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27258::DEBUG::2012-12-02 14:20:51,894::task::568::TaskManager.Task::(_updateState) Task=`544e89a1-9d56-4ff8-b19c-bc1060b38a08`::moving from state init -> state preparing >Thread-27258::INFO::2012-12-02 14:20:51,894::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27258::DEBUG::2012-12-02 14:20:51,895::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7799e97a-56c4-43d8-a32e-34bd39229a04`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27258::DEBUG::2012-12-02 14:20:51,895::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27258::DEBUG::2012-12-02 14:20:51,895::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27258::DEBUG::2012-12-02 14:20:51,895::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7799e97a-56c4-43d8-a32e-34bd39229a04`::Granted request >Thread-27258::DEBUG::2012-12-02 14:20:51,895::task::794::TaskManager.Task::(resourceAcquired) Task=`544e89a1-9d56-4ff8-b19c-bc1060b38a08`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27258::DEBUG::2012-12-02 14:20:51,896::task::957::TaskManager.Task::(_decref) Task=`544e89a1-9d56-4ff8-b19c-bc1060b38a08`::ref 1 aborting False >Thread-27258::INFO::2012-12-02 14:20:51,900::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27258::DEBUG::2012-12-02 14:20:51,900::task::1151::TaskManager.Task::(prepare) Task=`544e89a1-9d56-4ff8-b19c-bc1060b38a08`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27258::DEBUG::2012-12-02 14:20:51,900::task::568::TaskManager.Task::(_updateState) Task=`544e89a1-9d56-4ff8-b19c-bc1060b38a08`::moving from state preparing -> state finished >Thread-27258::DEBUG::2012-12-02 14:20:51,900::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27258::DEBUG::2012-12-02 14:20:51,900::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27258::DEBUG::2012-12-02 14:20:51,901::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27258::DEBUG::2012-12-02 14:20:51,901::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27258::DEBUG::2012-12-02 14:20:51,901::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27258::DEBUG::2012-12-02 14:20:51,901::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27258::DEBUG::2012-12-02 14:20:51,901::task::957::TaskManager.Task::(_decref) Task=`544e89a1-9d56-4ff8-b19c-bc1060b38a08`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:20:54,392::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:20:54,392::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:20:54,392::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:20:54,598::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:20:54,599::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:20:54,600::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:20:54,636::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:20:54,636::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:20:54,638::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:20:54,641::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:20:54,641::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:20:54,641::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-11821::DEBUG::2012-12-02 14:20:56,532::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:20:56,532::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:20:56,533::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:20:56,737::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:20:56,739::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:20:56,739::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:20:56,775::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:20:56,775::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:20:56,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:20:56,787::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:20:56,788::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:20:56,788::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-27384::DEBUG::2012-12-02 14:20:58,898::task::568::TaskManager.Task::(_updateState) Task=`3b56405b-2a6e-4586-9c4e-4cc7061501cf`::moving from state init -> state preparing >Thread-27384::INFO::2012-12-02 14:20:58,898::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27384::INFO::2012-12-02 14:20:58,898::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00631499290466', 'lastCheck': 1354450856.803298, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0014591217041', 'lastCheck': 1354450851.7697051, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00179696083069', 'lastCheck': 1354450854.64974, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00680804252625', 'lastCheck': 1354450851.838309, 'code': 0, 'valid': True}} >Thread-27384::DEBUG::2012-12-02 14:20:58,899::task::1151::TaskManager.Task::(prepare) Task=`3b56405b-2a6e-4586-9c4e-4cc7061501cf`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00631499290466', 'lastCheck': 1354450856.803298, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0014591217041', 'lastCheck': 1354450851.7697051, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00179696083069', 'lastCheck': 1354450854.64974, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00680804252625', 'lastCheck': 1354450851.838309, 'code': 0, 'valid': True}} >Thread-27384::DEBUG::2012-12-02 14:20:58,899::task::568::TaskManager.Task::(_updateState) Task=`3b56405b-2a6e-4586-9c4e-4cc7061501cf`::moving from state preparing -> state finished >Thread-27384::DEBUG::2012-12-02 14:20:58,899::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27384::DEBUG::2012-12-02 14:20:58,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27384::DEBUG::2012-12-02 14:20:58,899::task::957::TaskManager.Task::(_decref) Task=`3b56405b-2a6e-4586-9c4e-4cc7061501cf`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:21:01,769::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:21:01,770::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:21:01,770::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:21:01,977::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:21:01,978::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:21:01,979::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:21:02,013::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:21:02,014::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:21:02,017::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:21:02,020::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:21:02,021::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:21:02,021::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:21:02,021::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:21:02,021::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:21:02,023::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-27448::DEBUG::2012-12-02 14:21:02,063::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27448::DEBUG::2012-12-02 14:21:02,063::task::568::TaskManager.Task::(_updateState) Task=`34d204b6-c753-4587-bc55-c1f671f53ebe`::moving from state init -> state preparing >Thread-27448::INFO::2012-12-02 14:21:02,063::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-11946::DEBUG::2012-12-02 14:21:02,228::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:21:02,230::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:21:02,230::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:21:02,264::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:21:02,264::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:21:02,273::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:21:02,276::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:21:02,276::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:21:02,276::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-27448::INFO::2012-12-02 14:21:02,277::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27448::DEBUG::2012-12-02 14:21:02,278::task::1151::TaskManager.Task::(prepare) Task=`34d204b6-c753-4587-bc55-c1f671f53ebe`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27448::DEBUG::2012-12-02 14:21:02,280::task::568::TaskManager.Task::(_updateState) Task=`34d204b6-c753-4587-bc55-c1f671f53ebe`::moving from state preparing -> state finished >Thread-27448::DEBUG::2012-12-02 14:21:02,282::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27448::DEBUG::2012-12-02 14:21:02,282::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27448::DEBUG::2012-12-02 14:21:02,285::task::957::TaskManager.Task::(_decref) Task=`34d204b6-c753-4587-bc55-c1f671f53ebe`::ref 0 aborting False >Thread-27510::DEBUG::2012-12-02 14:21:02,289::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27510::DEBUG::2012-12-02 14:21:02,289::task::568::TaskManager.Task::(_updateState) Task=`97b39a25-d30b-4933-a45c-ce74ed62a022`::moving from state init -> state preparing >Thread-27510::INFO::2012-12-02 14:21:02,290::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27510::DEBUG::2012-12-02 14:21:02,290::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f5b957d3-a8c5-498b-8032-daabd40c4b69`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27510::DEBUG::2012-12-02 14:21:02,290::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27510::DEBUG::2012-12-02 14:21:02,290::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27510::DEBUG::2012-12-02 14:21:02,290::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f5b957d3-a8c5-498b-8032-daabd40c4b69`::Granted request >Thread-27510::DEBUG::2012-12-02 14:21:02,291::task::794::TaskManager.Task::(resourceAcquired) Task=`97b39a25-d30b-4933-a45c-ce74ed62a022`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27510::DEBUG::2012-12-02 14:21:02,291::task::957::TaskManager.Task::(_decref) Task=`97b39a25-d30b-4933-a45c-ce74ed62a022`::ref 1 aborting False >Thread-27510::INFO::2012-12-02 14:21:02,298::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27510::DEBUG::2012-12-02 14:21:02,298::task::1151::TaskManager.Task::(prepare) Task=`97b39a25-d30b-4933-a45c-ce74ed62a022`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27510::DEBUG::2012-12-02 14:21:02,298::task::568::TaskManager.Task::(_updateState) Task=`97b39a25-d30b-4933-a45c-ce74ed62a022`::moving from state preparing -> state finished >Thread-27510::DEBUG::2012-12-02 14:21:02,299::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27510::DEBUG::2012-12-02 14:21:02,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27510::DEBUG::2012-12-02 14:21:02,299::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27510::DEBUG::2012-12-02 14:21:02,299::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27510::DEBUG::2012-12-02 14:21:02,299::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27510::DEBUG::2012-12-02 14:21:02,299::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27510::DEBUG::2012-12-02 14:21:02,299::task::957::TaskManager.Task::(_decref) Task=`97b39a25-d30b-4933-a45c-ce74ed62a022`::ref 0 aborting False >Thread-27514::DEBUG::2012-12-02 14:21:09,038::task::568::TaskManager.Task::(_updateState) Task=`0847ed20-8f48-487d-b6ad-ebc5699cf97e`::moving from state init -> state preparing >Thread-27514::INFO::2012-12-02 14:21:09,038::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27514::INFO::2012-12-02 14:21:09,038::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0072329044342', 'lastCheck': 1354450866.8197269, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00328397750854', 'lastCheck': 1354450862.288491, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00198101997375', 'lastCheck': 1354450864.6603899, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00719594955444', 'lastCheck': 1354450862.2984281, 'code': 0, 'valid': True}} >Thread-27514::DEBUG::2012-12-02 14:21:09,038::task::1151::TaskManager.Task::(prepare) Task=`0847ed20-8f48-487d-b6ad-ebc5699cf97e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0072329044342', 'lastCheck': 1354450866.8197269, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00328397750854', 'lastCheck': 1354450862.288491, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00198101997375', 'lastCheck': 1354450864.6603899, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00719594955444', 'lastCheck': 1354450862.2984281, 'code': 0, 'valid': True}} >Thread-27514::DEBUG::2012-12-02 14:21:09,038::task::568::TaskManager.Task::(_updateState) Task=`0847ed20-8f48-487d-b6ad-ebc5699cf97e`::moving from state preparing -> state finished >Thread-27514::DEBUG::2012-12-02 14:21:09,038::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27514::DEBUG::2012-12-02 14:21:09,039::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27514::DEBUG::2012-12-02 14:21:09,039::task::957::TaskManager.Task::(_decref) Task=`0847ed20-8f48-487d-b6ad-ebc5699cf97e`::ref 0 aborting False >Thread-27517::DEBUG::2012-12-02 14:21:12,423::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27517::DEBUG::2012-12-02 14:21:12,423::task::568::TaskManager.Task::(_updateState) Task=`3b490f35-8b39-492c-8d7d-0fe73570153c`::moving from state init -> state preparing >Thread-27517::INFO::2012-12-02 14:21:12,424::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27517::INFO::2012-12-02 14:21:12,424::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27517::DEBUG::2012-12-02 14:21:12,424::task::1151::TaskManager.Task::(prepare) Task=`3b490f35-8b39-492c-8d7d-0fe73570153c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27517::DEBUG::2012-12-02 14:21:12,424::task::568::TaskManager.Task::(_updateState) Task=`3b490f35-8b39-492c-8d7d-0fe73570153c`::moving from state preparing -> state finished >Thread-27517::DEBUG::2012-12-02 14:21:12,424::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27517::DEBUG::2012-12-02 14:21:12,424::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27517::DEBUG::2012-12-02 14:21:12,424::task::957::TaskManager.Task::(_decref) Task=`3b490f35-8b39-492c-8d7d-0fe73570153c`::ref 0 aborting False >Thread-27518::DEBUG::2012-12-02 14:21:12,427::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27518::DEBUG::2012-12-02 14:21:12,427::task::568::TaskManager.Task::(_updateState) Task=`50c2b56a-93d7-4a0f-b319-d338d3ff0021`::moving from state init -> state preparing >Thread-27518::INFO::2012-12-02 14:21:12,427::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27518::DEBUG::2012-12-02 14:21:12,428::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`028cb279-6088-477a-99fe-c477e5591b9c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27518::DEBUG::2012-12-02 14:21:12,428::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27518::DEBUG::2012-12-02 14:21:12,428::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27518::DEBUG::2012-12-02 14:21:12,428::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`028cb279-6088-477a-99fe-c477e5591b9c`::Granted request >Thread-27518::DEBUG::2012-12-02 14:21:12,428::task::794::TaskManager.Task::(resourceAcquired) Task=`50c2b56a-93d7-4a0f-b319-d338d3ff0021`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27518::DEBUG::2012-12-02 14:21:12,428::task::957::TaskManager.Task::(_decref) Task=`50c2b56a-93d7-4a0f-b319-d338d3ff0021`::ref 1 aborting False >Thread-27518::INFO::2012-12-02 14:21:12,435::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27518::DEBUG::2012-12-02 14:21:12,435::task::1151::TaskManager.Task::(prepare) Task=`50c2b56a-93d7-4a0f-b319-d338d3ff0021`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27518::DEBUG::2012-12-02 14:21:12,435::task::568::TaskManager.Task::(_updateState) Task=`50c2b56a-93d7-4a0f-b319-d338d3ff0021`::moving from state preparing -> state finished >Thread-27518::DEBUG::2012-12-02 14:21:12,435::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27518::DEBUG::2012-12-02 14:21:12,435::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27518::DEBUG::2012-12-02 14:21:12,435::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27518::DEBUG::2012-12-02 14:21:12,436::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27518::DEBUG::2012-12-02 14:21:12,436::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27518::DEBUG::2012-12-02 14:21:12,436::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27518::DEBUG::2012-12-02 14:21:12,436::task::957::TaskManager.Task::(_decref) Task=`50c2b56a-93d7-4a0f-b319-d338d3ff0021`::ref 0 aborting False >Thread-27522::DEBUG::2012-12-02 14:21:19,112::task::568::TaskManager.Task::(_updateState) Task=`eec0d39c-0679-45a3-be92-56ea70675db3`::moving from state init -> state preparing >Thread-27522::INFO::2012-12-02 14:21:19,112::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27522::INFO::2012-12-02 14:21:19,112::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00885200500488', 'lastCheck': 1354450876.8382199, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206398963928', 'lastCheck': 1354450872.307337, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00200605392456', 'lastCheck': 1354450874.6722269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00834894180298', 'lastCheck': 1354450872.3180699, 'code': 0, 'valid': True}} >Thread-27522::DEBUG::2012-12-02 14:21:19,112::task::1151::TaskManager.Task::(prepare) Task=`eec0d39c-0679-45a3-be92-56ea70675db3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00885200500488', 'lastCheck': 1354450876.8382199, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206398963928', 'lastCheck': 1354450872.307337, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00200605392456', 'lastCheck': 1354450874.6722269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00834894180298', 'lastCheck': 1354450872.3180699, 'code': 0, 'valid': True}} >Thread-27522::DEBUG::2012-12-02 14:21:19,113::task::568::TaskManager.Task::(_updateState) Task=`eec0d39c-0679-45a3-be92-56ea70675db3`::moving from state preparing -> state finished >Thread-27522::DEBUG::2012-12-02 14:21:19,113::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27522::DEBUG::2012-12-02 14:21:19,113::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27522::DEBUG::2012-12-02 14:21:19,113::task::957::TaskManager.Task::(_decref) Task=`eec0d39c-0679-45a3-be92-56ea70675db3`::ref 0 aborting False >Thread-27525::DEBUG::2012-12-02 14:21:22,562::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27525::DEBUG::2012-12-02 14:21:22,562::task::568::TaskManager.Task::(_updateState) Task=`1e513185-6435-48f3-9911-c13ce3fa6996`::moving from state init -> state preparing >Thread-27525::INFO::2012-12-02 14:21:22,563::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27525::INFO::2012-12-02 14:21:22,563::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27525::DEBUG::2012-12-02 14:21:22,563::task::1151::TaskManager.Task::(prepare) Task=`1e513185-6435-48f3-9911-c13ce3fa6996`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27525::DEBUG::2012-12-02 14:21:22,563::task::568::TaskManager.Task::(_updateState) Task=`1e513185-6435-48f3-9911-c13ce3fa6996`::moving from state preparing -> state finished >Thread-27525::DEBUG::2012-12-02 14:21:22,563::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27525::DEBUG::2012-12-02 14:21:22,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27525::DEBUG::2012-12-02 14:21:22,564::task::957::TaskManager.Task::(_decref) Task=`1e513185-6435-48f3-9911-c13ce3fa6996`::ref 0 aborting False >Thread-27526::DEBUG::2012-12-02 14:21:22,566::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27526::DEBUG::2012-12-02 14:21:22,566::task::568::TaskManager.Task::(_updateState) Task=`6f2bdafc-2a1d-4f21-b87a-e5cfa2aa139b`::moving from state init -> state preparing >Thread-27526::INFO::2012-12-02 14:21:22,566::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27526::DEBUG::2012-12-02 14:21:22,567::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ded938bd-8462-43ea-ae60-0f8b6602ec4f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27526::DEBUG::2012-12-02 14:21:22,567::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27526::DEBUG::2012-12-02 14:21:22,567::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27526::DEBUG::2012-12-02 14:21:22,567::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ded938bd-8462-43ea-ae60-0f8b6602ec4f`::Granted request >Thread-27526::DEBUG::2012-12-02 14:21:22,567::task::794::TaskManager.Task::(resourceAcquired) Task=`6f2bdafc-2a1d-4f21-b87a-e5cfa2aa139b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27526::DEBUG::2012-12-02 14:21:22,568::task::957::TaskManager.Task::(_decref) Task=`6f2bdafc-2a1d-4f21-b87a-e5cfa2aa139b`::ref 1 aborting False >Thread-27526::INFO::2012-12-02 14:21:22,574::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27526::DEBUG::2012-12-02 14:21:22,574::task::1151::TaskManager.Task::(prepare) Task=`6f2bdafc-2a1d-4f21-b87a-e5cfa2aa139b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27526::DEBUG::2012-12-02 14:21:22,574::task::568::TaskManager.Task::(_updateState) Task=`6f2bdafc-2a1d-4f21-b87a-e5cfa2aa139b`::moving from state preparing -> state finished >Thread-27526::DEBUG::2012-12-02 14:21:22,574::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27526::DEBUG::2012-12-02 14:21:22,575::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27526::DEBUG::2012-12-02 14:21:22,575::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27526::DEBUG::2012-12-02 14:21:22,575::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27526::DEBUG::2012-12-02 14:21:22,575::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27526::DEBUG::2012-12-02 14:21:22,575::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27526::DEBUG::2012-12-02 14:21:22,575::task::957::TaskManager.Task::(_decref) Task=`6f2bdafc-2a1d-4f21-b87a-e5cfa2aa139b`::ref 0 aborting False >Thread-27530::DEBUG::2012-12-02 14:21:29,178::task::568::TaskManager.Task::(_updateState) Task=`b76085d1-a741-4e71-9078-fc5acf9eaa35`::moving from state init -> state preparing >Thread-27530::INFO::2012-12-02 14:21:29,178::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27530::INFO::2012-12-02 14:21:29,178::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00787496566772', 'lastCheck': 1354450886.8558509, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00420188903809', 'lastCheck': 1354450882.3311651, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00175619125366', 'lastCheck': 1354450884.682102, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00806903839111', 'lastCheck': 1354450882.338764, 'code': 0, 'valid': True}} >Thread-27530::DEBUG::2012-12-02 14:21:29,178::task::1151::TaskManager.Task::(prepare) Task=`b76085d1-a741-4e71-9078-fc5acf9eaa35`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00787496566772', 'lastCheck': 1354450886.8558509, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00420188903809', 'lastCheck': 1354450882.3311651, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00175619125366', 'lastCheck': 1354450884.682102, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00806903839111', 'lastCheck': 1354450882.338764, 'code': 0, 'valid': True}} >Thread-27530::DEBUG::2012-12-02 14:21:29,178::task::568::TaskManager.Task::(_updateState) Task=`b76085d1-a741-4e71-9078-fc5acf9eaa35`::moving from state preparing -> state finished >Thread-27530::DEBUG::2012-12-02 14:21:29,178::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27530::DEBUG::2012-12-02 14:21:29,179::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27530::DEBUG::2012-12-02 14:21:29,179::task::957::TaskManager.Task::(_decref) Task=`b76085d1-a741-4e71-9078-fc5acf9eaa35`::ref 0 aborting False >Thread-27533::DEBUG::2012-12-02 14:21:32,707::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27533::DEBUG::2012-12-02 14:21:32,708::task::568::TaskManager.Task::(_updateState) Task=`42a2707e-8fe6-43d4-aa31-bb673b2b5f80`::moving from state init -> state preparing >Thread-27533::INFO::2012-12-02 14:21:32,708::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27533::INFO::2012-12-02 14:21:32,708::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27533::DEBUG::2012-12-02 14:21:32,708::task::1151::TaskManager.Task::(prepare) Task=`42a2707e-8fe6-43d4-aa31-bb673b2b5f80`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27533::DEBUG::2012-12-02 14:21:32,709::task::568::TaskManager.Task::(_updateState) Task=`42a2707e-8fe6-43d4-aa31-bb673b2b5f80`::moving from state preparing -> state finished >Thread-27533::DEBUG::2012-12-02 14:21:32,709::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27533::DEBUG::2012-12-02 14:21:32,709::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27533::DEBUG::2012-12-02 14:21:32,709::task::957::TaskManager.Task::(_decref) Task=`42a2707e-8fe6-43d4-aa31-bb673b2b5f80`::ref 0 aborting False >Thread-27534::DEBUG::2012-12-02 14:21:32,712::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27534::DEBUG::2012-12-02 14:21:32,712::task::568::TaskManager.Task::(_updateState) Task=`2a59ffd9-40d9-4e7e-8601-48adee0e1396`::moving from state init -> state preparing >Thread-27534::INFO::2012-12-02 14:21:32,712::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27534::DEBUG::2012-12-02 14:21:32,712::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b64d08af-3d75-4ac0-9294-15740ab94bdd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27534::DEBUG::2012-12-02 14:21:32,712::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27534::DEBUG::2012-12-02 14:21:32,713::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27534::DEBUG::2012-12-02 14:21:32,713::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b64d08af-3d75-4ac0-9294-15740ab94bdd`::Granted request >Thread-27534::DEBUG::2012-12-02 14:21:32,713::task::794::TaskManager.Task::(resourceAcquired) Task=`2a59ffd9-40d9-4e7e-8601-48adee0e1396`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27534::DEBUG::2012-12-02 14:21:32,713::task::957::TaskManager.Task::(_decref) Task=`2a59ffd9-40d9-4e7e-8601-48adee0e1396`::ref 1 aborting False >Thread-27534::INFO::2012-12-02 14:21:32,718::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27534::DEBUG::2012-12-02 14:21:32,718::task::1151::TaskManager.Task::(prepare) Task=`2a59ffd9-40d9-4e7e-8601-48adee0e1396`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27534::DEBUG::2012-12-02 14:21:32,718::task::568::TaskManager.Task::(_updateState) Task=`2a59ffd9-40d9-4e7e-8601-48adee0e1396`::moving from state preparing -> state finished >Thread-27534::DEBUG::2012-12-02 14:21:32,718::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27534::DEBUG::2012-12-02 14:21:32,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27534::DEBUG::2012-12-02 14:21:32,719::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27534::DEBUG::2012-12-02 14:21:32,719::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27534::DEBUG::2012-12-02 14:21:32,719::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27534::DEBUG::2012-12-02 14:21:32,719::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27534::DEBUG::2012-12-02 14:21:32,719::task::957::TaskManager.Task::(_decref) Task=`2a59ffd9-40d9-4e7e-8601-48adee0e1396`::ref 0 aborting False >Thread-27538::DEBUG::2012-12-02 14:21:39,243::task::568::TaskManager.Task::(_updateState) Task=`58ef5575-a262-44be-b3eb-e039c6f4177a`::moving from state init -> state preparing >Thread-27538::INFO::2012-12-02 14:21:39,243::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27538::INFO::2012-12-02 14:21:39,243::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00835990905762', 'lastCheck': 1354450896.876281, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00492811203003', 'lastCheck': 1354450892.348388, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00254797935486', 'lastCheck': 1354450894.6935959, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00760316848755', 'lastCheck': 1354450892.357255, 'code': 0, 'valid': True}} >Thread-27538::DEBUG::2012-12-02 14:21:39,243::task::1151::TaskManager.Task::(prepare) Task=`58ef5575-a262-44be-b3eb-e039c6f4177a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00835990905762', 'lastCheck': 1354450896.876281, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00492811203003', 'lastCheck': 1354450892.348388, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00254797935486', 'lastCheck': 1354450894.6935959, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00760316848755', 'lastCheck': 1354450892.357255, 'code': 0, 'valid': True}} >Thread-27538::DEBUG::2012-12-02 14:21:39,244::task::568::TaskManager.Task::(_updateState) Task=`58ef5575-a262-44be-b3eb-e039c6f4177a`::moving from state preparing -> state finished >Thread-27538::DEBUG::2012-12-02 14:21:39,244::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27538::DEBUG::2012-12-02 14:21:39,244::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27538::DEBUG::2012-12-02 14:21:39,244::task::957::TaskManager.Task::(_decref) Task=`58ef5575-a262-44be-b3eb-e039c6f4177a`::ref 0 aborting False >Thread-27541::DEBUG::2012-12-02 14:21:42,846::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27541::DEBUG::2012-12-02 14:21:42,847::task::568::TaskManager.Task::(_updateState) Task=`edf28704-16aa-4524-8ff7-309e64f0a64e`::moving from state init -> state preparing >Thread-27541::INFO::2012-12-02 14:21:42,847::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27541::INFO::2012-12-02 14:21:42,847::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27541::DEBUG::2012-12-02 14:21:42,847::task::1151::TaskManager.Task::(prepare) Task=`edf28704-16aa-4524-8ff7-309e64f0a64e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27541::DEBUG::2012-12-02 14:21:42,847::task::568::TaskManager.Task::(_updateState) Task=`edf28704-16aa-4524-8ff7-309e64f0a64e`::moving from state preparing -> state finished >Thread-27541::DEBUG::2012-12-02 14:21:42,848::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27541::DEBUG::2012-12-02 14:21:42,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27541::DEBUG::2012-12-02 14:21:42,848::task::957::TaskManager.Task::(_decref) Task=`edf28704-16aa-4524-8ff7-309e64f0a64e`::ref 0 aborting False >Thread-27542::DEBUG::2012-12-02 14:21:42,850::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27542::DEBUG::2012-12-02 14:21:42,851::task::568::TaskManager.Task::(_updateState) Task=`8e89b924-0cf9-492c-b542-66ca07e0848f`::moving from state init -> state preparing >Thread-27542::INFO::2012-12-02 14:21:42,851::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27542::DEBUG::2012-12-02 14:21:42,851::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0d753f19-04c9-44c8-91f2-1595dceca040`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27542::DEBUG::2012-12-02 14:21:42,851::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27542::DEBUG::2012-12-02 14:21:42,851::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27542::DEBUG::2012-12-02 14:21:42,851::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0d753f19-04c9-44c8-91f2-1595dceca040`::Granted request >Thread-27542::DEBUG::2012-12-02 14:21:42,852::task::794::TaskManager.Task::(resourceAcquired) Task=`8e89b924-0cf9-492c-b542-66ca07e0848f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27542::DEBUG::2012-12-02 14:21:42,852::task::957::TaskManager.Task::(_decref) Task=`8e89b924-0cf9-492c-b542-66ca07e0848f`::ref 1 aborting False >Thread-27542::INFO::2012-12-02 14:21:42,858::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27542::DEBUG::2012-12-02 14:21:42,858::task::1151::TaskManager.Task::(prepare) Task=`8e89b924-0cf9-492c-b542-66ca07e0848f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27542::DEBUG::2012-12-02 14:21:42,858::task::568::TaskManager.Task::(_updateState) Task=`8e89b924-0cf9-492c-b542-66ca07e0848f`::moving from state preparing -> state finished >Thread-27542::DEBUG::2012-12-02 14:21:42,858::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27542::DEBUG::2012-12-02 14:21:42,858::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27542::DEBUG::2012-12-02 14:21:42,858::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27542::DEBUG::2012-12-02 14:21:42,858::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27542::DEBUG::2012-12-02 14:21:42,858::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27542::DEBUG::2012-12-02 14:21:42,858::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27542::DEBUG::2012-12-02 14:21:42,858::task::957::TaskManager.Task::(_decref) Task=`8e89b924-0cf9-492c-b542-66ca07e0848f`::ref 0 aborting False >Thread-27546::DEBUG::2012-12-02 14:21:49,306::task::568::TaskManager.Task::(_updateState) Task=`f795634a-a9b6-4e48-9898-88c6c608556d`::moving from state init -> state preparing >Thread-27546::INFO::2012-12-02 14:21:49,306::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27546::INFO::2012-12-02 14:21:49,307::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00875186920166', 'lastCheck': 1354450906.893822, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00290513038635', 'lastCheck': 1354450902.3696539, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00165510177612', 'lastCheck': 1354450904.702476, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00785613059998', 'lastCheck': 1354450902.3797691, 'code': 0, 'valid': True}} >Thread-27546::DEBUG::2012-12-02 14:21:49,307::task::1151::TaskManager.Task::(prepare) Task=`f795634a-a9b6-4e48-9898-88c6c608556d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00875186920166', 'lastCheck': 1354450906.893822, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00290513038635', 'lastCheck': 1354450902.3696539, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00165510177612', 'lastCheck': 1354450904.702476, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00785613059998', 'lastCheck': 1354450902.3797691, 'code': 0, 'valid': True}} >Thread-27546::DEBUG::2012-12-02 14:21:49,307::task::568::TaskManager.Task::(_updateState) Task=`f795634a-a9b6-4e48-9898-88c6c608556d`::moving from state preparing -> state finished >Thread-27546::DEBUG::2012-12-02 14:21:49,307::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27546::DEBUG::2012-12-02 14:21:49,307::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27546::DEBUG::2012-12-02 14:21:49,307::task::957::TaskManager.Task::(_decref) Task=`f795634a-a9b6-4e48-9898-88c6c608556d`::ref 0 aborting False >Thread-27549::DEBUG::2012-12-02 14:21:52,993::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27549::DEBUG::2012-12-02 14:21:52,994::task::568::TaskManager.Task::(_updateState) Task=`93dca271-07fc-4ed2-a106-7948d19fff0a`::moving from state init -> state preparing >Thread-27549::INFO::2012-12-02 14:21:52,994::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27549::INFO::2012-12-02 14:21:52,994::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27549::DEBUG::2012-12-02 14:21:52,994::task::1151::TaskManager.Task::(prepare) Task=`93dca271-07fc-4ed2-a106-7948d19fff0a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27549::DEBUG::2012-12-02 14:21:52,995::task::568::TaskManager.Task::(_updateState) Task=`93dca271-07fc-4ed2-a106-7948d19fff0a`::moving from state preparing -> state finished >Thread-27549::DEBUG::2012-12-02 14:21:52,995::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27549::DEBUG::2012-12-02 14:21:52,995::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27549::DEBUG::2012-12-02 14:21:52,995::task::957::TaskManager.Task::(_decref) Task=`93dca271-07fc-4ed2-a106-7948d19fff0a`::ref 0 aborting False >Thread-27550::DEBUG::2012-12-02 14:21:52,997::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27550::DEBUG::2012-12-02 14:21:52,998::task::568::TaskManager.Task::(_updateState) Task=`8044ebea-f9d4-4ca1-9121-12442f842924`::moving from state init -> state preparing >Thread-27550::INFO::2012-12-02 14:21:52,998::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27550::DEBUG::2012-12-02 14:21:52,998::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`eb2af9ae-6945-4abd-ba61-9aa598ad5fc7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27550::DEBUG::2012-12-02 14:21:52,998::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27550::DEBUG::2012-12-02 14:21:52,998::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27550::DEBUG::2012-12-02 14:21:52,998::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`eb2af9ae-6945-4abd-ba61-9aa598ad5fc7`::Granted request >Thread-27550::DEBUG::2012-12-02 14:21:52,999::task::794::TaskManager.Task::(resourceAcquired) Task=`8044ebea-f9d4-4ca1-9121-12442f842924`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27550::DEBUG::2012-12-02 14:21:52,999::task::957::TaskManager.Task::(_decref) Task=`8044ebea-f9d4-4ca1-9121-12442f842924`::ref 1 aborting False >Thread-27550::INFO::2012-12-02 14:21:53,003::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27550::DEBUG::2012-12-02 14:21:53,003::task::1151::TaskManager.Task::(prepare) Task=`8044ebea-f9d4-4ca1-9121-12442f842924`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27550::DEBUG::2012-12-02 14:21:53,003::task::568::TaskManager.Task::(_updateState) Task=`8044ebea-f9d4-4ca1-9121-12442f842924`::moving from state preparing -> state finished >Thread-27550::DEBUG::2012-12-02 14:21:53,003::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27550::DEBUG::2012-12-02 14:21:53,003::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27550::DEBUG::2012-12-02 14:21:53,003::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27550::DEBUG::2012-12-02 14:21:53,003::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27550::DEBUG::2012-12-02 14:21:53,004::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27550::DEBUG::2012-12-02 14:21:53,004::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27550::DEBUG::2012-12-02 14:21:53,004::task::957::TaskManager.Task::(_decref) Task=`8044ebea-f9d4-4ca1-9121-12442f842924`::ref 0 aborting False >Thread-27554::DEBUG::2012-12-02 14:21:59,370::task::568::TaskManager.Task::(_updateState) Task=`8d71d136-b0cc-44b9-b807-eefd514a7cd6`::moving from state init -> state preparing >Thread-27554::INFO::2012-12-02 14:21:59,370::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27554::INFO::2012-12-02 14:21:59,371::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00987410545349', 'lastCheck': 1354450916.9134221, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00144600868225', 'lastCheck': 1354450912.3805931, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178718566895', 'lastCheck': 1354450914.712327, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.003338098526', 'lastCheck': 1354450912.391629, 'code': 0, 'valid': True}} >Thread-27554::DEBUG::2012-12-02 14:21:59,371::task::1151::TaskManager.Task::(prepare) Task=`8d71d136-b0cc-44b9-b807-eefd514a7cd6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00987410545349', 'lastCheck': 1354450916.9134221, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00144600868225', 'lastCheck': 1354450912.3805931, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178718566895', 'lastCheck': 1354450914.712327, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.003338098526', 'lastCheck': 1354450912.391629, 'code': 0, 'valid': True}} >Thread-27554::DEBUG::2012-12-02 14:21:59,371::task::568::TaskManager.Task::(_updateState) Task=`8d71d136-b0cc-44b9-b807-eefd514a7cd6`::moving from state preparing -> state finished >Thread-27554::DEBUG::2012-12-02 14:21:59,371::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27554::DEBUG::2012-12-02 14:21:59,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27554::DEBUG::2012-12-02 14:21:59,371::task::957::TaskManager.Task::(_decref) Task=`8d71d136-b0cc-44b9-b807-eefd514a7cd6`::ref 0 aborting False >Thread-27557::DEBUG::2012-12-02 14:22:03,140::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27557::DEBUG::2012-12-02 14:22:03,141::task::568::TaskManager.Task::(_updateState) Task=`f78c0406-e7d8-4b36-942c-d73ea9676f25`::moving from state init -> state preparing >Thread-27557::INFO::2012-12-02 14:22:03,141::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27557::INFO::2012-12-02 14:22:03,141::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27557::DEBUG::2012-12-02 14:22:03,141::task::1151::TaskManager.Task::(prepare) Task=`f78c0406-e7d8-4b36-942c-d73ea9676f25`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27557::DEBUG::2012-12-02 14:22:03,141::task::568::TaskManager.Task::(_updateState) Task=`f78c0406-e7d8-4b36-942c-d73ea9676f25`::moving from state preparing -> state finished >Thread-27557::DEBUG::2012-12-02 14:22:03,141::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27557::DEBUG::2012-12-02 14:22:03,141::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27557::DEBUG::2012-12-02 14:22:03,142::task::957::TaskManager.Task::(_decref) Task=`f78c0406-e7d8-4b36-942c-d73ea9676f25`::ref 0 aborting False >Thread-27558::DEBUG::2012-12-02 14:22:03,144::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27558::DEBUG::2012-12-02 14:22:03,144::task::568::TaskManager.Task::(_updateState) Task=`8297cc27-b084-4711-9cb1-d71832f851cf`::moving from state init -> state preparing >Thread-27558::INFO::2012-12-02 14:22:03,144::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27558::DEBUG::2012-12-02 14:22:03,145::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0cd50535-ee4b-43ac-a805-4053498928d8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27558::DEBUG::2012-12-02 14:22:03,145::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27558::DEBUG::2012-12-02 14:22:03,145::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27558::DEBUG::2012-12-02 14:22:03,145::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0cd50535-ee4b-43ac-a805-4053498928d8`::Granted request >Thread-27558::DEBUG::2012-12-02 14:22:03,146::task::794::TaskManager.Task::(resourceAcquired) Task=`8297cc27-b084-4711-9cb1-d71832f851cf`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27558::DEBUG::2012-12-02 14:22:03,146::task::957::TaskManager.Task::(_decref) Task=`8297cc27-b084-4711-9cb1-d71832f851cf`::ref 1 aborting False >Thread-27558::INFO::2012-12-02 14:22:03,150::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27558::DEBUG::2012-12-02 14:22:03,150::task::1151::TaskManager.Task::(prepare) Task=`8297cc27-b084-4711-9cb1-d71832f851cf`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27558::DEBUG::2012-12-02 14:22:03,151::task::568::TaskManager.Task::(_updateState) Task=`8297cc27-b084-4711-9cb1-d71832f851cf`::moving from state preparing -> state finished >Thread-27558::DEBUG::2012-12-02 14:22:03,151::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27558::DEBUG::2012-12-02 14:22:03,151::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27558::DEBUG::2012-12-02 14:22:03,151::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27558::DEBUG::2012-12-02 14:22:03,151::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27558::DEBUG::2012-12-02 14:22:03,151::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27558::DEBUG::2012-12-02 14:22:03,151::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27558::DEBUG::2012-12-02 14:22:03,151::task::957::TaskManager.Task::(_decref) Task=`8297cc27-b084-4711-9cb1-d71832f851cf`::ref 0 aborting False >Thread-27562::DEBUG::2012-12-02 14:22:09,434::task::568::TaskManager.Task::(_updateState) Task=`df1c9617-8794-4520-8850-ec71a53024a7`::moving from state init -> state preparing >Thread-27562::INFO::2012-12-02 14:22:09,434::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27562::INFO::2012-12-02 14:22:09,434::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00773787498474', 'lastCheck': 1354450926.930521, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00445604324341', 'lastCheck': 1354450922.3989589, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00170302391052', 'lastCheck': 1354450924.721787, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0105919837952', 'lastCheck': 1354450922.41224, 'code': 0, 'valid': True}} >Thread-27562::DEBUG::2012-12-02 14:22:09,434::task::1151::TaskManager.Task::(prepare) Task=`df1c9617-8794-4520-8850-ec71a53024a7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00773787498474', 'lastCheck': 1354450926.930521, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00445604324341', 'lastCheck': 1354450922.3989589, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00170302391052', 'lastCheck': 1354450924.721787, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0105919837952', 'lastCheck': 1354450922.41224, 'code': 0, 'valid': True}} >Thread-27562::DEBUG::2012-12-02 14:22:09,434::task::568::TaskManager.Task::(_updateState) Task=`df1c9617-8794-4520-8850-ec71a53024a7`::moving from state preparing -> state finished >Thread-27562::DEBUG::2012-12-02 14:22:09,435::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27562::DEBUG::2012-12-02 14:22:09,435::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27562::DEBUG::2012-12-02 14:22:09,435::task::957::TaskManager.Task::(_decref) Task=`df1c9617-8794-4520-8850-ec71a53024a7`::ref 0 aborting False >Thread-27565::DEBUG::2012-12-02 14:22:13,285::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27565::DEBUG::2012-12-02 14:22:13,286::task::568::TaskManager.Task::(_updateState) Task=`7b0473f9-286b-42eb-a4aa-dcdc78ac49b8`::moving from state init -> state preparing >Thread-27565::INFO::2012-12-02 14:22:13,286::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27565::INFO::2012-12-02 14:22:13,286::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27565::DEBUG::2012-12-02 14:22:13,286::task::1151::TaskManager.Task::(prepare) Task=`7b0473f9-286b-42eb-a4aa-dcdc78ac49b8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27565::DEBUG::2012-12-02 14:22:13,286::task::568::TaskManager.Task::(_updateState) Task=`7b0473f9-286b-42eb-a4aa-dcdc78ac49b8`::moving from state preparing -> state finished >Thread-27565::DEBUG::2012-12-02 14:22:13,287::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27565::DEBUG::2012-12-02 14:22:13,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27565::DEBUG::2012-12-02 14:22:13,287::task::957::TaskManager.Task::(_decref) Task=`7b0473f9-286b-42eb-a4aa-dcdc78ac49b8`::ref 0 aborting False >Thread-27566::DEBUG::2012-12-02 14:22:13,289::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27566::DEBUG::2012-12-02 14:22:13,290::task::568::TaskManager.Task::(_updateState) Task=`f089d468-2cec-4f11-a603-aa931b6a4761`::moving from state init -> state preparing >Thread-27566::INFO::2012-12-02 14:22:13,290::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27566::DEBUG::2012-12-02 14:22:13,290::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8a74dcc4-53fb-4f4b-be3c-73611d92bbd7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27566::DEBUG::2012-12-02 14:22:13,290::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27566::DEBUG::2012-12-02 14:22:13,290::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27566::DEBUG::2012-12-02 14:22:13,290::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8a74dcc4-53fb-4f4b-be3c-73611d92bbd7`::Granted request >Thread-27566::DEBUG::2012-12-02 14:22:13,291::task::794::TaskManager.Task::(resourceAcquired) Task=`f089d468-2cec-4f11-a603-aa931b6a4761`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27566::DEBUG::2012-12-02 14:22:13,291::task::957::TaskManager.Task::(_decref) Task=`f089d468-2cec-4f11-a603-aa931b6a4761`::ref 1 aborting False >Thread-27566::INFO::2012-12-02 14:22:13,297::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27566::DEBUG::2012-12-02 14:22:13,297::task::1151::TaskManager.Task::(prepare) Task=`f089d468-2cec-4f11-a603-aa931b6a4761`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27566::DEBUG::2012-12-02 14:22:13,297::task::568::TaskManager.Task::(_updateState) Task=`f089d468-2cec-4f11-a603-aa931b6a4761`::moving from state preparing -> state finished >Thread-27566::DEBUG::2012-12-02 14:22:13,297::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27566::DEBUG::2012-12-02 14:22:13,297::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27566::DEBUG::2012-12-02 14:22:13,297::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27566::DEBUG::2012-12-02 14:22:13,298::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27566::DEBUG::2012-12-02 14:22:13,298::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27566::DEBUG::2012-12-02 14:22:13,298::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27566::DEBUG::2012-12-02 14:22:13,298::task::957::TaskManager.Task::(_decref) Task=`f089d468-2cec-4f11-a603-aa931b6a4761`::ref 0 aborting False >Thread-27570::DEBUG::2012-12-02 14:22:19,549::task::568::TaskManager.Task::(_updateState) Task=`6ad851b2-fae8-4190-9bdb-5a754e7e72f9`::moving from state init -> state preparing >Thread-27570::INFO::2012-12-02 14:22:19,549::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27570::INFO::2012-12-02 14:22:19,549::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00715398788452', 'lastCheck': 1354450936.9467959, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0031840801239', 'lastCheck': 1354450932.4201219, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022509098053', 'lastCheck': 1354450934.7325239, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00805997848511', 'lastCheck': 1354450932.4304881, 'code': 0, 'valid': True}} >Thread-27570::DEBUG::2012-12-02 14:22:19,549::task::1151::TaskManager.Task::(prepare) Task=`6ad851b2-fae8-4190-9bdb-5a754e7e72f9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00715398788452', 'lastCheck': 1354450936.9467959, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0031840801239', 'lastCheck': 1354450932.4201219, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022509098053', 'lastCheck': 1354450934.7325239, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00805997848511', 'lastCheck': 1354450932.4304881, 'code': 0, 'valid': True}} >Thread-27570::DEBUG::2012-12-02 14:22:19,549::task::568::TaskManager.Task::(_updateState) Task=`6ad851b2-fae8-4190-9bdb-5a754e7e72f9`::moving from state preparing -> state finished >Thread-27570::DEBUG::2012-12-02 14:22:19,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27570::DEBUG::2012-12-02 14:22:19,550::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27570::DEBUG::2012-12-02 14:22:19,550::task::957::TaskManager.Task::(_decref) Task=`6ad851b2-fae8-4190-9bdb-5a754e7e72f9`::ref 0 aborting False >Thread-27573::DEBUG::2012-12-02 14:22:23,416::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27573::DEBUG::2012-12-02 14:22:23,416::task::568::TaskManager.Task::(_updateState) Task=`e7c7473a-6e03-4186-bcbe-15b11966da11`::moving from state init -> state preparing >Thread-27573::INFO::2012-12-02 14:22:23,416::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27573::INFO::2012-12-02 14:22:23,417::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27573::DEBUG::2012-12-02 14:22:23,417::task::1151::TaskManager.Task::(prepare) Task=`e7c7473a-6e03-4186-bcbe-15b11966da11`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27573::DEBUG::2012-12-02 14:22:23,417::task::568::TaskManager.Task::(_updateState) Task=`e7c7473a-6e03-4186-bcbe-15b11966da11`::moving from state preparing -> state finished >Thread-27573::DEBUG::2012-12-02 14:22:23,417::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27573::DEBUG::2012-12-02 14:22:23,417::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27573::DEBUG::2012-12-02 14:22:23,417::task::957::TaskManager.Task::(_decref) Task=`e7c7473a-6e03-4186-bcbe-15b11966da11`::ref 0 aborting False >Thread-27574::DEBUG::2012-12-02 14:22:23,420::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27574::DEBUG::2012-12-02 14:22:23,420::task::568::TaskManager.Task::(_updateState) Task=`9662e5d2-abb5-4a97-b465-c91285789f10`::moving from state init -> state preparing >Thread-27574::INFO::2012-12-02 14:22:23,420::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27574::DEBUG::2012-12-02 14:22:23,421::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5a14819a-c41c-4f28-85d2-4985ae21a95a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27574::DEBUG::2012-12-02 14:22:23,421::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27574::DEBUG::2012-12-02 14:22:23,421::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27574::DEBUG::2012-12-02 14:22:23,421::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5a14819a-c41c-4f28-85d2-4985ae21a95a`::Granted request >Thread-27574::DEBUG::2012-12-02 14:22:23,421::task::794::TaskManager.Task::(resourceAcquired) Task=`9662e5d2-abb5-4a97-b465-c91285789f10`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27574::DEBUG::2012-12-02 14:22:23,422::task::957::TaskManager.Task::(_decref) Task=`9662e5d2-abb5-4a97-b465-c91285789f10`::ref 1 aborting False >Thread-27574::INFO::2012-12-02 14:22:23,427::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27574::DEBUG::2012-12-02 14:22:23,427::task::1151::TaskManager.Task::(prepare) Task=`9662e5d2-abb5-4a97-b465-c91285789f10`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27574::DEBUG::2012-12-02 14:22:23,427::task::568::TaskManager.Task::(_updateState) Task=`9662e5d2-abb5-4a97-b465-c91285789f10`::moving from state preparing -> state finished >Thread-27574::DEBUG::2012-12-02 14:22:23,427::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27574::DEBUG::2012-12-02 14:22:23,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27574::DEBUG::2012-12-02 14:22:23,428::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27574::DEBUG::2012-12-02 14:22:23,428::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27574::DEBUG::2012-12-02 14:22:23,428::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27574::DEBUG::2012-12-02 14:22:23,428::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27574::DEBUG::2012-12-02 14:22:23,428::task::957::TaskManager.Task::(_decref) Task=`9662e5d2-abb5-4a97-b465-c91285789f10`::ref 0 aborting False >Thread-27578::DEBUG::2012-12-02 14:22:29,671::task::568::TaskManager.Task::(_updateState) Task=`948e31f6-4f7a-4545-bb34-3ba4ba8c4350`::moving from state init -> state preparing >Thread-27578::INFO::2012-12-02 14:22:29,671::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27578::INFO::2012-12-02 14:22:29,671::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00781607627869', 'lastCheck': 1354450946.9636779, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00914001464844', 'lastCheck': 1354450942.443521, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00156307220459', 'lastCheck': 1354450944.741564, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00784707069397', 'lastCheck': 1354450942.4492121, 'code': 0, 'valid': True}} >Thread-27578::DEBUG::2012-12-02 14:22:29,671::task::1151::TaskManager.Task::(prepare) Task=`948e31f6-4f7a-4545-bb34-3ba4ba8c4350`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00781607627869', 'lastCheck': 1354450946.9636779, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00914001464844', 'lastCheck': 1354450942.443521, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00156307220459', 'lastCheck': 1354450944.741564, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00784707069397', 'lastCheck': 1354450942.4492121, 'code': 0, 'valid': True}} >Thread-27578::DEBUG::2012-12-02 14:22:29,672::task::568::TaskManager.Task::(_updateState) Task=`948e31f6-4f7a-4545-bb34-3ba4ba8c4350`::moving from state preparing -> state finished >Thread-27578::DEBUG::2012-12-02 14:22:29,672::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27578::DEBUG::2012-12-02 14:22:29,672::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27578::DEBUG::2012-12-02 14:22:29,672::task::957::TaskManager.Task::(_decref) Task=`948e31f6-4f7a-4545-bb34-3ba4ba8c4350`::ref 0 aborting False >Thread-27581::DEBUG::2012-12-02 14:22:33,526::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27581::DEBUG::2012-12-02 14:22:33,526::task::568::TaskManager.Task::(_updateState) Task=`0132205f-e431-4af0-8ddb-2e0311b1c482`::moving from state init -> state preparing >Thread-27581::INFO::2012-12-02 14:22:33,526::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27581::INFO::2012-12-02 14:22:33,527::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27581::DEBUG::2012-12-02 14:22:33,527::task::1151::TaskManager.Task::(prepare) Task=`0132205f-e431-4af0-8ddb-2e0311b1c482`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27581::DEBUG::2012-12-02 14:22:33,527::task::568::TaskManager.Task::(_updateState) Task=`0132205f-e431-4af0-8ddb-2e0311b1c482`::moving from state preparing -> state finished >Thread-27581::DEBUG::2012-12-02 14:22:33,527::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27581::DEBUG::2012-12-02 14:22:33,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27581::DEBUG::2012-12-02 14:22:33,527::task::957::TaskManager.Task::(_decref) Task=`0132205f-e431-4af0-8ddb-2e0311b1c482`::ref 0 aborting False >Thread-27582::DEBUG::2012-12-02 14:22:33,538::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27582::DEBUG::2012-12-02 14:22:33,538::task::568::TaskManager.Task::(_updateState) Task=`8d3e64dd-f7bb-412f-8f6a-9ec35ce0ebb9`::moving from state init -> state preparing >Thread-27582::INFO::2012-12-02 14:22:33,538::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27582::DEBUG::2012-12-02 14:22:33,539::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a052213a-2e7f-44ef-9219-a4d4ca2beccd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27582::DEBUG::2012-12-02 14:22:33,539::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27582::DEBUG::2012-12-02 14:22:33,539::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27582::DEBUG::2012-12-02 14:22:33,539::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a052213a-2e7f-44ef-9219-a4d4ca2beccd`::Granted request >Thread-27582::DEBUG::2012-12-02 14:22:33,540::task::794::TaskManager.Task::(resourceAcquired) Task=`8d3e64dd-f7bb-412f-8f6a-9ec35ce0ebb9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27582::DEBUG::2012-12-02 14:22:33,540::task::957::TaskManager.Task::(_decref) Task=`8d3e64dd-f7bb-412f-8f6a-9ec35ce0ebb9`::ref 1 aborting False >Thread-27582::INFO::2012-12-02 14:22:33,546::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27582::DEBUG::2012-12-02 14:22:33,546::task::1151::TaskManager.Task::(prepare) Task=`8d3e64dd-f7bb-412f-8f6a-9ec35ce0ebb9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27582::DEBUG::2012-12-02 14:22:33,546::task::568::TaskManager.Task::(_updateState) Task=`8d3e64dd-f7bb-412f-8f6a-9ec35ce0ebb9`::moving from state preparing -> state finished >Thread-27582::DEBUG::2012-12-02 14:22:33,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27582::DEBUG::2012-12-02 14:22:33,547::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27582::DEBUG::2012-12-02 14:22:33,547::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27582::DEBUG::2012-12-02 14:22:33,547::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27582::DEBUG::2012-12-02 14:22:33,547::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27582::DEBUG::2012-12-02 14:22:33,547::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27582::DEBUG::2012-12-02 14:22:33,547::task::957::TaskManager.Task::(_decref) Task=`8d3e64dd-f7bb-412f-8f6a-9ec35ce0ebb9`::ref 0 aborting False >Thread-27586::DEBUG::2012-12-02 14:22:39,830::task::568::TaskManager.Task::(_updateState) Task=`92b7ad84-abef-4a19-9f59-c70c79c893d1`::moving from state init -> state preparing >Thread-27586::INFO::2012-12-02 14:22:39,830::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27586::INFO::2012-12-02 14:22:39,830::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00728106498718', 'lastCheck': 1354450956.9817719, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00339508056641', 'lastCheck': 1354450952.459044, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00275087356567', 'lastCheck': 1354450954.754137, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00323700904846', 'lastCheck': 1354450952.4623909, 'code': 0, 'valid': True}} >Thread-27586::DEBUG::2012-12-02 14:22:39,830::task::1151::TaskManager.Task::(prepare) Task=`92b7ad84-abef-4a19-9f59-c70c79c893d1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00728106498718', 'lastCheck': 1354450956.9817719, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00339508056641', 'lastCheck': 1354450952.459044, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00275087356567', 'lastCheck': 1354450954.754137, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00323700904846', 'lastCheck': 1354450952.4623909, 'code': 0, 'valid': True}} >Thread-27586::DEBUG::2012-12-02 14:22:39,831::task::568::TaskManager.Task::(_updateState) Task=`92b7ad84-abef-4a19-9f59-c70c79c893d1`::moving from state preparing -> state finished >Thread-27586::DEBUG::2012-12-02 14:22:39,831::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27586::DEBUG::2012-12-02 14:22:39,831::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27586::DEBUG::2012-12-02 14:22:39,831::task::957::TaskManager.Task::(_decref) Task=`92b7ad84-abef-4a19-9f59-c70c79c893d1`::ref 0 aborting False >Thread-27589::DEBUG::2012-12-02 14:22:43,684::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27589::DEBUG::2012-12-02 14:22:43,684::task::568::TaskManager.Task::(_updateState) Task=`7f18e302-cd55-4c93-8301-1eb2385265a3`::moving from state init -> state preparing >Thread-27589::INFO::2012-12-02 14:22:43,684::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27589::INFO::2012-12-02 14:22:43,684::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27589::DEBUG::2012-12-02 14:22:43,685::task::1151::TaskManager.Task::(prepare) Task=`7f18e302-cd55-4c93-8301-1eb2385265a3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27589::DEBUG::2012-12-02 14:22:43,685::task::568::TaskManager.Task::(_updateState) Task=`7f18e302-cd55-4c93-8301-1eb2385265a3`::moving from state preparing -> state finished >Thread-27589::DEBUG::2012-12-02 14:22:43,685::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27589::DEBUG::2012-12-02 14:22:43,685::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27589::DEBUG::2012-12-02 14:22:43,685::task::957::TaskManager.Task::(_decref) Task=`7f18e302-cd55-4c93-8301-1eb2385265a3`::ref 0 aborting False >Thread-27590::DEBUG::2012-12-02 14:22:43,688::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27590::DEBUG::2012-12-02 14:22:43,688::task::568::TaskManager.Task::(_updateState) Task=`fc69e357-e9cc-423c-ac30-0eccd53513eb`::moving from state init -> state preparing >Thread-27590::INFO::2012-12-02 14:22:43,688::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27590::DEBUG::2012-12-02 14:22:43,688::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`312d4fd4-f12f-4dba-b650-3a23dd553c03`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27590::DEBUG::2012-12-02 14:22:43,688::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27590::DEBUG::2012-12-02 14:22:43,688::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27590::DEBUG::2012-12-02 14:22:43,688::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`312d4fd4-f12f-4dba-b650-3a23dd553c03`::Granted request >Thread-27590::DEBUG::2012-12-02 14:22:43,689::task::794::TaskManager.Task::(resourceAcquired) Task=`fc69e357-e9cc-423c-ac30-0eccd53513eb`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27590::DEBUG::2012-12-02 14:22:43,689::task::957::TaskManager.Task::(_decref) Task=`fc69e357-e9cc-423c-ac30-0eccd53513eb`::ref 1 aborting False >Thread-27590::INFO::2012-12-02 14:22:43,693::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27590::DEBUG::2012-12-02 14:22:43,693::task::1151::TaskManager.Task::(prepare) Task=`fc69e357-e9cc-423c-ac30-0eccd53513eb`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27590::DEBUG::2012-12-02 14:22:43,693::task::568::TaskManager.Task::(_updateState) Task=`fc69e357-e9cc-423c-ac30-0eccd53513eb`::moving from state preparing -> state finished >Thread-27590::DEBUG::2012-12-02 14:22:43,693::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27590::DEBUG::2012-12-02 14:22:43,693::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27590::DEBUG::2012-12-02 14:22:43,693::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27590::DEBUG::2012-12-02 14:22:43,694::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27590::DEBUG::2012-12-02 14:22:43,694::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27590::DEBUG::2012-12-02 14:22:43,694::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27590::DEBUG::2012-12-02 14:22:43,694::task::957::TaskManager.Task::(_decref) Task=`fc69e357-e9cc-423c-ac30-0eccd53513eb`::ref 0 aborting False >Thread-27594::DEBUG::2012-12-02 14:22:49,943::task::568::TaskManager.Task::(_updateState) Task=`d1d8a8e0-9b77-40fa-809b-a92d51d5087a`::moving from state init -> state preparing >Thread-27594::INFO::2012-12-02 14:22:49,944::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27594::INFO::2012-12-02 14:22:49,944::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00555610656738', 'lastCheck': 1354450966.9961171, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00381588935852', 'lastCheck': 1354450962.4790981, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00161385536194', 'lastCheck': 1354450964.762723, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00898599624634', 'lastCheck': 1354450962.4863429, 'code': 0, 'valid': True}} >Thread-27594::DEBUG::2012-12-02 14:22:49,944::task::1151::TaskManager.Task::(prepare) Task=`d1d8a8e0-9b77-40fa-809b-a92d51d5087a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00555610656738', 'lastCheck': 1354450966.9961171, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00381588935852', 'lastCheck': 1354450962.4790981, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00161385536194', 'lastCheck': 1354450964.762723, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00898599624634', 'lastCheck': 1354450962.4863429, 'code': 0, 'valid': True}} >Thread-27594::DEBUG::2012-12-02 14:22:49,944::task::568::TaskManager.Task::(_updateState) Task=`d1d8a8e0-9b77-40fa-809b-a92d51d5087a`::moving from state preparing -> state finished >Thread-27594::DEBUG::2012-12-02 14:22:49,944::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27594::DEBUG::2012-12-02 14:22:49,944::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27594::DEBUG::2012-12-02 14:22:49,945::task::957::TaskManager.Task::(_decref) Task=`d1d8a8e0-9b77-40fa-809b-a92d51d5087a`::ref 0 aborting False >Thread-27597::DEBUG::2012-12-02 14:22:53,839::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27597::DEBUG::2012-12-02 14:22:53,839::task::568::TaskManager.Task::(_updateState) Task=`563fcd98-6a63-479b-8479-b03b6bdc9579`::moving from state init -> state preparing >Thread-27597::INFO::2012-12-02 14:22:53,839::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27597::INFO::2012-12-02 14:22:53,839::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27597::DEBUG::2012-12-02 14:22:53,840::task::1151::TaskManager.Task::(prepare) Task=`563fcd98-6a63-479b-8479-b03b6bdc9579`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27597::DEBUG::2012-12-02 14:22:53,840::task::568::TaskManager.Task::(_updateState) Task=`563fcd98-6a63-479b-8479-b03b6bdc9579`::moving from state preparing -> state finished >Thread-27597::DEBUG::2012-12-02 14:22:53,840::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27597::DEBUG::2012-12-02 14:22:53,840::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27597::DEBUG::2012-12-02 14:22:53,840::task::957::TaskManager.Task::(_decref) Task=`563fcd98-6a63-479b-8479-b03b6bdc9579`::ref 0 aborting False >Thread-27598::DEBUG::2012-12-02 14:22:53,843::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27598::DEBUG::2012-12-02 14:22:53,843::task::568::TaskManager.Task::(_updateState) Task=`3116e25c-739f-44c5-a688-352aab4de2be`::moving from state init -> state preparing >Thread-27598::INFO::2012-12-02 14:22:53,843::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27598::DEBUG::2012-12-02 14:22:53,844::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`072c6803-f499-4603-bd87-f13595e751f7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27598::DEBUG::2012-12-02 14:22:53,844::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27598::DEBUG::2012-12-02 14:22:53,844::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27598::DEBUG::2012-12-02 14:22:53,844::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`072c6803-f499-4603-bd87-f13595e751f7`::Granted request >Thread-27598::DEBUG::2012-12-02 14:22:53,844::task::794::TaskManager.Task::(resourceAcquired) Task=`3116e25c-739f-44c5-a688-352aab4de2be`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27598::DEBUG::2012-12-02 14:22:53,844::task::957::TaskManager.Task::(_decref) Task=`3116e25c-739f-44c5-a688-352aab4de2be`::ref 1 aborting False >Thread-27598::INFO::2012-12-02 14:22:53,849::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27598::DEBUG::2012-12-02 14:22:53,849::task::1151::TaskManager.Task::(prepare) Task=`3116e25c-739f-44c5-a688-352aab4de2be`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27598::DEBUG::2012-12-02 14:22:53,849::task::568::TaskManager.Task::(_updateState) Task=`3116e25c-739f-44c5-a688-352aab4de2be`::moving from state preparing -> state finished >Thread-27598::DEBUG::2012-12-02 14:22:53,849::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27598::DEBUG::2012-12-02 14:22:53,849::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27598::DEBUG::2012-12-02 14:22:53,850::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27598::DEBUG::2012-12-02 14:22:53,850::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27598::DEBUG::2012-12-02 14:22:53,850::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27598::DEBUG::2012-12-02 14:22:53,850::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27598::DEBUG::2012-12-02 14:22:53,850::task::957::TaskManager.Task::(_decref) Task=`3116e25c-739f-44c5-a688-352aab4de2be`::ref 0 aborting False >Thread-27602::DEBUG::2012-12-02 14:23:00,031::task::568::TaskManager.Task::(_updateState) Task=`9658985f-fb1b-4e11-9f8b-fab0670975b3`::moving from state init -> state preparing >Thread-27602::INFO::2012-12-02 14:23:00,031::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27602::INFO::2012-12-02 14:23:00,031::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0147259235382', 'lastCheck': 1354450977.0207479, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00162601470947', 'lastCheck': 1354450972.5840709, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00156593322754', 'lastCheck': 1354450974.7717741, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00364303588867', 'lastCheck': 1354450972.585212, 'code': 0, 'valid': True}} >Thread-27602::DEBUG::2012-12-02 14:23:00,032::task::1151::TaskManager.Task::(prepare) Task=`9658985f-fb1b-4e11-9f8b-fab0670975b3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0147259235382', 'lastCheck': 1354450977.0207479, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00162601470947', 'lastCheck': 1354450972.5840709, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00156593322754', 'lastCheck': 1354450974.7717741, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00364303588867', 'lastCheck': 1354450972.585212, 'code': 0, 'valid': True}} >Thread-27602::DEBUG::2012-12-02 14:23:00,032::task::568::TaskManager.Task::(_updateState) Task=`9658985f-fb1b-4e11-9f8b-fab0670975b3`::moving from state preparing -> state finished >Thread-27602::DEBUG::2012-12-02 14:23:00,032::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27602::DEBUG::2012-12-02 14:23:00,032::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27602::DEBUG::2012-12-02 14:23:00,032::task::957::TaskManager.Task::(_decref) Task=`9658985f-fb1b-4e11-9f8b-fab0670975b3`::ref 0 aborting False >Thread-27605::DEBUG::2012-12-02 14:23:04,026::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27605::DEBUG::2012-12-02 14:23:04,027::task::568::TaskManager.Task::(_updateState) Task=`d5c297cf-7f4c-48a2-a744-d48b27a97b0a`::moving from state init -> state preparing >Thread-27605::INFO::2012-12-02 14:23:04,027::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27605::INFO::2012-12-02 14:23:04,027::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27605::DEBUG::2012-12-02 14:23:04,027::task::1151::TaskManager.Task::(prepare) Task=`d5c297cf-7f4c-48a2-a744-d48b27a97b0a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27605::DEBUG::2012-12-02 14:23:04,027::task::568::TaskManager.Task::(_updateState) Task=`d5c297cf-7f4c-48a2-a744-d48b27a97b0a`::moving from state preparing -> state finished >Thread-27605::DEBUG::2012-12-02 14:23:04,028::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27605::DEBUG::2012-12-02 14:23:04,028::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27605::DEBUG::2012-12-02 14:23:04,028::task::957::TaskManager.Task::(_decref) Task=`d5c297cf-7f4c-48a2-a744-d48b27a97b0a`::ref 0 aborting False >Thread-27606::DEBUG::2012-12-02 14:23:04,030::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27606::DEBUG::2012-12-02 14:23:04,031::task::568::TaskManager.Task::(_updateState) Task=`0a47d3ae-384b-4f96-b664-66a8aeb8547c`::moving from state init -> state preparing >Thread-27606::INFO::2012-12-02 14:23:04,031::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27606::DEBUG::2012-12-02 14:23:04,031::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e609d60c-8926-429e-a0af-fb8ce4c1e0ea`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27606::DEBUG::2012-12-02 14:23:04,031::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27606::DEBUG::2012-12-02 14:23:04,032::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27606::DEBUG::2012-12-02 14:23:04,032::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e609d60c-8926-429e-a0af-fb8ce4c1e0ea`::Granted request >Thread-27606::DEBUG::2012-12-02 14:23:04,032::task::794::TaskManager.Task::(resourceAcquired) Task=`0a47d3ae-384b-4f96-b664-66a8aeb8547c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27606::DEBUG::2012-12-02 14:23:04,032::task::957::TaskManager.Task::(_decref) Task=`0a47d3ae-384b-4f96-b664-66a8aeb8547c`::ref 1 aborting False >Thread-27606::INFO::2012-12-02 14:23:04,037::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27606::DEBUG::2012-12-02 14:23:04,037::task::1151::TaskManager.Task::(prepare) Task=`0a47d3ae-384b-4f96-b664-66a8aeb8547c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27606::DEBUG::2012-12-02 14:23:04,038::task::568::TaskManager.Task::(_updateState) Task=`0a47d3ae-384b-4f96-b664-66a8aeb8547c`::moving from state preparing -> state finished >Thread-27606::DEBUG::2012-12-02 14:23:04,038::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27606::DEBUG::2012-12-02 14:23:04,038::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27606::DEBUG::2012-12-02 14:23:04,038::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27606::DEBUG::2012-12-02 14:23:04,038::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27606::DEBUG::2012-12-02 14:23:04,038::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27606::DEBUG::2012-12-02 14:23:04,038::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27606::DEBUG::2012-12-02 14:23:04,038::task::957::TaskManager.Task::(_decref) Task=`0a47d3ae-384b-4f96-b664-66a8aeb8547c`::ref 0 aborting False >Thread-27610::DEBUG::2012-12-02 14:23:10,137::task::568::TaskManager.Task::(_updateState) Task=`bf356db1-5b72-4fec-9ee9-de7c2ec6b196`::moving from state init -> state preparing >Thread-27610::INFO::2012-12-02 14:23:10,137::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27610::INFO::2012-12-02 14:23:10,138::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00642204284668', 'lastCheck': 1354450987.0363779, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00295519828796', 'lastCheck': 1354450982.603266, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00197601318359', 'lastCheck': 1354450984.7820311, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00861406326294', 'lastCheck': 1354450982.609781, 'code': 0, 'valid': True}} >Thread-27610::DEBUG::2012-12-02 14:23:10,138::task::1151::TaskManager.Task::(prepare) Task=`bf356db1-5b72-4fec-9ee9-de7c2ec6b196`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00642204284668', 'lastCheck': 1354450987.0363779, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00295519828796', 'lastCheck': 1354450982.603266, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00197601318359', 'lastCheck': 1354450984.7820311, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00861406326294', 'lastCheck': 1354450982.609781, 'code': 0, 'valid': True}} >Thread-27610::DEBUG::2012-12-02 14:23:10,138::task::568::TaskManager.Task::(_updateState) Task=`bf356db1-5b72-4fec-9ee9-de7c2ec6b196`::moving from state preparing -> state finished >Thread-27610::DEBUG::2012-12-02 14:23:10,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27610::DEBUG::2012-12-02 14:23:10,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27610::DEBUG::2012-12-02 14:23:10,138::task::957::TaskManager.Task::(_decref) Task=`bf356db1-5b72-4fec-9ee9-de7c2ec6b196`::ref 0 aborting False >Thread-27613::DEBUG::2012-12-02 14:23:14,189::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27613::DEBUG::2012-12-02 14:23:14,190::task::568::TaskManager.Task::(_updateState) Task=`e3492b26-70cd-486e-9b82-1bd1c0c6e6a7`::moving from state init -> state preparing >Thread-27613::INFO::2012-12-02 14:23:14,190::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27613::INFO::2012-12-02 14:23:14,190::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27613::DEBUG::2012-12-02 14:23:14,190::task::1151::TaskManager.Task::(prepare) Task=`e3492b26-70cd-486e-9b82-1bd1c0c6e6a7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27613::DEBUG::2012-12-02 14:23:14,190::task::568::TaskManager.Task::(_updateState) Task=`e3492b26-70cd-486e-9b82-1bd1c0c6e6a7`::moving from state preparing -> state finished >Thread-27613::DEBUG::2012-12-02 14:23:14,191::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27613::DEBUG::2012-12-02 14:23:14,191::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27613::DEBUG::2012-12-02 14:23:14,191::task::957::TaskManager.Task::(_decref) Task=`e3492b26-70cd-486e-9b82-1bd1c0c6e6a7`::ref 0 aborting False >Thread-27614::DEBUG::2012-12-02 14:23:14,193::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27614::DEBUG::2012-12-02 14:23:14,194::task::568::TaskManager.Task::(_updateState) Task=`7dc06c7d-6b0e-40c4-a780-77a0328d13d6`::moving from state init -> state preparing >Thread-27614::INFO::2012-12-02 14:23:14,194::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27614::DEBUG::2012-12-02 14:23:14,194::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`997a21c0-a1ec-4956-a7d5-449998d725bf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27614::DEBUG::2012-12-02 14:23:14,194::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27614::DEBUG::2012-12-02 14:23:14,194::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27614::DEBUG::2012-12-02 14:23:14,195::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`997a21c0-a1ec-4956-a7d5-449998d725bf`::Granted request >Thread-27614::DEBUG::2012-12-02 14:23:14,195::task::794::TaskManager.Task::(resourceAcquired) Task=`7dc06c7d-6b0e-40c4-a780-77a0328d13d6`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27614::DEBUG::2012-12-02 14:23:14,195::task::957::TaskManager.Task::(_decref) Task=`7dc06c7d-6b0e-40c4-a780-77a0328d13d6`::ref 1 aborting False >Thread-27614::INFO::2012-12-02 14:23:14,200::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27614::DEBUG::2012-12-02 14:23:14,200::task::1151::TaskManager.Task::(prepare) Task=`7dc06c7d-6b0e-40c4-a780-77a0328d13d6`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27614::DEBUG::2012-12-02 14:23:14,200::task::568::TaskManager.Task::(_updateState) Task=`7dc06c7d-6b0e-40c4-a780-77a0328d13d6`::moving from state preparing -> state finished >Thread-27614::DEBUG::2012-12-02 14:23:14,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27614::DEBUG::2012-12-02 14:23:14,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27614::DEBUG::2012-12-02 14:23:14,200::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27614::DEBUG::2012-12-02 14:23:14,200::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27614::DEBUG::2012-12-02 14:23:14,200::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27614::DEBUG::2012-12-02 14:23:14,200::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27614::DEBUG::2012-12-02 14:23:14,201::task::957::TaskManager.Task::(_decref) Task=`7dc06c7d-6b0e-40c4-a780-77a0328d13d6`::ref 0 aborting False >Thread-27618::DEBUG::2012-12-02 14:23:20,243::task::568::TaskManager.Task::(_updateState) Task=`5646ea48-7d46-415e-8d0c-7838816cb3d0`::moving from state init -> state preparing >Thread-27618::INFO::2012-12-02 14:23:20,244::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27618::INFO::2012-12-02 14:23:20,244::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00711703300476', 'lastCheck': 1354450997.0529871, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00453495979309', 'lastCheck': 1354450992.6195979, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00184392929077', 'lastCheck': 1354450994.8319509, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00682616233826', 'lastCheck': 1354450992.6268079, 'code': 0, 'valid': True}} >Thread-27618::DEBUG::2012-12-02 14:23:20,244::task::1151::TaskManager.Task::(prepare) Task=`5646ea48-7d46-415e-8d0c-7838816cb3d0`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00711703300476', 'lastCheck': 1354450997.0529871, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00453495979309', 'lastCheck': 1354450992.6195979, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00184392929077', 'lastCheck': 1354450994.8319509, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00682616233826', 'lastCheck': 1354450992.6268079, 'code': 0, 'valid': True}} >Thread-27618::DEBUG::2012-12-02 14:23:20,244::task::568::TaskManager.Task::(_updateState) Task=`5646ea48-7d46-415e-8d0c-7838816cb3d0`::moving from state preparing -> state finished >Thread-27618::DEBUG::2012-12-02 14:23:20,244::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27618::DEBUG::2012-12-02 14:23:20,244::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27618::DEBUG::2012-12-02 14:23:20,245::task::957::TaskManager.Task::(_decref) Task=`5646ea48-7d46-415e-8d0c-7838816cb3d0`::ref 0 aborting False >Thread-27622::DEBUG::2012-12-02 14:23:24,362::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27622::DEBUG::2012-12-02 14:23:24,362::task::568::TaskManager.Task::(_updateState) Task=`4058f051-e72f-4ccd-9d2f-5c3198336c84`::moving from state init -> state preparing >Thread-27622::INFO::2012-12-02 14:23:24,363::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27622::INFO::2012-12-02 14:23:24,363::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27622::DEBUG::2012-12-02 14:23:24,363::task::1151::TaskManager.Task::(prepare) Task=`4058f051-e72f-4ccd-9d2f-5c3198336c84`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27622::DEBUG::2012-12-02 14:23:24,363::task::568::TaskManager.Task::(_updateState) Task=`4058f051-e72f-4ccd-9d2f-5c3198336c84`::moving from state preparing -> state finished >Thread-27622::DEBUG::2012-12-02 14:23:24,363::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27622::DEBUG::2012-12-02 14:23:24,363::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27622::DEBUG::2012-12-02 14:23:24,363::task::957::TaskManager.Task::(_decref) Task=`4058f051-e72f-4ccd-9d2f-5c3198336c84`::ref 0 aborting False >Thread-27623::DEBUG::2012-12-02 14:23:24,366::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27623::DEBUG::2012-12-02 14:23:24,366::task::568::TaskManager.Task::(_updateState) Task=`f1bfc272-e10b-4b88-95b3-cdb8936d533b`::moving from state init -> state preparing >Thread-27623::INFO::2012-12-02 14:23:24,366::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27623::DEBUG::2012-12-02 14:23:24,367::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`99a3284a-e682-4184-a225-8b4366ea1b92`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27623::DEBUG::2012-12-02 14:23:24,367::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27623::DEBUG::2012-12-02 14:23:24,367::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27623::DEBUG::2012-12-02 14:23:24,367::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`99a3284a-e682-4184-a225-8b4366ea1b92`::Granted request >Thread-27623::DEBUG::2012-12-02 14:23:24,367::task::794::TaskManager.Task::(resourceAcquired) Task=`f1bfc272-e10b-4b88-95b3-cdb8936d533b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27623::DEBUG::2012-12-02 14:23:24,367::task::957::TaskManager.Task::(_decref) Task=`f1bfc272-e10b-4b88-95b3-cdb8936d533b`::ref 1 aborting False >Thread-27623::INFO::2012-12-02 14:23:24,374::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27623::DEBUG::2012-12-02 14:23:24,374::task::1151::TaskManager.Task::(prepare) Task=`f1bfc272-e10b-4b88-95b3-cdb8936d533b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27623::DEBUG::2012-12-02 14:23:24,374::task::568::TaskManager.Task::(_updateState) Task=`f1bfc272-e10b-4b88-95b3-cdb8936d533b`::moving from state preparing -> state finished >Thread-27623::DEBUG::2012-12-02 14:23:24,374::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27623::DEBUG::2012-12-02 14:23:24,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27623::DEBUG::2012-12-02 14:23:24,375::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27623::DEBUG::2012-12-02 14:23:24,375::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27623::DEBUG::2012-12-02 14:23:24,375::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27623::DEBUG::2012-12-02 14:23:24,375::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27623::DEBUG::2012-12-02 14:23:24,375::task::957::TaskManager.Task::(_decref) Task=`f1bfc272-e10b-4b88-95b3-cdb8936d533b`::ref 0 aborting False >Thread-27626::DEBUG::2012-12-02 14:23:30,357::task::568::TaskManager.Task::(_updateState) Task=`f85f3174-1e7b-432d-9af3-e3fe24c8f6e6`::moving from state init -> state preparing >Thread-27626::INFO::2012-12-02 14:23:30,357::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27626::INFO::2012-12-02 14:23:30,357::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00896310806274', 'lastCheck': 1354451007.0707531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00463390350342', 'lastCheck': 1354451002.6397181, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00174117088318', 'lastCheck': 1354451004.8417799, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00798797607422', 'lastCheck': 1354451002.6463931, 'code': 0, 'valid': True}} >Thread-27626::DEBUG::2012-12-02 14:23:30,358::task::1151::TaskManager.Task::(prepare) Task=`f85f3174-1e7b-432d-9af3-e3fe24c8f6e6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00896310806274', 'lastCheck': 1354451007.0707531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00463390350342', 'lastCheck': 1354451002.6397181, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00174117088318', 'lastCheck': 1354451004.8417799, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00798797607422', 'lastCheck': 1354451002.6463931, 'code': 0, 'valid': True}} >Thread-27626::DEBUG::2012-12-02 14:23:30,358::task::568::TaskManager.Task::(_updateState) Task=`f85f3174-1e7b-432d-9af3-e3fe24c8f6e6`::moving from state preparing -> state finished >Thread-27626::DEBUG::2012-12-02 14:23:30,358::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27626::DEBUG::2012-12-02 14:23:30,358::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27626::DEBUG::2012-12-02 14:23:30,358::task::957::TaskManager.Task::(_decref) Task=`f85f3174-1e7b-432d-9af3-e3fe24c8f6e6`::ref 0 aborting False >Thread-27630::DEBUG::2012-12-02 14:23:34,624::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27630::DEBUG::2012-12-02 14:23:34,624::task::568::TaskManager.Task::(_updateState) Task=`76e5e2ff-d19f-40d3-8e06-a96587ef7c21`::moving from state init -> state preparing >Thread-27630::INFO::2012-12-02 14:23:34,625::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27630::INFO::2012-12-02 14:23:34,625::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27630::DEBUG::2012-12-02 14:23:34,625::task::1151::TaskManager.Task::(prepare) Task=`76e5e2ff-d19f-40d3-8e06-a96587ef7c21`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27630::DEBUG::2012-12-02 14:23:34,625::task::568::TaskManager.Task::(_updateState) Task=`76e5e2ff-d19f-40d3-8e06-a96587ef7c21`::moving from state preparing -> state finished >Thread-27630::DEBUG::2012-12-02 14:23:34,625::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27630::DEBUG::2012-12-02 14:23:34,625::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27630::DEBUG::2012-12-02 14:23:34,625::task::957::TaskManager.Task::(_decref) Task=`76e5e2ff-d19f-40d3-8e06-a96587ef7c21`::ref 0 aborting False >Thread-27631::DEBUG::2012-12-02 14:23:34,628::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27631::DEBUG::2012-12-02 14:23:34,628::task::568::TaskManager.Task::(_updateState) Task=`cd9a25c0-c530-461a-83e1-4b4dc593c8b5`::moving from state init -> state preparing >Thread-27631::INFO::2012-12-02 14:23:34,628::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27631::DEBUG::2012-12-02 14:23:34,628::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`01dc4fc1-8c2e-4db8-bafa-9c382af69175`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27631::DEBUG::2012-12-02 14:23:34,629::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27631::DEBUG::2012-12-02 14:23:34,629::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27631::DEBUG::2012-12-02 14:23:34,629::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`01dc4fc1-8c2e-4db8-bafa-9c382af69175`::Granted request >Thread-27631::DEBUG::2012-12-02 14:23:34,629::task::794::TaskManager.Task::(resourceAcquired) Task=`cd9a25c0-c530-461a-83e1-4b4dc593c8b5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27631::DEBUG::2012-12-02 14:23:34,629::task::957::TaskManager.Task::(_decref) Task=`cd9a25c0-c530-461a-83e1-4b4dc593c8b5`::ref 1 aborting False >Thread-27631::INFO::2012-12-02 14:23:34,635::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27631::DEBUG::2012-12-02 14:23:34,636::task::1151::TaskManager.Task::(prepare) Task=`cd9a25c0-c530-461a-83e1-4b4dc593c8b5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27631::DEBUG::2012-12-02 14:23:34,636::task::568::TaskManager.Task::(_updateState) Task=`cd9a25c0-c530-461a-83e1-4b4dc593c8b5`::moving from state preparing -> state finished >Thread-27631::DEBUG::2012-12-02 14:23:34,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27631::DEBUG::2012-12-02 14:23:34,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27631::DEBUG::2012-12-02 14:23:34,636::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27631::DEBUG::2012-12-02 14:23:34,636::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27631::DEBUG::2012-12-02 14:23:34,636::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27631::DEBUG::2012-12-02 14:23:34,636::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27631::DEBUG::2012-12-02 14:23:34,636::task::957::TaskManager.Task::(_decref) Task=`cd9a25c0-c530-461a-83e1-4b4dc593c8b5`::ref 0 aborting False >Thread-27634::DEBUG::2012-12-02 14:23:40,445::task::568::TaskManager.Task::(_updateState) Task=`acd295b8-53ad-498b-9637-2bcf79ab5a9a`::moving from state init -> state preparing >Thread-27634::INFO::2012-12-02 14:23:40,445::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27634::INFO::2012-12-02 14:23:40,445::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00811314582825', 'lastCheck': 1354451017.090405, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00448679924011', 'lastCheck': 1354451012.6566679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287795066833', 'lastCheck': 1354451014.8543551, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00780296325684', 'lastCheck': 1354451012.6650119, 'code': 0, 'valid': True}} >Thread-27634::DEBUG::2012-12-02 14:23:40,446::task::1151::TaskManager.Task::(prepare) Task=`acd295b8-53ad-498b-9637-2bcf79ab5a9a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00811314582825', 'lastCheck': 1354451017.090405, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00448679924011', 'lastCheck': 1354451012.6566679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287795066833', 'lastCheck': 1354451014.8543551, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00780296325684', 'lastCheck': 1354451012.6650119, 'code': 0, 'valid': True}} >Thread-27634::DEBUG::2012-12-02 14:23:40,446::task::568::TaskManager.Task::(_updateState) Task=`acd295b8-53ad-498b-9637-2bcf79ab5a9a`::moving from state preparing -> state finished >Thread-27634::DEBUG::2012-12-02 14:23:40,446::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27634::DEBUG::2012-12-02 14:23:40,446::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27634::DEBUG::2012-12-02 14:23:40,446::task::957::TaskManager.Task::(_decref) Task=`acd295b8-53ad-498b-9637-2bcf79ab5a9a`::ref 0 aborting False >Thread-27638::DEBUG::2012-12-02 14:23:44,787::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27638::DEBUG::2012-12-02 14:23:44,787::task::568::TaskManager.Task::(_updateState) Task=`ae257826-4bcd-40b4-9251-29c9e4ed5a0c`::moving from state init -> state preparing >Thread-27638::INFO::2012-12-02 14:23:44,787::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27638::INFO::2012-12-02 14:23:44,788::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27638::DEBUG::2012-12-02 14:23:44,788::task::1151::TaskManager.Task::(prepare) Task=`ae257826-4bcd-40b4-9251-29c9e4ed5a0c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27638::DEBUG::2012-12-02 14:23:44,788::task::568::TaskManager.Task::(_updateState) Task=`ae257826-4bcd-40b4-9251-29c9e4ed5a0c`::moving from state preparing -> state finished >Thread-27638::DEBUG::2012-12-02 14:23:44,788::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27638::DEBUG::2012-12-02 14:23:44,788::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27638::DEBUG::2012-12-02 14:23:44,788::task::957::TaskManager.Task::(_decref) Task=`ae257826-4bcd-40b4-9251-29c9e4ed5a0c`::ref 0 aborting False >Thread-27639::DEBUG::2012-12-02 14:23:44,791::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27639::DEBUG::2012-12-02 14:23:44,791::task::568::TaskManager.Task::(_updateState) Task=`8c0775ab-16a8-4f38-8db5-831916eff2d0`::moving from state init -> state preparing >Thread-27639::INFO::2012-12-02 14:23:44,791::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27639::DEBUG::2012-12-02 14:23:44,791::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`26e113d1-3f9d-40d2-820d-f49dc8ad7010`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27639::DEBUG::2012-12-02 14:23:44,791::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27639::DEBUG::2012-12-02 14:23:44,792::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27639::DEBUG::2012-12-02 14:23:44,792::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`26e113d1-3f9d-40d2-820d-f49dc8ad7010`::Granted request >Thread-27639::DEBUG::2012-12-02 14:23:44,792::task::794::TaskManager.Task::(resourceAcquired) Task=`8c0775ab-16a8-4f38-8db5-831916eff2d0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27639::DEBUG::2012-12-02 14:23:44,792::task::957::TaskManager.Task::(_decref) Task=`8c0775ab-16a8-4f38-8db5-831916eff2d0`::ref 1 aborting False >Thread-27639::INFO::2012-12-02 14:23:44,798::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27639::DEBUG::2012-12-02 14:23:44,798::task::1151::TaskManager.Task::(prepare) Task=`8c0775ab-16a8-4f38-8db5-831916eff2d0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27639::DEBUG::2012-12-02 14:23:44,798::task::568::TaskManager.Task::(_updateState) Task=`8c0775ab-16a8-4f38-8db5-831916eff2d0`::moving from state preparing -> state finished >Thread-27639::DEBUG::2012-12-02 14:23:44,798::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27639::DEBUG::2012-12-02 14:23:44,798::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27639::DEBUG::2012-12-02 14:23:44,798::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27639::DEBUG::2012-12-02 14:23:44,798::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27639::DEBUG::2012-12-02 14:23:44,798::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27639::DEBUG::2012-12-02 14:23:44,799::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27639::DEBUG::2012-12-02 14:23:44,799::task::957::TaskManager.Task::(_decref) Task=`8c0775ab-16a8-4f38-8db5-831916eff2d0`::ref 0 aborting False >Thread-27642::DEBUG::2012-12-02 14:23:50,695::task::568::TaskManager.Task::(_updateState) Task=`2d4cbd76-3f47-434d-91df-e494bb8648ce`::moving from state init -> state preparing >Thread-27642::INFO::2012-12-02 14:23:50,695::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27642::INFO::2012-12-02 14:23:50,696::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00847721099854', 'lastCheck': 1354451027.1076019, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00588417053223', 'lastCheck': 1354451022.6777451, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00201106071472', 'lastCheck': 1354451024.863251, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00771403312683', 'lastCheck': 1354451022.685874, 'code': 0, 'valid': True}} >Thread-27642::DEBUG::2012-12-02 14:23:50,696::task::1151::TaskManager.Task::(prepare) Task=`2d4cbd76-3f47-434d-91df-e494bb8648ce`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00847721099854', 'lastCheck': 1354451027.1076019, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00588417053223', 'lastCheck': 1354451022.6777451, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00201106071472', 'lastCheck': 1354451024.863251, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00771403312683', 'lastCheck': 1354451022.685874, 'code': 0, 'valid': True}} >Thread-27642::DEBUG::2012-12-02 14:23:50,696::task::568::TaskManager.Task::(_updateState) Task=`2d4cbd76-3f47-434d-91df-e494bb8648ce`::moving from state preparing -> state finished >Thread-27642::DEBUG::2012-12-02 14:23:50,696::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27642::DEBUG::2012-12-02 14:23:50,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27642::DEBUG::2012-12-02 14:23:50,696::task::957::TaskManager.Task::(_decref) Task=`2d4cbd76-3f47-434d-91df-e494bb8648ce`::ref 0 aborting False >Thread-27646::DEBUG::2012-12-02 14:23:54,950::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27646::DEBUG::2012-12-02 14:23:54,950::task::568::TaskManager.Task::(_updateState) Task=`924a6f49-216e-41a5-a1a9-75fcf1508fc7`::moving from state init -> state preparing >Thread-27646::INFO::2012-12-02 14:23:54,951::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27646::INFO::2012-12-02 14:23:54,951::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27646::DEBUG::2012-12-02 14:23:54,951::task::1151::TaskManager.Task::(prepare) Task=`924a6f49-216e-41a5-a1a9-75fcf1508fc7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27646::DEBUG::2012-12-02 14:23:54,951::task::568::TaskManager.Task::(_updateState) Task=`924a6f49-216e-41a5-a1a9-75fcf1508fc7`::moving from state preparing -> state finished >Thread-27646::DEBUG::2012-12-02 14:23:54,951::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27646::DEBUG::2012-12-02 14:23:54,951::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27646::DEBUG::2012-12-02 14:23:54,951::task::957::TaskManager.Task::(_decref) Task=`924a6f49-216e-41a5-a1a9-75fcf1508fc7`::ref 0 aborting False >Thread-27647::DEBUG::2012-12-02 14:23:54,954::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27647::DEBUG::2012-12-02 14:23:54,954::task::568::TaskManager.Task::(_updateState) Task=`f5c7b56d-382c-4cb4-8d02-073bb183cc39`::moving from state init -> state preparing >Thread-27647::INFO::2012-12-02 14:23:54,954::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27647::DEBUG::2012-12-02 14:23:54,954::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6d5cbea3-c0e6-43e0-be04-826e8b490da1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27647::DEBUG::2012-12-02 14:23:54,955::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27647::DEBUG::2012-12-02 14:23:54,955::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27647::DEBUG::2012-12-02 14:23:54,955::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6d5cbea3-c0e6-43e0-be04-826e8b490da1`::Granted request >Thread-27647::DEBUG::2012-12-02 14:23:54,955::task::794::TaskManager.Task::(resourceAcquired) Task=`f5c7b56d-382c-4cb4-8d02-073bb183cc39`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27647::DEBUG::2012-12-02 14:23:54,955::task::957::TaskManager.Task::(_decref) Task=`f5c7b56d-382c-4cb4-8d02-073bb183cc39`::ref 1 aborting False >Thread-27647::INFO::2012-12-02 14:23:54,960::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27647::DEBUG::2012-12-02 14:23:54,960::task::1151::TaskManager.Task::(prepare) Task=`f5c7b56d-382c-4cb4-8d02-073bb183cc39`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27647::DEBUG::2012-12-02 14:23:54,960::task::568::TaskManager.Task::(_updateState) Task=`f5c7b56d-382c-4cb4-8d02-073bb183cc39`::moving from state preparing -> state finished >Thread-27647::DEBUG::2012-12-02 14:23:54,960::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27647::DEBUG::2012-12-02 14:23:54,960::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27647::DEBUG::2012-12-02 14:23:54,960::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27647::DEBUG::2012-12-02 14:23:54,961::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27647::DEBUG::2012-12-02 14:23:54,961::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27647::DEBUG::2012-12-02 14:23:54,961::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27647::DEBUG::2012-12-02 14:23:54,961::task::957::TaskManager.Task::(_decref) Task=`f5c7b56d-382c-4cb4-8d02-073bb183cc39`::ref 0 aborting False >Thread-27650::DEBUG::2012-12-02 14:24:00,771::task::568::TaskManager.Task::(_updateState) Task=`0b876eae-370f-4d62-9ca8-281583cdbf16`::moving from state init -> state preparing >Thread-27650::INFO::2012-12-02 14:24:00,771::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27650::INFO::2012-12-02 14:24:00,771::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00875878334045', 'lastCheck': 1354451037.127305, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00298190116882', 'lastCheck': 1354451032.694037, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172996520996', 'lastCheck': 1354451034.8725891, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00732779502869', 'lastCheck': 1354451032.703685, 'code': 0, 'valid': True}} >Thread-27650::DEBUG::2012-12-02 14:24:00,771::task::1151::TaskManager.Task::(prepare) Task=`0b876eae-370f-4d62-9ca8-281583cdbf16`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00875878334045', 'lastCheck': 1354451037.127305, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00298190116882', 'lastCheck': 1354451032.694037, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172996520996', 'lastCheck': 1354451034.8725891, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00732779502869', 'lastCheck': 1354451032.703685, 'code': 0, 'valid': True}} >Thread-27650::DEBUG::2012-12-02 14:24:00,772::task::568::TaskManager.Task::(_updateState) Task=`0b876eae-370f-4d62-9ca8-281583cdbf16`::moving from state preparing -> state finished >Thread-27650::DEBUG::2012-12-02 14:24:00,772::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27650::DEBUG::2012-12-02 14:24:00,772::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27650::DEBUG::2012-12-02 14:24:00,772::task::957::TaskManager.Task::(_decref) Task=`0b876eae-370f-4d62-9ca8-281583cdbf16`::ref 0 aborting False >Thread-27654::DEBUG::2012-12-02 14:24:05,103::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27654::DEBUG::2012-12-02 14:24:05,104::task::568::TaskManager.Task::(_updateState) Task=`8353f728-9dc1-4b30-a000-e0d1735bf031`::moving from state init -> state preparing >Thread-27654::INFO::2012-12-02 14:24:05,104::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27654::INFO::2012-12-02 14:24:05,104::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27654::DEBUG::2012-12-02 14:24:05,104::task::1151::TaskManager.Task::(prepare) Task=`8353f728-9dc1-4b30-a000-e0d1735bf031`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27654::DEBUG::2012-12-02 14:24:05,104::task::568::TaskManager.Task::(_updateState) Task=`8353f728-9dc1-4b30-a000-e0d1735bf031`::moving from state preparing -> state finished >Thread-27654::DEBUG::2012-12-02 14:24:05,104::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27654::DEBUG::2012-12-02 14:24:05,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27654::DEBUG::2012-12-02 14:24:05,105::task::957::TaskManager.Task::(_decref) Task=`8353f728-9dc1-4b30-a000-e0d1735bf031`::ref 0 aborting False >Thread-27655::DEBUG::2012-12-02 14:24:05,107::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27655::DEBUG::2012-12-02 14:24:05,107::task::568::TaskManager.Task::(_updateState) Task=`f354dc0c-812c-4df6-b66c-8fedf8a718ce`::moving from state init -> state preparing >Thread-27655::INFO::2012-12-02 14:24:05,108::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27655::DEBUG::2012-12-02 14:24:05,108::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`efcb1cf0-f275-4ecf-b622-4b356da19a03`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27655::DEBUG::2012-12-02 14:24:05,108::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27655::DEBUG::2012-12-02 14:24:05,108::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27655::DEBUG::2012-12-02 14:24:05,108::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`efcb1cf0-f275-4ecf-b622-4b356da19a03`::Granted request >Thread-27655::DEBUG::2012-12-02 14:24:05,108::task::794::TaskManager.Task::(resourceAcquired) Task=`f354dc0c-812c-4df6-b66c-8fedf8a718ce`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27655::DEBUG::2012-12-02 14:24:05,108::task::957::TaskManager.Task::(_decref) Task=`f354dc0c-812c-4df6-b66c-8fedf8a718ce`::ref 1 aborting False >Thread-27655::INFO::2012-12-02 14:24:05,113::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27655::DEBUG::2012-12-02 14:24:05,113::task::1151::TaskManager.Task::(prepare) Task=`f354dc0c-812c-4df6-b66c-8fedf8a718ce`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27655::DEBUG::2012-12-02 14:24:05,113::task::568::TaskManager.Task::(_updateState) Task=`f354dc0c-812c-4df6-b66c-8fedf8a718ce`::moving from state preparing -> state finished >Thread-27655::DEBUG::2012-12-02 14:24:05,113::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27655::DEBUG::2012-12-02 14:24:05,113::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27655::DEBUG::2012-12-02 14:24:05,113::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27655::DEBUG::2012-12-02 14:24:05,113::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27655::DEBUG::2012-12-02 14:24:05,113::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27655::DEBUG::2012-12-02 14:24:05,113::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27655::DEBUG::2012-12-02 14:24:05,114::task::957::TaskManager.Task::(_decref) Task=`f354dc0c-812c-4df6-b66c-8fedf8a718ce`::ref 0 aborting False >Thread-27658::DEBUG::2012-12-02 14:24:10,876::task::568::TaskManager.Task::(_updateState) Task=`66f5bdc6-a3f7-47d3-bd2a-dbcd18112b12`::moving from state init -> state preparing >Thread-27658::INFO::2012-12-02 14:24:10,876::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27658::INFO::2012-12-02 14:24:10,876::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00671696662903', 'lastCheck': 1354451047.1431899, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0019371509552', 'lastCheck': 1354451042.7041709, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00158214569092', 'lastCheck': 1354451044.8820629, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00732517242432', 'lastCheck': 1354451042.7195251, 'code': 0, 'valid': True}} >Thread-27658::DEBUG::2012-12-02 14:24:10,877::task::1151::TaskManager.Task::(prepare) Task=`66f5bdc6-a3f7-47d3-bd2a-dbcd18112b12`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00671696662903', 'lastCheck': 1354451047.1431899, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0019371509552', 'lastCheck': 1354451042.7041709, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00158214569092', 'lastCheck': 1354451044.8820629, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00732517242432', 'lastCheck': 1354451042.7195251, 'code': 0, 'valid': True}} >Thread-27658::DEBUG::2012-12-02 14:24:10,877::task::568::TaskManager.Task::(_updateState) Task=`66f5bdc6-a3f7-47d3-bd2a-dbcd18112b12`::moving from state preparing -> state finished >Thread-27658::DEBUG::2012-12-02 14:24:10,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27658::DEBUG::2012-12-02 14:24:10,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27658::DEBUG::2012-12-02 14:24:10,877::task::957::TaskManager.Task::(_decref) Task=`66f5bdc6-a3f7-47d3-bd2a-dbcd18112b12`::ref 0 aborting False >Thread-27662::DEBUG::2012-12-02 14:24:15,275::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27662::DEBUG::2012-12-02 14:24:15,276::task::568::TaskManager.Task::(_updateState) Task=`cbb7d234-8d25-42a7-a69e-7bb2863e72cf`::moving from state init -> state preparing >Thread-27662::INFO::2012-12-02 14:24:15,276::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27662::INFO::2012-12-02 14:24:15,276::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27662::DEBUG::2012-12-02 14:24:15,276::task::1151::TaskManager.Task::(prepare) Task=`cbb7d234-8d25-42a7-a69e-7bb2863e72cf`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27662::DEBUG::2012-12-02 14:24:15,276::task::568::TaskManager.Task::(_updateState) Task=`cbb7d234-8d25-42a7-a69e-7bb2863e72cf`::moving from state preparing -> state finished >Thread-27662::DEBUG::2012-12-02 14:24:15,277::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27662::DEBUG::2012-12-02 14:24:15,277::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27662::DEBUG::2012-12-02 14:24:15,277::task::957::TaskManager.Task::(_decref) Task=`cbb7d234-8d25-42a7-a69e-7bb2863e72cf`::ref 0 aborting False >Thread-27663::DEBUG::2012-12-02 14:24:15,279::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27663::DEBUG::2012-12-02 14:24:15,280::task::568::TaskManager.Task::(_updateState) Task=`8c1a6b49-3b7f-4101-875f-d8a7906fcbe1`::moving from state init -> state preparing >Thread-27663::INFO::2012-12-02 14:24:15,280::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27663::DEBUG::2012-12-02 14:24:15,280::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b8ab0bbe-8431-4295-94d7-df50203c563f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27663::DEBUG::2012-12-02 14:24:15,280::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27663::DEBUG::2012-12-02 14:24:15,280::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27663::DEBUG::2012-12-02 14:24:15,281::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b8ab0bbe-8431-4295-94d7-df50203c563f`::Granted request >Thread-27663::DEBUG::2012-12-02 14:24:15,281::task::794::TaskManager.Task::(resourceAcquired) Task=`8c1a6b49-3b7f-4101-875f-d8a7906fcbe1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27663::DEBUG::2012-12-02 14:24:15,281::task::957::TaskManager.Task::(_decref) Task=`8c1a6b49-3b7f-4101-875f-d8a7906fcbe1`::ref 1 aborting False >Thread-27663::INFO::2012-12-02 14:24:15,287::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27663::DEBUG::2012-12-02 14:24:15,287::task::1151::TaskManager.Task::(prepare) Task=`8c1a6b49-3b7f-4101-875f-d8a7906fcbe1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27663::DEBUG::2012-12-02 14:24:15,287::task::568::TaskManager.Task::(_updateState) Task=`8c1a6b49-3b7f-4101-875f-d8a7906fcbe1`::moving from state preparing -> state finished >Thread-27663::DEBUG::2012-12-02 14:24:15,287::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27663::DEBUG::2012-12-02 14:24:15,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27663::DEBUG::2012-12-02 14:24:15,287::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27663::DEBUG::2012-12-02 14:24:15,287::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27663::DEBUG::2012-12-02 14:24:15,287::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27663::DEBUG::2012-12-02 14:24:15,287::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27663::DEBUG::2012-12-02 14:24:15,288::task::957::TaskManager.Task::(_decref) Task=`8c1a6b49-3b7f-4101-875f-d8a7906fcbe1`::ref 0 aborting False >Thread-27666::DEBUG::2012-12-02 14:24:20,990::task::568::TaskManager.Task::(_updateState) Task=`9e0ebd3a-21e7-40bf-a44a-3f4ac201fd1e`::moving from state init -> state preparing >Thread-27666::INFO::2012-12-02 14:24:20,990::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27666::INFO::2012-12-02 14:24:20,990::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007807970047', 'lastCheck': 1354451057.1593411, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00351691246033', 'lastCheck': 1354451052.7170119, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00284290313721', 'lastCheck': 1354451054.8944869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00688290596008', 'lastCheck': 1354451052.734787, 'code': 0, 'valid': True}} >Thread-27666::DEBUG::2012-12-02 14:24:20,991::task::1151::TaskManager.Task::(prepare) Task=`9e0ebd3a-21e7-40bf-a44a-3f4ac201fd1e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007807970047', 'lastCheck': 1354451057.1593411, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00351691246033', 'lastCheck': 1354451052.7170119, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00284290313721', 'lastCheck': 1354451054.8944869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00688290596008', 'lastCheck': 1354451052.734787, 'code': 0, 'valid': True}} >Thread-27666::DEBUG::2012-12-02 14:24:20,991::task::568::TaskManager.Task::(_updateState) Task=`9e0ebd3a-21e7-40bf-a44a-3f4ac201fd1e`::moving from state preparing -> state finished >Thread-27666::DEBUG::2012-12-02 14:24:20,991::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27666::DEBUG::2012-12-02 14:24:20,991::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27666::DEBUG::2012-12-02 14:24:20,991::task::957::TaskManager.Task::(_decref) Task=`9e0ebd3a-21e7-40bf-a44a-3f4ac201fd1e`::ref 0 aborting False >Thread-27670::DEBUG::2012-12-02 14:24:25,380::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27670::DEBUG::2012-12-02 14:24:25,380::task::568::TaskManager.Task::(_updateState) Task=`23342a2a-1de6-477e-a863-0f06bc7c58ba`::moving from state init -> state preparing >Thread-27670::INFO::2012-12-02 14:24:25,381::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27670::INFO::2012-12-02 14:24:25,381::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27670::DEBUG::2012-12-02 14:24:25,381::task::1151::TaskManager.Task::(prepare) Task=`23342a2a-1de6-477e-a863-0f06bc7c58ba`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27670::DEBUG::2012-12-02 14:24:25,381::task::568::TaskManager.Task::(_updateState) Task=`23342a2a-1de6-477e-a863-0f06bc7c58ba`::moving from state preparing -> state finished >Thread-27670::DEBUG::2012-12-02 14:24:25,381::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27670::DEBUG::2012-12-02 14:24:25,381::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27670::DEBUG::2012-12-02 14:24:25,381::task::957::TaskManager.Task::(_decref) Task=`23342a2a-1de6-477e-a863-0f06bc7c58ba`::ref 0 aborting False >Thread-27671::DEBUG::2012-12-02 14:24:25,384::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27671::DEBUG::2012-12-02 14:24:25,384::task::568::TaskManager.Task::(_updateState) Task=`ee7d34af-a923-42a9-9114-edaa5ef4d26e`::moving from state init -> state preparing >Thread-27671::INFO::2012-12-02 14:24:25,385::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27671::DEBUG::2012-12-02 14:24:25,385::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`76b0d037-5fe0-4cc2-bcfe-4d8fb197fc1b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27671::DEBUG::2012-12-02 14:24:25,385::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27671::DEBUG::2012-12-02 14:24:25,385::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27671::DEBUG::2012-12-02 14:24:25,385::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`76b0d037-5fe0-4cc2-bcfe-4d8fb197fc1b`::Granted request >Thread-27671::DEBUG::2012-12-02 14:24:25,386::task::794::TaskManager.Task::(resourceAcquired) Task=`ee7d34af-a923-42a9-9114-edaa5ef4d26e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27671::DEBUG::2012-12-02 14:24:25,386::task::957::TaskManager.Task::(_decref) Task=`ee7d34af-a923-42a9-9114-edaa5ef4d26e`::ref 1 aborting False >Thread-27671::INFO::2012-12-02 14:24:25,392::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27671::DEBUG::2012-12-02 14:24:25,392::task::1151::TaskManager.Task::(prepare) Task=`ee7d34af-a923-42a9-9114-edaa5ef4d26e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27671::DEBUG::2012-12-02 14:24:25,392::task::568::TaskManager.Task::(_updateState) Task=`ee7d34af-a923-42a9-9114-edaa5ef4d26e`::moving from state preparing -> state finished >Thread-27671::DEBUG::2012-12-02 14:24:25,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27671::DEBUG::2012-12-02 14:24:25,393::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27671::DEBUG::2012-12-02 14:24:25,393::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27671::DEBUG::2012-12-02 14:24:25,393::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27671::DEBUG::2012-12-02 14:24:25,393::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27671::DEBUG::2012-12-02 14:24:25,393::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27671::DEBUG::2012-12-02 14:24:25,393::task::957::TaskManager.Task::(_decref) Task=`ee7d34af-a923-42a9-9114-edaa5ef4d26e`::ref 0 aborting False >Thread-27674::DEBUG::2012-12-02 14:24:31,079::task::568::TaskManager.Task::(_updateState) Task=`cc5017bc-2dc1-485f-a1f9-c75e8f16d07d`::moving from state init -> state preparing >Thread-27674::INFO::2012-12-02 14:24:31,079::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27674::INFO::2012-12-02 14:24:31,079::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0082380771637', 'lastCheck': 1354451067.178571, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00190114974976', 'lastCheck': 1354451062.7315171, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00184893608093', 'lastCheck': 1354451064.905381, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00707411766052', 'lastCheck': 1354451062.7505951, 'code': 0, 'valid': True}} >Thread-27674::DEBUG::2012-12-02 14:24:31,079::task::1151::TaskManager.Task::(prepare) Task=`cc5017bc-2dc1-485f-a1f9-c75e8f16d07d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0082380771637', 'lastCheck': 1354451067.178571, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00190114974976', 'lastCheck': 1354451062.7315171, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00184893608093', 'lastCheck': 1354451064.905381, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00707411766052', 'lastCheck': 1354451062.7505951, 'code': 0, 'valid': True}} >Thread-27674::DEBUG::2012-12-02 14:24:31,079::task::568::TaskManager.Task::(_updateState) Task=`cc5017bc-2dc1-485f-a1f9-c75e8f16d07d`::moving from state preparing -> state finished >Thread-27674::DEBUG::2012-12-02 14:24:31,079::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27674::DEBUG::2012-12-02 14:24:31,080::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27674::DEBUG::2012-12-02 14:24:31,080::task::957::TaskManager.Task::(_decref) Task=`cc5017bc-2dc1-485f-a1f9-c75e8f16d07d`::ref 0 aborting False >Thread-27678::DEBUG::2012-12-02 14:24:35,486::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27678::DEBUG::2012-12-02 14:24:35,487::task::568::TaskManager.Task::(_updateState) Task=`7810dae0-99fb-4be4-a233-b53d45a6e84b`::moving from state init -> state preparing >Thread-27678::INFO::2012-12-02 14:24:35,487::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27678::INFO::2012-12-02 14:24:35,487::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27678::DEBUG::2012-12-02 14:24:35,487::task::1151::TaskManager.Task::(prepare) Task=`7810dae0-99fb-4be4-a233-b53d45a6e84b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27678::DEBUG::2012-12-02 14:24:35,487::task::568::TaskManager.Task::(_updateState) Task=`7810dae0-99fb-4be4-a233-b53d45a6e84b`::moving from state preparing -> state finished >Thread-27678::DEBUG::2012-12-02 14:24:35,488::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27678::DEBUG::2012-12-02 14:24:35,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27678::DEBUG::2012-12-02 14:24:35,488::task::957::TaskManager.Task::(_decref) Task=`7810dae0-99fb-4be4-a233-b53d45a6e84b`::ref 0 aborting False >Thread-27679::DEBUG::2012-12-02 14:24:35,490::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27679::DEBUG::2012-12-02 14:24:35,491::task::568::TaskManager.Task::(_updateState) Task=`b711cfa7-de69-46d3-ac19-c26c3f0382ad`::moving from state init -> state preparing >Thread-27679::INFO::2012-12-02 14:24:35,491::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27679::DEBUG::2012-12-02 14:24:35,491::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0dfa7e9c-f3d6-4fff-8f01-6d8b7ba6fd9c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27679::DEBUG::2012-12-02 14:24:35,491::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27679::DEBUG::2012-12-02 14:24:35,491::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27679::DEBUG::2012-12-02 14:24:35,491::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0dfa7e9c-f3d6-4fff-8f01-6d8b7ba6fd9c`::Granted request >Thread-27679::DEBUG::2012-12-02 14:24:35,492::task::794::TaskManager.Task::(resourceAcquired) Task=`b711cfa7-de69-46d3-ac19-c26c3f0382ad`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27679::DEBUG::2012-12-02 14:24:35,492::task::957::TaskManager.Task::(_decref) Task=`b711cfa7-de69-46d3-ac19-c26c3f0382ad`::ref 1 aborting False >Thread-27679::INFO::2012-12-02 14:24:35,498::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27679::DEBUG::2012-12-02 14:24:35,498::task::1151::TaskManager.Task::(prepare) Task=`b711cfa7-de69-46d3-ac19-c26c3f0382ad`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27679::DEBUG::2012-12-02 14:24:35,498::task::568::TaskManager.Task::(_updateState) Task=`b711cfa7-de69-46d3-ac19-c26c3f0382ad`::moving from state preparing -> state finished >Thread-27679::DEBUG::2012-12-02 14:24:35,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27679::DEBUG::2012-12-02 14:24:35,498::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27679::DEBUG::2012-12-02 14:24:35,498::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27679::DEBUG::2012-12-02 14:24:35,498::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27679::DEBUG::2012-12-02 14:24:35,498::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27679::DEBUG::2012-12-02 14:24:35,498::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27679::DEBUG::2012-12-02 14:24:35,498::task::957::TaskManager.Task::(_decref) Task=`b711cfa7-de69-46d3-ac19-c26c3f0382ad`::ref 0 aborting False >Thread-27682::DEBUG::2012-12-02 14:24:41,183::task::568::TaskManager.Task::(_updateState) Task=`6efb0740-ca4e-43e2-9ce6-74ee2ec5cf94`::moving from state init -> state preparing >Thread-27682::INFO::2012-12-02 14:24:41,184::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27682::INFO::2012-12-02 14:24:41,184::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00703406333923', 'lastCheck': 1354451077.1966469, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00268912315369', 'lastCheck': 1354451072.743469, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00277185440063', 'lastCheck': 1354451074.916394, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00652098655701', 'lastCheck': 1354451072.7656331, 'code': 0, 'valid': True}} >Thread-27682::DEBUG::2012-12-02 14:24:41,184::task::1151::TaskManager.Task::(prepare) Task=`6efb0740-ca4e-43e2-9ce6-74ee2ec5cf94`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00703406333923', 'lastCheck': 1354451077.1966469, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00268912315369', 'lastCheck': 1354451072.743469, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00277185440063', 'lastCheck': 1354451074.916394, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00652098655701', 'lastCheck': 1354451072.7656331, 'code': 0, 'valid': True}} >Thread-27682::DEBUG::2012-12-02 14:24:41,184::task::568::TaskManager.Task::(_updateState) Task=`6efb0740-ca4e-43e2-9ce6-74ee2ec5cf94`::moving from state preparing -> state finished >Thread-27682::DEBUG::2012-12-02 14:24:41,184::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27682::DEBUG::2012-12-02 14:24:41,184::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27682::DEBUG::2012-12-02 14:24:41,185::task::957::TaskManager.Task::(_decref) Task=`6efb0740-ca4e-43e2-9ce6-74ee2ec5cf94`::ref 0 aborting False >Thread-27686::DEBUG::2012-12-02 14:24:45,625::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27686::DEBUG::2012-12-02 14:24:45,625::task::568::TaskManager.Task::(_updateState) Task=`5e095abb-3ced-4645-a1e6-7243068aa438`::moving from state init -> state preparing >Thread-27686::INFO::2012-12-02 14:24:45,625::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27686::INFO::2012-12-02 14:24:45,625::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27686::DEBUG::2012-12-02 14:24:45,626::task::1151::TaskManager.Task::(prepare) Task=`5e095abb-3ced-4645-a1e6-7243068aa438`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27686::DEBUG::2012-12-02 14:24:45,626::task::568::TaskManager.Task::(_updateState) Task=`5e095abb-3ced-4645-a1e6-7243068aa438`::moving from state preparing -> state finished >Thread-27686::DEBUG::2012-12-02 14:24:45,626::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27686::DEBUG::2012-12-02 14:24:45,626::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27686::DEBUG::2012-12-02 14:24:45,626::task::957::TaskManager.Task::(_decref) Task=`5e095abb-3ced-4645-a1e6-7243068aa438`::ref 0 aborting False >Thread-27687::DEBUG::2012-12-02 14:24:45,629::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27687::DEBUG::2012-12-02 14:24:45,629::task::568::TaskManager.Task::(_updateState) Task=`cd4aca8e-bc03-4cd4-ae5f-49417d486439`::moving from state init -> state preparing >Thread-27687::INFO::2012-12-02 14:24:45,629::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27687::DEBUG::2012-12-02 14:24:45,629::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c31bc104-810c-40cd-b0fb-109cb7f7e3fb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27687::DEBUG::2012-12-02 14:24:45,629::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27687::DEBUG::2012-12-02 14:24:45,630::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27687::DEBUG::2012-12-02 14:24:45,630::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c31bc104-810c-40cd-b0fb-109cb7f7e3fb`::Granted request >Thread-27687::DEBUG::2012-12-02 14:24:45,630::task::794::TaskManager.Task::(resourceAcquired) Task=`cd4aca8e-bc03-4cd4-ae5f-49417d486439`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27687::DEBUG::2012-12-02 14:24:45,630::task::957::TaskManager.Task::(_decref) Task=`cd4aca8e-bc03-4cd4-ae5f-49417d486439`::ref 1 aborting False >Thread-27687::INFO::2012-12-02 14:24:45,636::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27687::DEBUG::2012-12-02 14:24:45,636::task::1151::TaskManager.Task::(prepare) Task=`cd4aca8e-bc03-4cd4-ae5f-49417d486439`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27687::DEBUG::2012-12-02 14:24:45,636::task::568::TaskManager.Task::(_updateState) Task=`cd4aca8e-bc03-4cd4-ae5f-49417d486439`::moving from state preparing -> state finished >Thread-27687::DEBUG::2012-12-02 14:24:45,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27687::DEBUG::2012-12-02 14:24:45,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27687::DEBUG::2012-12-02 14:24:45,637::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27687::DEBUG::2012-12-02 14:24:45,637::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27687::DEBUG::2012-12-02 14:24:45,637::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27687::DEBUG::2012-12-02 14:24:45,637::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27687::DEBUG::2012-12-02 14:24:45,637::task::957::TaskManager.Task::(_decref) Task=`cd4aca8e-bc03-4cd4-ae5f-49417d486439`::ref 0 aborting False >Thread-27690::DEBUG::2012-12-02 14:24:51,304::task::568::TaskManager.Task::(_updateState) Task=`f55e7d8d-da59-43bf-8add-11c2bfa65068`::moving from state init -> state preparing >Thread-27690::INFO::2012-12-02 14:24:51,304::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27690::INFO::2012-12-02 14:24:51,305::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.008141040802', 'lastCheck': 1354451087.216037, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00192713737488', 'lastCheck': 1354451082.756021, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172781944275', 'lastCheck': 1354451084.9269381, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778794288635', 'lastCheck': 1354451082.7842491, 'code': 0, 'valid': True}} >Thread-27690::DEBUG::2012-12-02 14:24:51,305::task::1151::TaskManager.Task::(prepare) Task=`f55e7d8d-da59-43bf-8add-11c2bfa65068`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.008141040802', 'lastCheck': 1354451087.216037, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00192713737488', 'lastCheck': 1354451082.756021, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172781944275', 'lastCheck': 1354451084.9269381, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778794288635', 'lastCheck': 1354451082.7842491, 'code': 0, 'valid': True}} >Thread-27690::DEBUG::2012-12-02 14:24:51,305::task::568::TaskManager.Task::(_updateState) Task=`f55e7d8d-da59-43bf-8add-11c2bfa65068`::moving from state preparing -> state finished >Thread-27690::DEBUG::2012-12-02 14:24:51,305::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27690::DEBUG::2012-12-02 14:24:51,305::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27690::DEBUG::2012-12-02 14:24:51,305::task::957::TaskManager.Task::(_decref) Task=`f55e7d8d-da59-43bf-8add-11c2bfa65068`::ref 0 aborting False >Thread-27694::DEBUG::2012-12-02 14:24:55,760::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27694::DEBUG::2012-12-02 14:24:55,760::task::568::TaskManager.Task::(_updateState) Task=`0fe84ccb-ccfd-4393-a5d4-94c99130e6c5`::moving from state init -> state preparing >Thread-27694::INFO::2012-12-02 14:24:55,761::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27694::INFO::2012-12-02 14:24:55,761::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27694::DEBUG::2012-12-02 14:24:55,761::task::1151::TaskManager.Task::(prepare) Task=`0fe84ccb-ccfd-4393-a5d4-94c99130e6c5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27694::DEBUG::2012-12-02 14:24:55,761::task::568::TaskManager.Task::(_updateState) Task=`0fe84ccb-ccfd-4393-a5d4-94c99130e6c5`::moving from state preparing -> state finished >Thread-27694::DEBUG::2012-12-02 14:24:55,761::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27694::DEBUG::2012-12-02 14:24:55,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27694::DEBUG::2012-12-02 14:24:55,761::task::957::TaskManager.Task::(_decref) Task=`0fe84ccb-ccfd-4393-a5d4-94c99130e6c5`::ref 0 aborting False >Thread-27695::DEBUG::2012-12-02 14:24:55,764::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27695::DEBUG::2012-12-02 14:24:55,764::task::568::TaskManager.Task::(_updateState) Task=`c431e2be-797d-4c20-9477-109f3ad29865`::moving from state init -> state preparing >Thread-27695::INFO::2012-12-02 14:24:55,764::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27695::DEBUG::2012-12-02 14:24:55,765::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c076c934-eac2-400f-9bba-8c1359af6580`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27695::DEBUG::2012-12-02 14:24:55,765::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27695::DEBUG::2012-12-02 14:24:55,765::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27695::DEBUG::2012-12-02 14:24:55,765::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c076c934-eac2-400f-9bba-8c1359af6580`::Granted request >Thread-27695::DEBUG::2012-12-02 14:24:55,765::task::794::TaskManager.Task::(resourceAcquired) Task=`c431e2be-797d-4c20-9477-109f3ad29865`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27695::DEBUG::2012-12-02 14:24:55,765::task::957::TaskManager.Task::(_decref) Task=`c431e2be-797d-4c20-9477-109f3ad29865`::ref 1 aborting False >Thread-27695::INFO::2012-12-02 14:24:55,770::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27695::DEBUG::2012-12-02 14:24:55,770::task::1151::TaskManager.Task::(prepare) Task=`c431e2be-797d-4c20-9477-109f3ad29865`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27695::DEBUG::2012-12-02 14:24:55,770::task::568::TaskManager.Task::(_updateState) Task=`c431e2be-797d-4c20-9477-109f3ad29865`::moving from state preparing -> state finished >Thread-27695::DEBUG::2012-12-02 14:24:55,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27695::DEBUG::2012-12-02 14:24:55,770::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27695::DEBUG::2012-12-02 14:24:55,770::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27695::DEBUG::2012-12-02 14:24:55,770::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27695::DEBUG::2012-12-02 14:24:55,770::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27695::DEBUG::2012-12-02 14:24:55,771::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27695::DEBUG::2012-12-02 14:24:55,771::task::957::TaskManager.Task::(_decref) Task=`c431e2be-797d-4c20-9477-109f3ad29865`::ref 0 aborting False >Thread-27698::DEBUG::2012-12-02 14:25:01,365::task::568::TaskManager.Task::(_updateState) Task=`c49afd30-2ea7-46fe-b467-a4cc6eef00d6`::moving from state init -> state preparing >Thread-27698::INFO::2012-12-02 14:25:01,365::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27698::INFO::2012-12-02 14:25:01,366::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00816011428833', 'lastCheck': 1354451097.2332799, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00190901756287', 'lastCheck': 1354451092.7676351, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00157403945923', 'lastCheck': 1354451094.937305, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00725507736206', 'lastCheck': 1354451092.7985711, 'code': 0, 'valid': True}} >Thread-27698::DEBUG::2012-12-02 14:25:01,366::task::1151::TaskManager.Task::(prepare) Task=`c49afd30-2ea7-46fe-b467-a4cc6eef00d6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00816011428833', 'lastCheck': 1354451097.2332799, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00190901756287', 'lastCheck': 1354451092.7676351, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00157403945923', 'lastCheck': 1354451094.937305, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00725507736206', 'lastCheck': 1354451092.7985711, 'code': 0, 'valid': True}} >Thread-27698::DEBUG::2012-12-02 14:25:01,366::task::568::TaskManager.Task::(_updateState) Task=`c49afd30-2ea7-46fe-b467-a4cc6eef00d6`::moving from state preparing -> state finished >Thread-27698::DEBUG::2012-12-02 14:25:01,366::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27698::DEBUG::2012-12-02 14:25:01,366::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27698::DEBUG::2012-12-02 14:25:01,366::task::957::TaskManager.Task::(_decref) Task=`c49afd30-2ea7-46fe-b467-a4cc6eef00d6`::ref 0 aborting False >Thread-27702::DEBUG::2012-12-02 14:25:05,873::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27702::DEBUG::2012-12-02 14:25:05,873::task::568::TaskManager.Task::(_updateState) Task=`99ce3337-9093-42ad-9441-bc5e8150ba1c`::moving from state init -> state preparing >Thread-27702::INFO::2012-12-02 14:25:05,874::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27702::INFO::2012-12-02 14:25:05,874::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27702::DEBUG::2012-12-02 14:25:05,874::task::1151::TaskManager.Task::(prepare) Task=`99ce3337-9093-42ad-9441-bc5e8150ba1c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27702::DEBUG::2012-12-02 14:25:05,874::task::568::TaskManager.Task::(_updateState) Task=`99ce3337-9093-42ad-9441-bc5e8150ba1c`::moving from state preparing -> state finished >Thread-27702::DEBUG::2012-12-02 14:25:05,874::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27702::DEBUG::2012-12-02 14:25:05,874::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27702::DEBUG::2012-12-02 14:25:05,874::task::957::TaskManager.Task::(_decref) Task=`99ce3337-9093-42ad-9441-bc5e8150ba1c`::ref 0 aborting False >Thread-27703::DEBUG::2012-12-02 14:25:05,877::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27703::DEBUG::2012-12-02 14:25:05,877::task::568::TaskManager.Task::(_updateState) Task=`d363d713-b963-4445-a0a0-b3ecd5b26734`::moving from state init -> state preparing >Thread-27703::INFO::2012-12-02 14:25:05,877::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27703::DEBUG::2012-12-02 14:25:05,878::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`56122534-f324-4a27-ad89-07ca442381ab`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27703::DEBUG::2012-12-02 14:25:05,878::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27703::DEBUG::2012-12-02 14:25:05,878::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27703::DEBUG::2012-12-02 14:25:05,878::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`56122534-f324-4a27-ad89-07ca442381ab`::Granted request >Thread-27703::DEBUG::2012-12-02 14:25:05,878::task::794::TaskManager.Task::(resourceAcquired) Task=`d363d713-b963-4445-a0a0-b3ecd5b26734`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27703::DEBUG::2012-12-02 14:25:05,878::task::957::TaskManager.Task::(_decref) Task=`d363d713-b963-4445-a0a0-b3ecd5b26734`::ref 1 aborting False >Thread-27703::INFO::2012-12-02 14:25:05,885::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27703::DEBUG::2012-12-02 14:25:05,885::task::1151::TaskManager.Task::(prepare) Task=`d363d713-b963-4445-a0a0-b3ecd5b26734`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27703::DEBUG::2012-12-02 14:25:05,885::task::568::TaskManager.Task::(_updateState) Task=`d363d713-b963-4445-a0a0-b3ecd5b26734`::moving from state preparing -> state finished >Thread-27703::DEBUG::2012-12-02 14:25:05,885::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27703::DEBUG::2012-12-02 14:25:05,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27703::DEBUG::2012-12-02 14:25:05,885::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27703::DEBUG::2012-12-02 14:25:05,886::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27703::DEBUG::2012-12-02 14:25:05,886::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27703::DEBUG::2012-12-02 14:25:05,886::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27703::DEBUG::2012-12-02 14:25:05,886::task::957::TaskManager.Task::(_decref) Task=`d363d713-b963-4445-a0a0-b3ecd5b26734`::ref 0 aborting False >Thread-27706::DEBUG::2012-12-02 14:25:11,521::task::568::TaskManager.Task::(_updateState) Task=`f0c86ed8-acdb-467d-80db-148a7c150a08`::moving from state init -> state preparing >Thread-27706::INFO::2012-12-02 14:25:11,521::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27706::INFO::2012-12-02 14:25:11,521::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00623893737793', 'lastCheck': 1354451107.2471581, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.001797914505', 'lastCheck': 1354451102.7793031, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00194406509399', 'lastCheck': 1354451104.9463611, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00867319107056', 'lastCheck': 1354451102.815505, 'code': 0, 'valid': True}} >Thread-27706::DEBUG::2012-12-02 14:25:11,522::task::1151::TaskManager.Task::(prepare) Task=`f0c86ed8-acdb-467d-80db-148a7c150a08`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00623893737793', 'lastCheck': 1354451107.2471581, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.001797914505', 'lastCheck': 1354451102.7793031, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00194406509399', 'lastCheck': 1354451104.9463611, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00867319107056', 'lastCheck': 1354451102.815505, 'code': 0, 'valid': True}} >Thread-27706::DEBUG::2012-12-02 14:25:11,522::task::568::TaskManager.Task::(_updateState) Task=`f0c86ed8-acdb-467d-80db-148a7c150a08`::moving from state preparing -> state finished >Thread-27706::DEBUG::2012-12-02 14:25:11,522::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27706::DEBUG::2012-12-02 14:25:11,522::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27706::DEBUG::2012-12-02 14:25:11,522::task::957::TaskManager.Task::(_decref) Task=`f0c86ed8-acdb-467d-80db-148a7c150a08`::ref 0 aborting False >Thread-27710::DEBUG::2012-12-02 14:25:16,010::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27710::DEBUG::2012-12-02 14:25:16,011::task::568::TaskManager.Task::(_updateState) Task=`a772ffce-44c2-41a9-993e-8faf1621029e`::moving from state init -> state preparing >Thread-27710::INFO::2012-12-02 14:25:16,011::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27710::INFO::2012-12-02 14:25:16,011::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27710::DEBUG::2012-12-02 14:25:16,011::task::1151::TaskManager.Task::(prepare) Task=`a772ffce-44c2-41a9-993e-8faf1621029e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27710::DEBUG::2012-12-02 14:25:16,012::task::568::TaskManager.Task::(_updateState) Task=`a772ffce-44c2-41a9-993e-8faf1621029e`::moving from state preparing -> state finished >Thread-27710::DEBUG::2012-12-02 14:25:16,012::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27710::DEBUG::2012-12-02 14:25:16,012::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27710::DEBUG::2012-12-02 14:25:16,012::task::957::TaskManager.Task::(_decref) Task=`a772ffce-44c2-41a9-993e-8faf1621029e`::ref 0 aborting False >Thread-27711::DEBUG::2012-12-02 14:25:16,014::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27711::DEBUG::2012-12-02 14:25:16,015::task::568::TaskManager.Task::(_updateState) Task=`382b0d26-7354-40db-8497-8d931f228646`::moving from state init -> state preparing >Thread-27711::INFO::2012-12-02 14:25:16,015::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27711::DEBUG::2012-12-02 14:25:16,015::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c4224352-2e65-4811-a438-8220e94c7b07`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27711::DEBUG::2012-12-02 14:25:16,015::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27711::DEBUG::2012-12-02 14:25:16,016::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27711::DEBUG::2012-12-02 14:25:16,016::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c4224352-2e65-4811-a438-8220e94c7b07`::Granted request >Thread-27711::DEBUG::2012-12-02 14:25:16,016::task::794::TaskManager.Task::(resourceAcquired) Task=`382b0d26-7354-40db-8497-8d931f228646`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27711::DEBUG::2012-12-02 14:25:16,016::task::957::TaskManager.Task::(_decref) Task=`382b0d26-7354-40db-8497-8d931f228646`::ref 1 aborting False >Thread-27711::INFO::2012-12-02 14:25:16,023::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27711::DEBUG::2012-12-02 14:25:16,023::task::1151::TaskManager.Task::(prepare) Task=`382b0d26-7354-40db-8497-8d931f228646`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27711::DEBUG::2012-12-02 14:25:16,023::task::568::TaskManager.Task::(_updateState) Task=`382b0d26-7354-40db-8497-8d931f228646`::moving from state preparing -> state finished >Thread-27711::DEBUG::2012-12-02 14:25:16,023::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27711::DEBUG::2012-12-02 14:25:16,023::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27711::DEBUG::2012-12-02 14:25:16,023::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27711::DEBUG::2012-12-02 14:25:16,023::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27711::DEBUG::2012-12-02 14:25:16,024::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27711::DEBUG::2012-12-02 14:25:16,024::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27711::DEBUG::2012-12-02 14:25:16,024::task::957::TaskManager.Task::(_decref) Task=`382b0d26-7354-40db-8497-8d931f228646`::ref 0 aborting False >Thread-27714::DEBUG::2012-12-02 14:25:21,616::task::568::TaskManager.Task::(_updateState) Task=`23db103c-491c-435e-b4bd-49e07eb068e2`::moving from state init -> state preparing >Thread-27714::INFO::2012-12-02 14:25:21,616::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27714::INFO::2012-12-02 14:25:21,616::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00617909431458', 'lastCheck': 1354451117.2617359, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00322604179382', 'lastCheck': 1354451112.79339, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00261807441711', 'lastCheck': 1354451114.958909, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00673508644104', 'lastCheck': 1354451112.831048, 'code': 0, 'valid': True}} >Thread-27714::DEBUG::2012-12-02 14:25:21,617::task::1151::TaskManager.Task::(prepare) Task=`23db103c-491c-435e-b4bd-49e07eb068e2`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00617909431458', 'lastCheck': 1354451117.2617359, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00322604179382', 'lastCheck': 1354451112.79339, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00261807441711', 'lastCheck': 1354451114.958909, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00673508644104', 'lastCheck': 1354451112.831048, 'code': 0, 'valid': True}} >Thread-27714::DEBUG::2012-12-02 14:25:21,617::task::568::TaskManager.Task::(_updateState) Task=`23db103c-491c-435e-b4bd-49e07eb068e2`::moving from state preparing -> state finished >Thread-27714::DEBUG::2012-12-02 14:25:21,617::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27714::DEBUG::2012-12-02 14:25:21,617::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27714::DEBUG::2012-12-02 14:25:21,617::task::957::TaskManager.Task::(_decref) Task=`23db103c-491c-435e-b4bd-49e07eb068e2`::ref 0 aborting False >Thread-27718::DEBUG::2012-12-02 14:25:26,139::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27718::DEBUG::2012-12-02 14:25:26,140::task::568::TaskManager.Task::(_updateState) Task=`84a60c12-5d2d-47dc-ab6b-72a056171597`::moving from state init -> state preparing >Thread-27718::INFO::2012-12-02 14:25:26,140::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27718::INFO::2012-12-02 14:25:26,140::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27718::DEBUG::2012-12-02 14:25:26,140::task::1151::TaskManager.Task::(prepare) Task=`84a60c12-5d2d-47dc-ab6b-72a056171597`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27718::DEBUG::2012-12-02 14:25:26,141::task::568::TaskManager.Task::(_updateState) Task=`84a60c12-5d2d-47dc-ab6b-72a056171597`::moving from state preparing -> state finished >Thread-27718::DEBUG::2012-12-02 14:25:26,141::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27718::DEBUG::2012-12-02 14:25:26,141::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27718::DEBUG::2012-12-02 14:25:26,141::task::957::TaskManager.Task::(_decref) Task=`84a60c12-5d2d-47dc-ab6b-72a056171597`::ref 0 aborting False >Thread-27719::DEBUG::2012-12-02 14:25:26,144::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27719::DEBUG::2012-12-02 14:25:26,144::task::568::TaskManager.Task::(_updateState) Task=`ea2fb555-41a9-4fa7-90fa-81f20e1e73de`::moving from state init -> state preparing >Thread-27719::INFO::2012-12-02 14:25:26,144::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27719::DEBUG::2012-12-02 14:25:26,144::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ce507fde-8cc0-4a25-804e-3e27ae6117d4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27719::DEBUG::2012-12-02 14:25:26,144::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27719::DEBUG::2012-12-02 14:25:26,145::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27719::DEBUG::2012-12-02 14:25:26,145::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ce507fde-8cc0-4a25-804e-3e27ae6117d4`::Granted request >Thread-27719::DEBUG::2012-12-02 14:25:26,145::task::794::TaskManager.Task::(resourceAcquired) Task=`ea2fb555-41a9-4fa7-90fa-81f20e1e73de`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27719::DEBUG::2012-12-02 14:25:26,145::task::957::TaskManager.Task::(_decref) Task=`ea2fb555-41a9-4fa7-90fa-81f20e1e73de`::ref 1 aborting False >Thread-27719::INFO::2012-12-02 14:25:26,150::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27719::DEBUG::2012-12-02 14:25:26,150::task::1151::TaskManager.Task::(prepare) Task=`ea2fb555-41a9-4fa7-90fa-81f20e1e73de`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27719::DEBUG::2012-12-02 14:25:26,150::task::568::TaskManager.Task::(_updateState) Task=`ea2fb555-41a9-4fa7-90fa-81f20e1e73de`::moving from state preparing -> state finished >Thread-27719::DEBUG::2012-12-02 14:25:26,150::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27719::DEBUG::2012-12-02 14:25:26,150::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27719::DEBUG::2012-12-02 14:25:26,150::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27719::DEBUG::2012-12-02 14:25:26,150::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27719::DEBUG::2012-12-02 14:25:26,150::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27719::DEBUG::2012-12-02 14:25:26,150::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27719::DEBUG::2012-12-02 14:25:26,150::task::957::TaskManager.Task::(_decref) Task=`ea2fb555-41a9-4fa7-90fa-81f20e1e73de`::ref 0 aborting False >Thread-27722::DEBUG::2012-12-02 14:25:31,680::task::568::TaskManager.Task::(_updateState) Task=`cbb88688-b8d8-43da-873e-d32f6d24b8c3`::moving from state init -> state preparing >Thread-27722::INFO::2012-12-02 14:25:31,680::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27722::INFO::2012-12-02 14:25:31,680::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0063750743866', 'lastCheck': 1354451127.2786901, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00210785865784', 'lastCheck': 1354451122.8074419, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181794166565', 'lastCheck': 1354451124.969599, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00336408615112', 'lastCheck': 1354451122.8421309, 'code': 0, 'valid': True}} >Thread-27722::DEBUG::2012-12-02 14:25:31,681::task::1151::TaskManager.Task::(prepare) Task=`cbb88688-b8d8-43da-873e-d32f6d24b8c3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0063750743866', 'lastCheck': 1354451127.2786901, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00210785865784', 'lastCheck': 1354451122.8074419, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181794166565', 'lastCheck': 1354451124.969599, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00336408615112', 'lastCheck': 1354451122.8421309, 'code': 0, 'valid': True}} >Thread-27722::DEBUG::2012-12-02 14:25:31,681::task::568::TaskManager.Task::(_updateState) Task=`cbb88688-b8d8-43da-873e-d32f6d24b8c3`::moving from state preparing -> state finished >Thread-27722::DEBUG::2012-12-02 14:25:31,681::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27722::DEBUG::2012-12-02 14:25:31,681::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27722::DEBUG::2012-12-02 14:25:31,681::task::957::TaskManager.Task::(_decref) Task=`cbb88688-b8d8-43da-873e-d32f6d24b8c3`::ref 0 aborting False >Thread-27726::DEBUG::2012-12-02 14:25:36,361::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27726::DEBUG::2012-12-02 14:25:36,361::task::568::TaskManager.Task::(_updateState) Task=`4166e0fc-338f-4129-b7b9-e826b558727e`::moving from state init -> state preparing >Thread-27726::INFO::2012-12-02 14:25:36,361::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27726::INFO::2012-12-02 14:25:36,361::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27726::DEBUG::2012-12-02 14:25:36,362::task::1151::TaskManager.Task::(prepare) Task=`4166e0fc-338f-4129-b7b9-e826b558727e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27726::DEBUG::2012-12-02 14:25:36,362::task::568::TaskManager.Task::(_updateState) Task=`4166e0fc-338f-4129-b7b9-e826b558727e`::moving from state preparing -> state finished >Thread-27726::DEBUG::2012-12-02 14:25:36,362::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27726::DEBUG::2012-12-02 14:25:36,362::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27726::DEBUG::2012-12-02 14:25:36,362::task::957::TaskManager.Task::(_decref) Task=`4166e0fc-338f-4129-b7b9-e826b558727e`::ref 0 aborting False >Thread-27727::DEBUG::2012-12-02 14:25:36,365::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27727::DEBUG::2012-12-02 14:25:36,365::task::568::TaskManager.Task::(_updateState) Task=`7d67480b-3ac4-4ef1-a0ea-3ff53ac422f0`::moving from state init -> state preparing >Thread-27727::INFO::2012-12-02 14:25:36,365::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27727::DEBUG::2012-12-02 14:25:36,365::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e4d89ec-0e64-4096-807b-485bc93b3625`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27727::DEBUG::2012-12-02 14:25:36,366::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27727::DEBUG::2012-12-02 14:25:36,366::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27727::DEBUG::2012-12-02 14:25:36,366::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e4d89ec-0e64-4096-807b-485bc93b3625`::Granted request >Thread-27727::DEBUG::2012-12-02 14:25:36,366::task::794::TaskManager.Task::(resourceAcquired) Task=`7d67480b-3ac4-4ef1-a0ea-3ff53ac422f0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27727::DEBUG::2012-12-02 14:25:36,366::task::957::TaskManager.Task::(_decref) Task=`7d67480b-3ac4-4ef1-a0ea-3ff53ac422f0`::ref 1 aborting False >Thread-27727::INFO::2012-12-02 14:25:36,373::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27727::DEBUG::2012-12-02 14:25:36,373::task::1151::TaskManager.Task::(prepare) Task=`7d67480b-3ac4-4ef1-a0ea-3ff53ac422f0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27727::DEBUG::2012-12-02 14:25:36,373::task::568::TaskManager.Task::(_updateState) Task=`7d67480b-3ac4-4ef1-a0ea-3ff53ac422f0`::moving from state preparing -> state finished >Thread-27727::DEBUG::2012-12-02 14:25:36,373::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27727::DEBUG::2012-12-02 14:25:36,373::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27727::DEBUG::2012-12-02 14:25:36,373::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27727::DEBUG::2012-12-02 14:25:36,374::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27727::DEBUG::2012-12-02 14:25:36,374::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27727::DEBUG::2012-12-02 14:25:36,374::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27727::DEBUG::2012-12-02 14:25:36,374::task::957::TaskManager.Task::(_decref) Task=`7d67480b-3ac4-4ef1-a0ea-3ff53ac422f0`::ref 0 aborting False >Thread-27730::DEBUG::2012-12-02 14:25:41,803::task::568::TaskManager.Task::(_updateState) Task=`a1314ffb-40ec-4003-98c1-c97c9836614a`::moving from state init -> state preparing >Thread-27730::INFO::2012-12-02 14:25:41,803::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27730::INFO::2012-12-02 14:25:41,803::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00884294509888', 'lastCheck': 1354451137.2995019, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00355696678162', 'lastCheck': 1354451132.8210621, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00323605537415', 'lastCheck': 1354451134.982655, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00720691680908', 'lastCheck': 1354451132.8570869, 'code': 0, 'valid': True}} >Thread-27730::DEBUG::2012-12-02 14:25:41,803::task::1151::TaskManager.Task::(prepare) Task=`a1314ffb-40ec-4003-98c1-c97c9836614a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00884294509888', 'lastCheck': 1354451137.2995019, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00355696678162', 'lastCheck': 1354451132.8210621, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00323605537415', 'lastCheck': 1354451134.982655, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00720691680908', 'lastCheck': 1354451132.8570869, 'code': 0, 'valid': True}} >Thread-27730::DEBUG::2012-12-02 14:25:41,803::task::568::TaskManager.Task::(_updateState) Task=`a1314ffb-40ec-4003-98c1-c97c9836614a`::moving from state preparing -> state finished >Thread-27730::DEBUG::2012-12-02 14:25:41,803::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27730::DEBUG::2012-12-02 14:25:41,804::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27730::DEBUG::2012-12-02 14:25:41,804::task::957::TaskManager.Task::(_decref) Task=`a1314ffb-40ec-4003-98c1-c97c9836614a`::ref 0 aborting False >Thread-27734::DEBUG::2012-12-02 14:25:46,491::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27734::DEBUG::2012-12-02 14:25:46,492::task::568::TaskManager.Task::(_updateState) Task=`ce633f0b-ad64-4349-b2d1-0f7a51c352bf`::moving from state init -> state preparing >Thread-27734::INFO::2012-12-02 14:25:46,492::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27734::INFO::2012-12-02 14:25:46,492::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27734::DEBUG::2012-12-02 14:25:46,492::task::1151::TaskManager.Task::(prepare) Task=`ce633f0b-ad64-4349-b2d1-0f7a51c352bf`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27734::DEBUG::2012-12-02 14:25:46,492::task::568::TaskManager.Task::(_updateState) Task=`ce633f0b-ad64-4349-b2d1-0f7a51c352bf`::moving from state preparing -> state finished >Thread-27734::DEBUG::2012-12-02 14:25:46,492::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27734::DEBUG::2012-12-02 14:25:46,493::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27734::DEBUG::2012-12-02 14:25:46,493::task::957::TaskManager.Task::(_decref) Task=`ce633f0b-ad64-4349-b2d1-0f7a51c352bf`::ref 0 aborting False >Thread-27735::DEBUG::2012-12-02 14:25:46,495::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27735::DEBUG::2012-12-02 14:25:46,495::task::568::TaskManager.Task::(_updateState) Task=`76a2e0da-3bfb-407c-9167-40cc3f732641`::moving from state init -> state preparing >Thread-27735::INFO::2012-12-02 14:25:46,496::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27735::DEBUG::2012-12-02 14:25:46,496::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9b8cf29b-7a70-4f03-aa04-d63a96dcbabb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27735::DEBUG::2012-12-02 14:25:46,496::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27735::DEBUG::2012-12-02 14:25:46,496::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27735::DEBUG::2012-12-02 14:25:46,496::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9b8cf29b-7a70-4f03-aa04-d63a96dcbabb`::Granted request >Thread-27735::DEBUG::2012-12-02 14:25:46,496::task::794::TaskManager.Task::(resourceAcquired) Task=`76a2e0da-3bfb-407c-9167-40cc3f732641`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27735::DEBUG::2012-12-02 14:25:46,497::task::957::TaskManager.Task::(_decref) Task=`76a2e0da-3bfb-407c-9167-40cc3f732641`::ref 1 aborting False >Thread-27735::INFO::2012-12-02 14:25:46,501::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27735::DEBUG::2012-12-02 14:25:46,501::task::1151::TaskManager.Task::(prepare) Task=`76a2e0da-3bfb-407c-9167-40cc3f732641`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27735::DEBUG::2012-12-02 14:25:46,501::task::568::TaskManager.Task::(_updateState) Task=`76a2e0da-3bfb-407c-9167-40cc3f732641`::moving from state preparing -> state finished >Thread-27735::DEBUG::2012-12-02 14:25:46,501::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27735::DEBUG::2012-12-02 14:25:46,501::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27735::DEBUG::2012-12-02 14:25:46,501::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27735::DEBUG::2012-12-02 14:25:46,501::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27735::DEBUG::2012-12-02 14:25:46,502::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27735::DEBUG::2012-12-02 14:25:46,502::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27735::DEBUG::2012-12-02 14:25:46,502::task::957::TaskManager.Task::(_decref) Task=`76a2e0da-3bfb-407c-9167-40cc3f732641`::ref 0 aborting False >Thread-27738::DEBUG::2012-12-02 14:25:51,898::task::568::TaskManager.Task::(_updateState) Task=`543cbdd7-8081-4815-9c01-2c230af49ea9`::moving from state init -> state preparing >Thread-27738::INFO::2012-12-02 14:25:51,898::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27738::INFO::2012-12-02 14:25:51,899::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00903701782227', 'lastCheck': 1354451147.3174579, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00187110900879', 'lastCheck': 1354451142.8344691, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018789768219', 'lastCheck': 1354451144.993325, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00822877883911', 'lastCheck': 1354451142.8757, 'code': 0, 'valid': True}} >Thread-27738::DEBUG::2012-12-02 14:25:51,899::task::1151::TaskManager.Task::(prepare) Task=`543cbdd7-8081-4815-9c01-2c230af49ea9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00903701782227', 'lastCheck': 1354451147.3174579, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00187110900879', 'lastCheck': 1354451142.8344691, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018789768219', 'lastCheck': 1354451144.993325, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00822877883911', 'lastCheck': 1354451142.8757, 'code': 0, 'valid': True}} >Thread-27738::DEBUG::2012-12-02 14:25:51,899::task::568::TaskManager.Task::(_updateState) Task=`543cbdd7-8081-4815-9c01-2c230af49ea9`::moving from state preparing -> state finished >Thread-27738::DEBUG::2012-12-02 14:25:51,899::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27738::DEBUG::2012-12-02 14:25:51,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27738::DEBUG::2012-12-02 14:25:51,899::task::957::TaskManager.Task::(_decref) Task=`543cbdd7-8081-4815-9c01-2c230af49ea9`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:25:54,993::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:25:54,993::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:25:54,994::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:25:55,198::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:25:55,199::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:25:55,200::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:25:55,235::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:25:55,236::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:25:55,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:25:55,241::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:25:55,241::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:25:55,241::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-27803::DEBUG::2012-12-02 14:25:56,621::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27803::DEBUG::2012-12-02 14:25:56,621::task::568::TaskManager.Task::(_updateState) Task=`beca49b7-67f4-4c42-a7ed-ce49e5c13d88`::moving from state init -> state preparing >Thread-27803::INFO::2012-12-02 14:25:56,621::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27803::INFO::2012-12-02 14:25:56,622::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27803::DEBUG::2012-12-02 14:25:56,622::task::1151::TaskManager.Task::(prepare) Task=`beca49b7-67f4-4c42-a7ed-ce49e5c13d88`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27803::DEBUG::2012-12-02 14:25:56,622::task::568::TaskManager.Task::(_updateState) Task=`beca49b7-67f4-4c42-a7ed-ce49e5c13d88`::moving from state preparing -> state finished >Thread-27803::DEBUG::2012-12-02 14:25:56,622::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27803::DEBUG::2012-12-02 14:25:56,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27803::DEBUG::2012-12-02 14:25:56,622::task::957::TaskManager.Task::(_decref) Task=`beca49b7-67f4-4c42-a7ed-ce49e5c13d88`::ref 0 aborting False >Thread-27804::DEBUG::2012-12-02 14:25:56,625::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27804::DEBUG::2012-12-02 14:25:56,625::task::568::TaskManager.Task::(_updateState) Task=`91e4f91b-14ef-43b1-9d31-08b749d58fa3`::moving from state init -> state preparing >Thread-27804::INFO::2012-12-02 14:25:56,625::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27804::DEBUG::2012-12-02 14:25:56,625::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c21dbf6c-0c52-4c52-b278-f81db5fcd8ce`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27804::DEBUG::2012-12-02 14:25:56,626::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27804::DEBUG::2012-12-02 14:25:56,626::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27804::DEBUG::2012-12-02 14:25:56,626::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c21dbf6c-0c52-4c52-b278-f81db5fcd8ce`::Granted request >Thread-27804::DEBUG::2012-12-02 14:25:56,626::task::794::TaskManager.Task::(resourceAcquired) Task=`91e4f91b-14ef-43b1-9d31-08b749d58fa3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27804::DEBUG::2012-12-02 14:25:56,626::task::957::TaskManager.Task::(_decref) Task=`91e4f91b-14ef-43b1-9d31-08b749d58fa3`::ref 1 aborting False >Thread-27804::INFO::2012-12-02 14:25:56,631::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27804::DEBUG::2012-12-02 14:25:56,631::task::1151::TaskManager.Task::(prepare) Task=`91e4f91b-14ef-43b1-9d31-08b749d58fa3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27804::DEBUG::2012-12-02 14:25:56,631::task::568::TaskManager.Task::(_updateState) Task=`91e4f91b-14ef-43b1-9d31-08b749d58fa3`::moving from state preparing -> state finished >Thread-27804::DEBUG::2012-12-02 14:25:56,631::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27804::DEBUG::2012-12-02 14:25:56,631::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27804::DEBUG::2012-12-02 14:25:56,631::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27804::DEBUG::2012-12-02 14:25:56,631::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27804::DEBUG::2012-12-02 14:25:56,631::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27804::DEBUG::2012-12-02 14:25:56,631::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27804::DEBUG::2012-12-02 14:25:56,632::task::957::TaskManager.Task::(_decref) Task=`91e4f91b-14ef-43b1-9d31-08b749d58fa3`::ref 0 aborting False >Thread-11821::DEBUG::2012-12-02 14:25:57,317::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:25:57,317::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:25:57,318::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:25:57,524::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:25:57,526::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:25:57,526::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:25:57,562::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:25:57,562::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:25:57,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:25:57,577::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:25:57,578::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:25:57,578::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-27868::DEBUG::2012-12-02 14:26:02,021::task::568::TaskManager.Task::(_updateState) Task=`483b21a0-bc61-4faf-a704-49df1efef5ab`::moving from state init -> state preparing >Thread-27868::INFO::2012-12-02 14:26:02,021::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27868::INFO::2012-12-02 14:26:02,022::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00324082374573', 'lastCheck': 1354451157.589709, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198698043823', 'lastCheck': 1354451152.8467641, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0016028881073', 'lastCheck': 1354451155.251518, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00771284103394', 'lastCheck': 1354451152.8922551, 'code': 0, 'valid': True}} >Thread-27868::DEBUG::2012-12-02 14:26:02,022::task::1151::TaskManager.Task::(prepare) Task=`483b21a0-bc61-4faf-a704-49df1efef5ab`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00324082374573', 'lastCheck': 1354451157.589709, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198698043823', 'lastCheck': 1354451152.8467641, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0016028881073', 'lastCheck': 1354451155.251518, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00771284103394', 'lastCheck': 1354451152.8922551, 'code': 0, 'valid': True}} >Thread-27868::DEBUG::2012-12-02 14:26:02,022::task::568::TaskManager.Task::(_updateState) Task=`483b21a0-bc61-4faf-a704-49df1efef5ab`::moving from state preparing -> state finished >Thread-27868::DEBUG::2012-12-02 14:26:02,022::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27868::DEBUG::2012-12-02 14:26:02,022::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27868::DEBUG::2012-12-02 14:26:02,022::task::957::TaskManager.Task::(_decref) Task=`483b21a0-bc61-4faf-a704-49df1efef5ab`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:26:02,846::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:26:02,847::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:26:02,847::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:26:03,053::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:26:03,055::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:26:03,055::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:26:03,091::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:26:03,091::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:26:03,093::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:26:03,096::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:26:03,096::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:26:03,097::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:26:03,097::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:26:03,097::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:26:03,098::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:26:03,306::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:26:03,308::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:26:03,308::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:26:03,342::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:26:03,342::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:26:03,349::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:26:03,352::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:26:03,352::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:26:03,353::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-27994::DEBUG::2012-12-02 14:26:06,793::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27994::DEBUG::2012-12-02 14:26:06,793::task::568::TaskManager.Task::(_updateState) Task=`a124af88-cf96-4272-bdc7-8944f578ae3c`::moving from state init -> state preparing >Thread-27994::INFO::2012-12-02 14:26:06,793::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27994::INFO::2012-12-02 14:26:06,793::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27994::DEBUG::2012-12-02 14:26:06,793::task::1151::TaskManager.Task::(prepare) Task=`a124af88-cf96-4272-bdc7-8944f578ae3c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-27994::DEBUG::2012-12-02 14:26:06,794::task::568::TaskManager.Task::(_updateState) Task=`a124af88-cf96-4272-bdc7-8944f578ae3c`::moving from state preparing -> state finished >Thread-27994::DEBUG::2012-12-02 14:26:06,794::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27994::DEBUG::2012-12-02 14:26:06,794::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27994::DEBUG::2012-12-02 14:26:06,794::task::957::TaskManager.Task::(_decref) Task=`a124af88-cf96-4272-bdc7-8944f578ae3c`::ref 0 aborting False >Thread-27995::DEBUG::2012-12-02 14:26:06,797::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-27995::DEBUG::2012-12-02 14:26:06,797::task::568::TaskManager.Task::(_updateState) Task=`b4ae60f6-e665-4b1c-9de8-2b8a6ab3eb84`::moving from state init -> state preparing >Thread-27995::INFO::2012-12-02 14:26:06,797::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-27995::DEBUG::2012-12-02 14:26:06,797::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d0d9b287-aa9f-4408-87bb-5425260f20ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-27995::DEBUG::2012-12-02 14:26:06,797::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-27995::DEBUG::2012-12-02 14:26:06,797::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-27995::DEBUG::2012-12-02 14:26:06,798::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d0d9b287-aa9f-4408-87bb-5425260f20ba`::Granted request >Thread-27995::DEBUG::2012-12-02 14:26:06,798::task::794::TaskManager.Task::(resourceAcquired) Task=`b4ae60f6-e665-4b1c-9de8-2b8a6ab3eb84`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-27995::DEBUG::2012-12-02 14:26:06,798::task::957::TaskManager.Task::(_decref) Task=`b4ae60f6-e665-4b1c-9de8-2b8a6ab3eb84`::ref 1 aborting False >Thread-27995::INFO::2012-12-02 14:26:06,802::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27995::DEBUG::2012-12-02 14:26:06,802::task::1151::TaskManager.Task::(prepare) Task=`b4ae60f6-e665-4b1c-9de8-2b8a6ab3eb84`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-27995::DEBUG::2012-12-02 14:26:06,802::task::568::TaskManager.Task::(_updateState) Task=`b4ae60f6-e665-4b1c-9de8-2b8a6ab3eb84`::moving from state preparing -> state finished >Thread-27995::DEBUG::2012-12-02 14:26:06,802::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-27995::DEBUG::2012-12-02 14:26:06,803::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27995::DEBUG::2012-12-02 14:26:06,803::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-27995::DEBUG::2012-12-02 14:26:06,803::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-27995::DEBUG::2012-12-02 14:26:06,803::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-27995::DEBUG::2012-12-02 14:26:06,803::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-27995::DEBUG::2012-12-02 14:26:06,803::task::957::TaskManager.Task::(_decref) Task=`b4ae60f6-e665-4b1c-9de8-2b8a6ab3eb84`::ref 0 aborting False >Thread-27998::DEBUG::2012-12-02 14:26:12,137::task::568::TaskManager.Task::(_updateState) Task=`77cd07bf-6942-4cb1-8580-1b82e5fa945a`::moving from state init -> state preparing >Thread-27998::INFO::2012-12-02 14:26:12,137::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-27998::INFO::2012-12-02 14:26:12,137::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00931882858276', 'lastCheck': 1354451167.607929, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00250816345215', 'lastCheck': 1354451163.3603251, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00167202949524', 'lastCheck': 1354451165.261795, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00326013565063', 'lastCheck': 1354451163.3649759, 'code': 0, 'valid': True}} >Thread-27998::DEBUG::2012-12-02 14:26:12,137::task::1151::TaskManager.Task::(prepare) Task=`77cd07bf-6942-4cb1-8580-1b82e5fa945a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00931882858276', 'lastCheck': 1354451167.607929, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00250816345215', 'lastCheck': 1354451163.3603251, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00167202949524', 'lastCheck': 1354451165.261795, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00326013565063', 'lastCheck': 1354451163.3649759, 'code': 0, 'valid': True}} >Thread-27998::DEBUG::2012-12-02 14:26:12,138::task::568::TaskManager.Task::(_updateState) Task=`77cd07bf-6942-4cb1-8580-1b82e5fa945a`::moving from state preparing -> state finished >Thread-27998::DEBUG::2012-12-02 14:26:12,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-27998::DEBUG::2012-12-02 14:26:12,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-27998::DEBUG::2012-12-02 14:26:12,138::task::957::TaskManager.Task::(_decref) Task=`77cd07bf-6942-4cb1-8580-1b82e5fa945a`::ref 0 aborting False >Thread-28002::DEBUG::2012-12-02 14:26:16,930::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28002::DEBUG::2012-12-02 14:26:16,930::task::568::TaskManager.Task::(_updateState) Task=`104e1169-1552-4f5d-87f8-a7f05366bd47`::moving from state init -> state preparing >Thread-28002::INFO::2012-12-02 14:26:16,930::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28002::INFO::2012-12-02 14:26:16,930::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28002::DEBUG::2012-12-02 14:26:16,931::task::1151::TaskManager.Task::(prepare) Task=`104e1169-1552-4f5d-87f8-a7f05366bd47`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28002::DEBUG::2012-12-02 14:26:16,931::task::568::TaskManager.Task::(_updateState) Task=`104e1169-1552-4f5d-87f8-a7f05366bd47`::moving from state preparing -> state finished >Thread-28002::DEBUG::2012-12-02 14:26:16,931::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28002::DEBUG::2012-12-02 14:26:16,931::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28002::DEBUG::2012-12-02 14:26:16,931::task::957::TaskManager.Task::(_decref) Task=`104e1169-1552-4f5d-87f8-a7f05366bd47`::ref 0 aborting False >Thread-28003::DEBUG::2012-12-02 14:26:16,934::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28003::DEBUG::2012-12-02 14:26:16,934::task::568::TaskManager.Task::(_updateState) Task=`1311a8b2-802c-487e-a07e-6cb7ed277087`::moving from state init -> state preparing >Thread-28003::INFO::2012-12-02 14:26:16,934::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28003::DEBUG::2012-12-02 14:26:16,934::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9636cd9a-afaf-48dc-83a0-7a9ca74ad26d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28003::DEBUG::2012-12-02 14:26:16,935::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28003::DEBUG::2012-12-02 14:26:16,935::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28003::DEBUG::2012-12-02 14:26:16,935::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9636cd9a-afaf-48dc-83a0-7a9ca74ad26d`::Granted request >Thread-28003::DEBUG::2012-12-02 14:26:16,935::task::794::TaskManager.Task::(resourceAcquired) Task=`1311a8b2-802c-487e-a07e-6cb7ed277087`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28003::DEBUG::2012-12-02 14:26:16,935::task::957::TaskManager.Task::(_decref) Task=`1311a8b2-802c-487e-a07e-6cb7ed277087`::ref 1 aborting False >Thread-28003::INFO::2012-12-02 14:26:16,940::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28003::DEBUG::2012-12-02 14:26:16,940::task::1151::TaskManager.Task::(prepare) Task=`1311a8b2-802c-487e-a07e-6cb7ed277087`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28003::DEBUG::2012-12-02 14:26:16,940::task::568::TaskManager.Task::(_updateState) Task=`1311a8b2-802c-487e-a07e-6cb7ed277087`::moving from state preparing -> state finished >Thread-28003::DEBUG::2012-12-02 14:26:16,940::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28003::DEBUG::2012-12-02 14:26:16,940::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28003::DEBUG::2012-12-02 14:26:16,940::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28003::DEBUG::2012-12-02 14:26:16,940::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28003::DEBUG::2012-12-02 14:26:16,940::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28003::DEBUG::2012-12-02 14:26:16,940::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28003::DEBUG::2012-12-02 14:26:16,941::task::957::TaskManager.Task::(_decref) Task=`1311a8b2-802c-487e-a07e-6cb7ed277087`::ref 0 aborting False >Thread-28006::DEBUG::2012-12-02 14:26:22,255::task::568::TaskManager.Task::(_updateState) Task=`3a3d7cc0-5573-4737-8135-8eee68f5300d`::moving from state init -> state preparing >Thread-28006::INFO::2012-12-02 14:26:22,256::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28006::INFO::2012-12-02 14:26:22,256::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00814318656921', 'lastCheck': 1354451177.625679, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00502300262451', 'lastCheck': 1354451173.380343, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00263595581055', 'lastCheck': 1354451175.2736859, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809907913208', 'lastCheck': 1354451173.3847389, 'code': 0, 'valid': True}} >Thread-28006::DEBUG::2012-12-02 14:26:22,256::task::1151::TaskManager.Task::(prepare) Task=`3a3d7cc0-5573-4737-8135-8eee68f5300d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00814318656921', 'lastCheck': 1354451177.625679, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00502300262451', 'lastCheck': 1354451173.380343, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00263595581055', 'lastCheck': 1354451175.2736859, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809907913208', 'lastCheck': 1354451173.3847389, 'code': 0, 'valid': True}} >Thread-28006::DEBUG::2012-12-02 14:26:22,256::task::568::TaskManager.Task::(_updateState) Task=`3a3d7cc0-5573-4737-8135-8eee68f5300d`::moving from state preparing -> state finished >Thread-28006::DEBUG::2012-12-02 14:26:22,256::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28006::DEBUG::2012-12-02 14:26:22,256::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28006::DEBUG::2012-12-02 14:26:22,257::task::957::TaskManager.Task::(_decref) Task=`3a3d7cc0-5573-4737-8135-8eee68f5300d`::ref 0 aborting False >Thread-28010::DEBUG::2012-12-02 14:26:27,076::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28010::DEBUG::2012-12-02 14:26:27,077::task::568::TaskManager.Task::(_updateState) Task=`cc34e8de-6d95-4510-a8c3-c34cdc6bc16a`::moving from state init -> state preparing >Thread-28010::INFO::2012-12-02 14:26:27,077::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28010::INFO::2012-12-02 14:26:27,077::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28010::DEBUG::2012-12-02 14:26:27,077::task::1151::TaskManager.Task::(prepare) Task=`cc34e8de-6d95-4510-a8c3-c34cdc6bc16a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28010::DEBUG::2012-12-02 14:26:27,078::task::568::TaskManager.Task::(_updateState) Task=`cc34e8de-6d95-4510-a8c3-c34cdc6bc16a`::moving from state preparing -> state finished >Thread-28010::DEBUG::2012-12-02 14:26:27,078::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28010::DEBUG::2012-12-02 14:26:27,078::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28010::DEBUG::2012-12-02 14:26:27,078::task::957::TaskManager.Task::(_decref) Task=`cc34e8de-6d95-4510-a8c3-c34cdc6bc16a`::ref 0 aborting False >Thread-28011::DEBUG::2012-12-02 14:26:27,080::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28011::DEBUG::2012-12-02 14:26:27,081::task::568::TaskManager.Task::(_updateState) Task=`b5807130-0cec-4df5-8722-f2ad6336486e`::moving from state init -> state preparing >Thread-28011::INFO::2012-12-02 14:26:27,081::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28011::DEBUG::2012-12-02 14:26:27,081::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5e8ae0b1-6622-400a-86e0-9a98cb89be11`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28011::DEBUG::2012-12-02 14:26:27,081::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28011::DEBUG::2012-12-02 14:26:27,082::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28011::DEBUG::2012-12-02 14:26:27,082::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5e8ae0b1-6622-400a-86e0-9a98cb89be11`::Granted request >Thread-28011::DEBUG::2012-12-02 14:26:27,082::task::794::TaskManager.Task::(resourceAcquired) Task=`b5807130-0cec-4df5-8722-f2ad6336486e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28011::DEBUG::2012-12-02 14:26:27,082::task::957::TaskManager.Task::(_decref) Task=`b5807130-0cec-4df5-8722-f2ad6336486e`::ref 1 aborting False >Thread-28011::INFO::2012-12-02 14:26:27,087::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28011::DEBUG::2012-12-02 14:26:27,087::task::1151::TaskManager.Task::(prepare) Task=`b5807130-0cec-4df5-8722-f2ad6336486e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28011::DEBUG::2012-12-02 14:26:27,088::task::568::TaskManager.Task::(_updateState) Task=`b5807130-0cec-4df5-8722-f2ad6336486e`::moving from state preparing -> state finished >Thread-28011::DEBUG::2012-12-02 14:26:27,088::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28011::DEBUG::2012-12-02 14:26:27,088::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28011::DEBUG::2012-12-02 14:26:27,088::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28011::DEBUG::2012-12-02 14:26:27,088::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28011::DEBUG::2012-12-02 14:26:27,088::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28011::DEBUG::2012-12-02 14:26:27,088::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28011::DEBUG::2012-12-02 14:26:27,088::task::957::TaskManager.Task::(_decref) Task=`b5807130-0cec-4df5-8722-f2ad6336486e`::ref 0 aborting False >Thread-28014::DEBUG::2012-12-02 14:26:32,404::task::568::TaskManager.Task::(_updateState) Task=`d3e1e528-cdc5-4524-8cef-d605d59c1fee`::moving from state init -> state preparing >Thread-28014::INFO::2012-12-02 14:26:32,404::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28014::INFO::2012-12-02 14:26:32,404::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00489592552185', 'lastCheck': 1354451187.6399629, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00311684608459', 'lastCheck': 1354451183.399559, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.001953125', 'lastCheck': 1354451185.2850871, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00870084762573', 'lastCheck': 1354451183.405153, 'code': 0, 'valid': True}} >Thread-28014::DEBUG::2012-12-02 14:26:32,404::task::1151::TaskManager.Task::(prepare) Task=`d3e1e528-cdc5-4524-8cef-d605d59c1fee`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00489592552185', 'lastCheck': 1354451187.6399629, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00311684608459', 'lastCheck': 1354451183.399559, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.001953125', 'lastCheck': 1354451185.2850871, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00870084762573', 'lastCheck': 1354451183.405153, 'code': 0, 'valid': True}} >Thread-28014::DEBUG::2012-12-02 14:26:32,405::task::568::TaskManager.Task::(_updateState) Task=`d3e1e528-cdc5-4524-8cef-d605d59c1fee`::moving from state preparing -> state finished >Thread-28014::DEBUG::2012-12-02 14:26:32,405::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28014::DEBUG::2012-12-02 14:26:32,405::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28014::DEBUG::2012-12-02 14:26:32,405::task::957::TaskManager.Task::(_decref) Task=`d3e1e528-cdc5-4524-8cef-d605d59c1fee`::ref 0 aborting False >Thread-28018::DEBUG::2012-12-02 14:26:37,182::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28018::DEBUG::2012-12-02 14:26:37,182::task::568::TaskManager.Task::(_updateState) Task=`9463f77e-0611-4c90-acbf-3930fa527cb9`::moving from state init -> state preparing >Thread-28018::INFO::2012-12-02 14:26:37,182::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28018::INFO::2012-12-02 14:26:37,183::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28018::DEBUG::2012-12-02 14:26:37,183::task::1151::TaskManager.Task::(prepare) Task=`9463f77e-0611-4c90-acbf-3930fa527cb9`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28018::DEBUG::2012-12-02 14:26:37,183::task::568::TaskManager.Task::(_updateState) Task=`9463f77e-0611-4c90-acbf-3930fa527cb9`::moving from state preparing -> state finished >Thread-28018::DEBUG::2012-12-02 14:26:37,183::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28018::DEBUG::2012-12-02 14:26:37,183::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28018::DEBUG::2012-12-02 14:26:37,183::task::957::TaskManager.Task::(_decref) Task=`9463f77e-0611-4c90-acbf-3930fa527cb9`::ref 0 aborting False >Thread-28019::DEBUG::2012-12-02 14:26:37,186::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28019::DEBUG::2012-12-02 14:26:37,186::task::568::TaskManager.Task::(_updateState) Task=`925864a7-a5f2-4527-9699-389f1d204c18`::moving from state init -> state preparing >Thread-28019::INFO::2012-12-02 14:26:37,186::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28019::DEBUG::2012-12-02 14:26:37,187::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d4543096-8999-4a27-9aab-091f0f6d3845`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28019::DEBUG::2012-12-02 14:26:37,187::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28019::DEBUG::2012-12-02 14:26:37,187::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28019::DEBUG::2012-12-02 14:26:37,187::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d4543096-8999-4a27-9aab-091f0f6d3845`::Granted request >Thread-28019::DEBUG::2012-12-02 14:26:37,187::task::794::TaskManager.Task::(resourceAcquired) Task=`925864a7-a5f2-4527-9699-389f1d204c18`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28019::DEBUG::2012-12-02 14:26:37,187::task::957::TaskManager.Task::(_decref) Task=`925864a7-a5f2-4527-9699-389f1d204c18`::ref 1 aborting False >Thread-28019::INFO::2012-12-02 14:26:37,194::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28019::DEBUG::2012-12-02 14:26:37,194::task::1151::TaskManager.Task::(prepare) Task=`925864a7-a5f2-4527-9699-389f1d204c18`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28019::DEBUG::2012-12-02 14:26:37,194::task::568::TaskManager.Task::(_updateState) Task=`925864a7-a5f2-4527-9699-389f1d204c18`::moving from state preparing -> state finished >Thread-28019::DEBUG::2012-12-02 14:26:37,194::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28019::DEBUG::2012-12-02 14:26:37,194::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28019::DEBUG::2012-12-02 14:26:37,195::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28019::DEBUG::2012-12-02 14:26:37,195::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28019::DEBUG::2012-12-02 14:26:37,195::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28019::DEBUG::2012-12-02 14:26:37,195::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28019::DEBUG::2012-12-02 14:26:37,195::task::957::TaskManager.Task::(_decref) Task=`925864a7-a5f2-4527-9699-389f1d204c18`::ref 0 aborting False >Thread-28022::DEBUG::2012-12-02 14:26:42,515::task::568::TaskManager.Task::(_updateState) Task=`0a1dfc33-1940-4e0e-9597-273a93939727`::moving from state init -> state preparing >Thread-28022::INFO::2012-12-02 14:26:42,516::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28022::INFO::2012-12-02 14:26:42,516::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00861501693726', 'lastCheck': 1354451197.6609819, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00207185745239', 'lastCheck': 1354451193.422868, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00277709960938', 'lastCheck': 1354451195.2980559, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00347900390625', 'lastCheck': 1354451193.423322, 'code': 0, 'valid': True}} >Thread-28022::DEBUG::2012-12-02 14:26:42,516::task::1151::TaskManager.Task::(prepare) Task=`0a1dfc33-1940-4e0e-9597-273a93939727`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00861501693726', 'lastCheck': 1354451197.6609819, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00207185745239', 'lastCheck': 1354451193.422868, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00277709960938', 'lastCheck': 1354451195.2980559, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00347900390625', 'lastCheck': 1354451193.423322, 'code': 0, 'valid': True}} >Thread-28022::DEBUG::2012-12-02 14:26:42,516::task::568::TaskManager.Task::(_updateState) Task=`0a1dfc33-1940-4e0e-9597-273a93939727`::moving from state preparing -> state finished >Thread-28022::DEBUG::2012-12-02 14:26:42,516::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28022::DEBUG::2012-12-02 14:26:42,516::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28022::DEBUG::2012-12-02 14:26:42,516::task::957::TaskManager.Task::(_decref) Task=`0a1dfc33-1940-4e0e-9597-273a93939727`::ref 0 aborting False >Thread-28026::DEBUG::2012-12-02 14:26:47,321::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28026::DEBUG::2012-12-02 14:26:47,321::task::568::TaskManager.Task::(_updateState) Task=`f1353eb5-8ab0-4f1e-8b74-e267a4c6a927`::moving from state init -> state preparing >Thread-28026::INFO::2012-12-02 14:26:47,321::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28026::INFO::2012-12-02 14:26:47,321::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28026::DEBUG::2012-12-02 14:26:47,322::task::1151::TaskManager.Task::(prepare) Task=`f1353eb5-8ab0-4f1e-8b74-e267a4c6a927`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28026::DEBUG::2012-12-02 14:26:47,322::task::568::TaskManager.Task::(_updateState) Task=`f1353eb5-8ab0-4f1e-8b74-e267a4c6a927`::moving from state preparing -> state finished >Thread-28026::DEBUG::2012-12-02 14:26:47,322::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28026::DEBUG::2012-12-02 14:26:47,322::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28026::DEBUG::2012-12-02 14:26:47,322::task::957::TaskManager.Task::(_decref) Task=`f1353eb5-8ab0-4f1e-8b74-e267a4c6a927`::ref 0 aborting False >Thread-28027::DEBUG::2012-12-02 14:26:47,324::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28027::DEBUG::2012-12-02 14:26:47,325::task::568::TaskManager.Task::(_updateState) Task=`f002a136-8154-4b25-ba14-33303481ee40`::moving from state init -> state preparing >Thread-28027::INFO::2012-12-02 14:26:47,325::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28027::DEBUG::2012-12-02 14:26:47,325::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1d1b2280-adc5-416f-a298-ceace986c9ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28027::DEBUG::2012-12-02 14:26:47,325::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28027::DEBUG::2012-12-02 14:26:47,326::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28027::DEBUG::2012-12-02 14:26:47,326::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1d1b2280-adc5-416f-a298-ceace986c9ba`::Granted request >Thread-28027::DEBUG::2012-12-02 14:26:47,326::task::794::TaskManager.Task::(resourceAcquired) Task=`f002a136-8154-4b25-ba14-33303481ee40`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28027::DEBUG::2012-12-02 14:26:47,326::task::957::TaskManager.Task::(_decref) Task=`f002a136-8154-4b25-ba14-33303481ee40`::ref 1 aborting False >Thread-28027::INFO::2012-12-02 14:26:47,330::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28027::DEBUG::2012-12-02 14:26:47,331::task::1151::TaskManager.Task::(prepare) Task=`f002a136-8154-4b25-ba14-33303481ee40`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28027::DEBUG::2012-12-02 14:26:47,331::task::568::TaskManager.Task::(_updateState) Task=`f002a136-8154-4b25-ba14-33303481ee40`::moving from state preparing -> state finished >Thread-28027::DEBUG::2012-12-02 14:26:47,331::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28027::DEBUG::2012-12-02 14:26:47,331::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28027::DEBUG::2012-12-02 14:26:47,331::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28027::DEBUG::2012-12-02 14:26:47,331::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28027::DEBUG::2012-12-02 14:26:47,331::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28027::DEBUG::2012-12-02 14:26:47,331::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28027::DEBUG::2012-12-02 14:26:47,331::task::957::TaskManager.Task::(_decref) Task=`f002a136-8154-4b25-ba14-33303481ee40`::ref 0 aborting False >Thread-28030::DEBUG::2012-12-02 14:26:52,613::task::568::TaskManager.Task::(_updateState) Task=`3319d619-29fc-40ed-8ebe-601a01fcbe36`::moving from state init -> state preparing >Thread-28030::INFO::2012-12-02 14:26:52,613::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28030::INFO::2012-12-02 14:26:52,613::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00847291946411', 'lastCheck': 1354451207.6793921, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00177788734436', 'lastCheck': 1354451203.4464481, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177097320557', 'lastCheck': 1354451205.3104241, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00954699516296', 'lastCheck': 1354451203.453012, 'code': 0, 'valid': True}} >Thread-28030::DEBUG::2012-12-02 14:26:52,613::task::1151::TaskManager.Task::(prepare) Task=`3319d619-29fc-40ed-8ebe-601a01fcbe36`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00847291946411', 'lastCheck': 1354451207.6793921, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00177788734436', 'lastCheck': 1354451203.4464481, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177097320557', 'lastCheck': 1354451205.3104241, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00954699516296', 'lastCheck': 1354451203.453012, 'code': 0, 'valid': True}} >Thread-28030::DEBUG::2012-12-02 14:26:52,614::task::568::TaskManager.Task::(_updateState) Task=`3319d619-29fc-40ed-8ebe-601a01fcbe36`::moving from state preparing -> state finished >Thread-28030::DEBUG::2012-12-02 14:26:52,614::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28030::DEBUG::2012-12-02 14:26:52,614::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28030::DEBUG::2012-12-02 14:26:52,614::task::957::TaskManager.Task::(_decref) Task=`3319d619-29fc-40ed-8ebe-601a01fcbe36`::ref 0 aborting False >Thread-28034::DEBUG::2012-12-02 14:26:57,425::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28034::DEBUG::2012-12-02 14:26:57,426::task::568::TaskManager.Task::(_updateState) Task=`ccd0652c-2873-4136-b3e2-fc7ca447390d`::moving from state init -> state preparing >Thread-28034::INFO::2012-12-02 14:26:57,426::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28034::INFO::2012-12-02 14:26:57,426::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28034::DEBUG::2012-12-02 14:26:57,426::task::1151::TaskManager.Task::(prepare) Task=`ccd0652c-2873-4136-b3e2-fc7ca447390d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28034::DEBUG::2012-12-02 14:26:57,426::task::568::TaskManager.Task::(_updateState) Task=`ccd0652c-2873-4136-b3e2-fc7ca447390d`::moving from state preparing -> state finished >Thread-28034::DEBUG::2012-12-02 14:26:57,426::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28034::DEBUG::2012-12-02 14:26:57,426::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28034::DEBUG::2012-12-02 14:26:57,427::task::957::TaskManager.Task::(_decref) Task=`ccd0652c-2873-4136-b3e2-fc7ca447390d`::ref 0 aborting False >Thread-28035::DEBUG::2012-12-02 14:26:57,429::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28035::DEBUG::2012-12-02 14:26:57,429::task::568::TaskManager.Task::(_updateState) Task=`e964ad52-e0c5-444a-aa0e-76dc2bd3cb8c`::moving from state init -> state preparing >Thread-28035::INFO::2012-12-02 14:26:57,430::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28035::DEBUG::2012-12-02 14:26:57,430::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`009b4cf6-b190-4b6f-8fb5-65a720249556`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28035::DEBUG::2012-12-02 14:26:57,430::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28035::DEBUG::2012-12-02 14:26:57,430::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28035::DEBUG::2012-12-02 14:26:57,430::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`009b4cf6-b190-4b6f-8fb5-65a720249556`::Granted request >Thread-28035::DEBUG::2012-12-02 14:26:57,431::task::794::TaskManager.Task::(resourceAcquired) Task=`e964ad52-e0c5-444a-aa0e-76dc2bd3cb8c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28035::DEBUG::2012-12-02 14:26:57,431::task::957::TaskManager.Task::(_decref) Task=`e964ad52-e0c5-444a-aa0e-76dc2bd3cb8c`::ref 1 aborting False >Thread-28035::INFO::2012-12-02 14:26:57,436::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28035::DEBUG::2012-12-02 14:26:57,437::task::1151::TaskManager.Task::(prepare) Task=`e964ad52-e0c5-444a-aa0e-76dc2bd3cb8c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28035::DEBUG::2012-12-02 14:26:57,437::task::568::TaskManager.Task::(_updateState) Task=`e964ad52-e0c5-444a-aa0e-76dc2bd3cb8c`::moving from state preparing -> state finished >Thread-28035::DEBUG::2012-12-02 14:26:57,437::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28035::DEBUG::2012-12-02 14:26:57,437::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28035::DEBUG::2012-12-02 14:26:57,437::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28035::DEBUG::2012-12-02 14:26:57,437::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28035::DEBUG::2012-12-02 14:26:57,437::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28035::DEBUG::2012-12-02 14:26:57,437::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28035::DEBUG::2012-12-02 14:26:57,437::task::957::TaskManager.Task::(_decref) Task=`e964ad52-e0c5-444a-aa0e-76dc2bd3cb8c`::ref 0 aborting False >Thread-28038::DEBUG::2012-12-02 14:27:02,717::task::568::TaskManager.Task::(_updateState) Task=`b9b60df0-c2c7-4fc0-bfaf-37e1bb40d78e`::moving from state init -> state preparing >Thread-28038::INFO::2012-12-02 14:27:02,718::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28038::INFO::2012-12-02 14:27:02,718::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00851583480835', 'lastCheck': 1354451217.697907, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0035400390625', 'lastCheck': 1354451213.461684, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00212097167969', 'lastCheck': 1354451215.3211291, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00958490371704', 'lastCheck': 1354451213.4720781, 'code': 0, 'valid': True}} >Thread-28038::DEBUG::2012-12-02 14:27:02,718::task::1151::TaskManager.Task::(prepare) Task=`b9b60df0-c2c7-4fc0-bfaf-37e1bb40d78e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00851583480835', 'lastCheck': 1354451217.697907, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0035400390625', 'lastCheck': 1354451213.461684, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00212097167969', 'lastCheck': 1354451215.3211291, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00958490371704', 'lastCheck': 1354451213.4720781, 'code': 0, 'valid': True}} >Thread-28038::DEBUG::2012-12-02 14:27:02,718::task::568::TaskManager.Task::(_updateState) Task=`b9b60df0-c2c7-4fc0-bfaf-37e1bb40d78e`::moving from state preparing -> state finished >Thread-28038::DEBUG::2012-12-02 14:27:02,718::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28038::DEBUG::2012-12-02 14:27:02,718::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28038::DEBUG::2012-12-02 14:27:02,718::task::957::TaskManager.Task::(_decref) Task=`b9b60df0-c2c7-4fc0-bfaf-37e1bb40d78e`::ref 0 aborting False >Thread-28042::DEBUG::2012-12-02 14:27:07,580::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28042::DEBUG::2012-12-02 14:27:07,581::task::568::TaskManager.Task::(_updateState) Task=`f3c0ee69-739e-4b0d-add2-d68d64dae344`::moving from state init -> state preparing >Thread-28042::INFO::2012-12-02 14:27:07,581::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28042::INFO::2012-12-02 14:27:07,581::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28042::DEBUG::2012-12-02 14:27:07,581::task::1151::TaskManager.Task::(prepare) Task=`f3c0ee69-739e-4b0d-add2-d68d64dae344`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28042::DEBUG::2012-12-02 14:27:07,581::task::568::TaskManager.Task::(_updateState) Task=`f3c0ee69-739e-4b0d-add2-d68d64dae344`::moving from state preparing -> state finished >Thread-28042::DEBUG::2012-12-02 14:27:07,581::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28042::DEBUG::2012-12-02 14:27:07,582::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28042::DEBUG::2012-12-02 14:27:07,582::task::957::TaskManager.Task::(_decref) Task=`f3c0ee69-739e-4b0d-add2-d68d64dae344`::ref 0 aborting False >Thread-28043::DEBUG::2012-12-02 14:27:07,584::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28043::DEBUG::2012-12-02 14:27:07,585::task::568::TaskManager.Task::(_updateState) Task=`6e50684c-a1cd-4375-9465-7c8b06ba2143`::moving from state init -> state preparing >Thread-28043::INFO::2012-12-02 14:27:07,585::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28043::DEBUG::2012-12-02 14:27:07,585::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e8f3769c-cd12-4bfd-8744-64ff24952383`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28043::DEBUG::2012-12-02 14:27:07,585::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28043::DEBUG::2012-12-02 14:27:07,585::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28043::DEBUG::2012-12-02 14:27:07,585::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e8f3769c-cd12-4bfd-8744-64ff24952383`::Granted request >Thread-28043::DEBUG::2012-12-02 14:27:07,586::task::794::TaskManager.Task::(resourceAcquired) Task=`6e50684c-a1cd-4375-9465-7c8b06ba2143`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28043::DEBUG::2012-12-02 14:27:07,586::task::957::TaskManager.Task::(_decref) Task=`6e50684c-a1cd-4375-9465-7c8b06ba2143`::ref 1 aborting False >Thread-28043::INFO::2012-12-02 14:27:07,590::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28043::DEBUG::2012-12-02 14:27:07,590::task::1151::TaskManager.Task::(prepare) Task=`6e50684c-a1cd-4375-9465-7c8b06ba2143`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28043::DEBUG::2012-12-02 14:27:07,590::task::568::TaskManager.Task::(_updateState) Task=`6e50684c-a1cd-4375-9465-7c8b06ba2143`::moving from state preparing -> state finished >Thread-28043::DEBUG::2012-12-02 14:27:07,590::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28043::DEBUG::2012-12-02 14:27:07,590::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28043::DEBUG::2012-12-02 14:27:07,591::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28043::DEBUG::2012-12-02 14:27:07,591::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28043::DEBUG::2012-12-02 14:27:07,591::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28043::DEBUG::2012-12-02 14:27:07,591::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28043::DEBUG::2012-12-02 14:27:07,591::task::957::TaskManager.Task::(_decref) Task=`6e50684c-a1cd-4375-9465-7c8b06ba2143`::ref 0 aborting False >Thread-28046::DEBUG::2012-12-02 14:27:12,817::task::568::TaskManager.Task::(_updateState) Task=`e826b916-a5e8-45e9-9b77-c84ec8191a0c`::moving from state init -> state preparing >Thread-28046::INFO::2012-12-02 14:27:12,817::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28046::INFO::2012-12-02 14:27:12,817::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00820589065552', 'lastCheck': 1354451227.715595, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00267505645752', 'lastCheck': 1354451223.4813941, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00176095962524', 'lastCheck': 1354451225.331192, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00858592987061', 'lastCheck': 1354451223.492497, 'code': 0, 'valid': True}} >Thread-28046::DEBUG::2012-12-02 14:27:12,817::task::1151::TaskManager.Task::(prepare) Task=`e826b916-a5e8-45e9-9b77-c84ec8191a0c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00820589065552', 'lastCheck': 1354451227.715595, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00267505645752', 'lastCheck': 1354451223.4813941, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00176095962524', 'lastCheck': 1354451225.331192, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00858592987061', 'lastCheck': 1354451223.492497, 'code': 0, 'valid': True}} >Thread-28046::DEBUG::2012-12-02 14:27:12,818::task::568::TaskManager.Task::(_updateState) Task=`e826b916-a5e8-45e9-9b77-c84ec8191a0c`::moving from state preparing -> state finished >Thread-28046::DEBUG::2012-12-02 14:27:12,818::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28046::DEBUG::2012-12-02 14:27:12,818::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28046::DEBUG::2012-12-02 14:27:12,818::task::957::TaskManager.Task::(_decref) Task=`e826b916-a5e8-45e9-9b77-c84ec8191a0c`::ref 0 aborting False >Thread-28050::DEBUG::2012-12-02 14:27:17,722::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28050::DEBUG::2012-12-02 14:27:17,722::task::568::TaskManager.Task::(_updateState) Task=`ad35bffd-8806-4c4e-9c86-20c0139540ec`::moving from state init -> state preparing >Thread-28050::INFO::2012-12-02 14:27:17,722::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28050::INFO::2012-12-02 14:27:17,723::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28050::DEBUG::2012-12-02 14:27:17,723::task::1151::TaskManager.Task::(prepare) Task=`ad35bffd-8806-4c4e-9c86-20c0139540ec`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28050::DEBUG::2012-12-02 14:27:17,723::task::568::TaskManager.Task::(_updateState) Task=`ad35bffd-8806-4c4e-9c86-20c0139540ec`::moving from state preparing -> state finished >Thread-28050::DEBUG::2012-12-02 14:27:17,723::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28050::DEBUG::2012-12-02 14:27:17,723::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28050::DEBUG::2012-12-02 14:27:17,723::task::957::TaskManager.Task::(_decref) Task=`ad35bffd-8806-4c4e-9c86-20c0139540ec`::ref 0 aborting False >Thread-28051::DEBUG::2012-12-02 14:27:17,726::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28051::DEBUG::2012-12-02 14:27:17,726::task::568::TaskManager.Task::(_updateState) Task=`48152c02-9d80-43fb-94fb-429736a72ebd`::moving from state init -> state preparing >Thread-28051::INFO::2012-12-02 14:27:17,726::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28051::DEBUG::2012-12-02 14:27:17,726::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`52f346f4-9604-473b-a58b-476ce4672575`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28051::DEBUG::2012-12-02 14:27:17,726::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28051::DEBUG::2012-12-02 14:27:17,726::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28051::DEBUG::2012-12-02 14:27:17,726::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`52f346f4-9604-473b-a58b-476ce4672575`::Granted request >Thread-28051::DEBUG::2012-12-02 14:27:17,727::task::794::TaskManager.Task::(resourceAcquired) Task=`48152c02-9d80-43fb-94fb-429736a72ebd`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28051::DEBUG::2012-12-02 14:27:17,727::task::957::TaskManager.Task::(_decref) Task=`48152c02-9d80-43fb-94fb-429736a72ebd`::ref 1 aborting False >Thread-28051::INFO::2012-12-02 14:27:17,731::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28051::DEBUG::2012-12-02 14:27:17,731::task::1151::TaskManager.Task::(prepare) Task=`48152c02-9d80-43fb-94fb-429736a72ebd`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28051::DEBUG::2012-12-02 14:27:17,732::task::568::TaskManager.Task::(_updateState) Task=`48152c02-9d80-43fb-94fb-429736a72ebd`::moving from state preparing -> state finished >Thread-28051::DEBUG::2012-12-02 14:27:17,732::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28051::DEBUG::2012-12-02 14:27:17,732::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28051::DEBUG::2012-12-02 14:27:17,732::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28051::DEBUG::2012-12-02 14:27:17,733::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28051::DEBUG::2012-12-02 14:27:17,733::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28051::DEBUG::2012-12-02 14:27:17,733::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28051::DEBUG::2012-12-02 14:27:17,733::task::957::TaskManager.Task::(_decref) Task=`48152c02-9d80-43fb-94fb-429736a72ebd`::ref 0 aborting False >Thread-28054::DEBUG::2012-12-02 14:27:22,880::task::568::TaskManager.Task::(_updateState) Task=`40294911-efec-4f98-9b3d-9c5aad92623d`::moving from state init -> state preparing >Thread-28054::INFO::2012-12-02 14:27:22,880::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28054::INFO::2012-12-02 14:27:22,881::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833106040955', 'lastCheck': 1354451237.734405, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00395512580872', 'lastCheck': 1354451233.4994061, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00266003608704', 'lastCheck': 1354451235.3432209, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0076630115509', 'lastCheck': 1354451233.509161, 'code': 0, 'valid': True}} >Thread-28054::DEBUG::2012-12-02 14:27:22,881::task::1151::TaskManager.Task::(prepare) Task=`40294911-efec-4f98-9b3d-9c5aad92623d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833106040955', 'lastCheck': 1354451237.734405, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00395512580872', 'lastCheck': 1354451233.4994061, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00266003608704', 'lastCheck': 1354451235.3432209, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0076630115509', 'lastCheck': 1354451233.509161, 'code': 0, 'valid': True}} >Thread-28054::DEBUG::2012-12-02 14:27:22,881::task::568::TaskManager.Task::(_updateState) Task=`40294911-efec-4f98-9b3d-9c5aad92623d`::moving from state preparing -> state finished >Thread-28054::DEBUG::2012-12-02 14:27:22,881::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28054::DEBUG::2012-12-02 14:27:22,881::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28054::DEBUG::2012-12-02 14:27:22,881::task::957::TaskManager.Task::(_decref) Task=`40294911-efec-4f98-9b3d-9c5aad92623d`::ref 0 aborting False >Thread-28058::DEBUG::2012-12-02 14:27:27,875::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28058::DEBUG::2012-12-02 14:27:27,876::task::568::TaskManager.Task::(_updateState) Task=`de3a490a-843d-4c27-81c4-c9cb45a14f33`::moving from state init -> state preparing >Thread-28058::INFO::2012-12-02 14:27:27,876::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28058::INFO::2012-12-02 14:27:27,876::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28058::DEBUG::2012-12-02 14:27:27,876::task::1151::TaskManager.Task::(prepare) Task=`de3a490a-843d-4c27-81c4-c9cb45a14f33`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28058::DEBUG::2012-12-02 14:27:27,876::task::568::TaskManager.Task::(_updateState) Task=`de3a490a-843d-4c27-81c4-c9cb45a14f33`::moving from state preparing -> state finished >Thread-28058::DEBUG::2012-12-02 14:27:27,876::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28058::DEBUG::2012-12-02 14:27:27,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28058::DEBUG::2012-12-02 14:27:27,877::task::957::TaskManager.Task::(_decref) Task=`de3a490a-843d-4c27-81c4-c9cb45a14f33`::ref 0 aborting False >Thread-28059::DEBUG::2012-12-02 14:27:27,879::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28059::DEBUG::2012-12-02 14:27:27,880::task::568::TaskManager.Task::(_updateState) Task=`4e1aba10-1b06-4a85-a0c8-46d90422b456`::moving from state init -> state preparing >Thread-28059::INFO::2012-12-02 14:27:27,880::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28059::DEBUG::2012-12-02 14:27:27,880::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8301192c-bf5f-46d5-a5ff-53105fe8e05c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28059::DEBUG::2012-12-02 14:27:27,880::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28059::DEBUG::2012-12-02 14:27:27,880::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28059::DEBUG::2012-12-02 14:27:27,880::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8301192c-bf5f-46d5-a5ff-53105fe8e05c`::Granted request >Thread-28059::DEBUG::2012-12-02 14:27:27,881::task::794::TaskManager.Task::(resourceAcquired) Task=`4e1aba10-1b06-4a85-a0c8-46d90422b456`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28059::DEBUG::2012-12-02 14:27:27,881::task::957::TaskManager.Task::(_decref) Task=`4e1aba10-1b06-4a85-a0c8-46d90422b456`::ref 1 aborting False >Thread-28059::INFO::2012-12-02 14:27:27,887::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28059::DEBUG::2012-12-02 14:27:27,887::task::1151::TaskManager.Task::(prepare) Task=`4e1aba10-1b06-4a85-a0c8-46d90422b456`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28059::DEBUG::2012-12-02 14:27:27,887::task::568::TaskManager.Task::(_updateState) Task=`4e1aba10-1b06-4a85-a0c8-46d90422b456`::moving from state preparing -> state finished >Thread-28059::DEBUG::2012-12-02 14:27:27,887::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28059::DEBUG::2012-12-02 14:27:27,887::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28059::DEBUG::2012-12-02 14:27:27,887::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28059::DEBUG::2012-12-02 14:27:27,887::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28059::DEBUG::2012-12-02 14:27:27,887::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28059::DEBUG::2012-12-02 14:27:27,887::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28059::DEBUG::2012-12-02 14:27:27,887::task::957::TaskManager.Task::(_decref) Task=`4e1aba10-1b06-4a85-a0c8-46d90422b456`::ref 0 aborting False >Thread-28062::DEBUG::2012-12-02 14:27:32,968::task::568::TaskManager.Task::(_updateState) Task=`73a6942e-11ca-4316-9293-c75b93004f25`::moving from state init -> state preparing >Thread-28062::INFO::2012-12-02 14:27:32,968::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28062::INFO::2012-12-02 14:27:32,969::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00790596008301', 'lastCheck': 1354451247.7517731, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00181007385254', 'lastCheck': 1354451243.5185981, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00164699554443', 'lastCheck': 1354451245.3525491, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00741219520569', 'lastCheck': 1354451243.526937, 'code': 0, 'valid': True}} >Thread-28062::DEBUG::2012-12-02 14:27:32,969::task::1151::TaskManager.Task::(prepare) Task=`73a6942e-11ca-4316-9293-c75b93004f25`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00790596008301', 'lastCheck': 1354451247.7517731, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00181007385254', 'lastCheck': 1354451243.5185981, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00164699554443', 'lastCheck': 1354451245.3525491, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00741219520569', 'lastCheck': 1354451243.526937, 'code': 0, 'valid': True}} >Thread-28062::DEBUG::2012-12-02 14:27:32,969::task::568::TaskManager.Task::(_updateState) Task=`73a6942e-11ca-4316-9293-c75b93004f25`::moving from state preparing -> state finished >Thread-28062::DEBUG::2012-12-02 14:27:32,969::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28062::DEBUG::2012-12-02 14:27:32,969::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28062::DEBUG::2012-12-02 14:27:32,969::task::957::TaskManager.Task::(_decref) Task=`73a6942e-11ca-4316-9293-c75b93004f25`::ref 0 aborting False >Thread-28066::DEBUG::2012-12-02 14:27:38,006::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28066::DEBUG::2012-12-02 14:27:38,006::task::568::TaskManager.Task::(_updateState) Task=`bfbe863b-cd83-4041-ab61-590703953169`::moving from state init -> state preparing >Thread-28066::INFO::2012-12-02 14:27:38,007::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28066::INFO::2012-12-02 14:27:38,007::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28066::DEBUG::2012-12-02 14:27:38,007::task::1151::TaskManager.Task::(prepare) Task=`bfbe863b-cd83-4041-ab61-590703953169`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28066::DEBUG::2012-12-02 14:27:38,007::task::568::TaskManager.Task::(_updateState) Task=`bfbe863b-cd83-4041-ab61-590703953169`::moving from state preparing -> state finished >Thread-28066::DEBUG::2012-12-02 14:27:38,007::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28066::DEBUG::2012-12-02 14:27:38,007::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28066::DEBUG::2012-12-02 14:27:38,008::task::957::TaskManager.Task::(_decref) Task=`bfbe863b-cd83-4041-ab61-590703953169`::ref 0 aborting False >Thread-28067::DEBUG::2012-12-02 14:27:38,010::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28067::DEBUG::2012-12-02 14:27:38,010::task::568::TaskManager.Task::(_updateState) Task=`92529862-9d04-456d-8c04-73917436cac0`::moving from state init -> state preparing >Thread-28067::INFO::2012-12-02 14:27:38,011::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28067::DEBUG::2012-12-02 14:27:38,011::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`31bd893b-32a4-490f-a7e5-a6e7523dfdb2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28067::DEBUG::2012-12-02 14:27:38,011::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28067::DEBUG::2012-12-02 14:27:38,011::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28067::DEBUG::2012-12-02 14:27:38,011::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`31bd893b-32a4-490f-a7e5-a6e7523dfdb2`::Granted request >Thread-28067::DEBUG::2012-12-02 14:27:38,012::task::794::TaskManager.Task::(resourceAcquired) Task=`92529862-9d04-456d-8c04-73917436cac0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28067::DEBUG::2012-12-02 14:27:38,012::task::957::TaskManager.Task::(_decref) Task=`92529862-9d04-456d-8c04-73917436cac0`::ref 1 aborting False >Thread-28067::INFO::2012-12-02 14:27:38,017::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28067::DEBUG::2012-12-02 14:27:38,017::task::1151::TaskManager.Task::(prepare) Task=`92529862-9d04-456d-8c04-73917436cac0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28067::DEBUG::2012-12-02 14:27:38,017::task::568::TaskManager.Task::(_updateState) Task=`92529862-9d04-456d-8c04-73917436cac0`::moving from state preparing -> state finished >Thread-28067::DEBUG::2012-12-02 14:27:38,017::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28067::DEBUG::2012-12-02 14:27:38,018::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28067::DEBUG::2012-12-02 14:27:38,018::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28067::DEBUG::2012-12-02 14:27:38,018::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28067::DEBUG::2012-12-02 14:27:38,018::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28067::DEBUG::2012-12-02 14:27:38,018::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28067::DEBUG::2012-12-02 14:27:38,018::task::957::TaskManager.Task::(_decref) Task=`92529862-9d04-456d-8c04-73917436cac0`::ref 0 aborting False >Thread-28070::DEBUG::2012-12-02 14:27:43,124::task::568::TaskManager.Task::(_updateState) Task=`c9717ec7-16f5-4f02-8cfc-69a0dd987fa7`::moving from state init -> state preparing >Thread-28070::INFO::2012-12-02 14:27:43,124::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28070::INFO::2012-12-02 14:27:43,125::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00798511505127', 'lastCheck': 1354451257.77001, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00420212745667', 'lastCheck': 1354451253.5354841, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00303506851196', 'lastCheck': 1354451255.364439, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00786304473877', 'lastCheck': 1354451253.5453949, 'code': 0, 'valid': True}} >Thread-28070::DEBUG::2012-12-02 14:27:43,125::task::1151::TaskManager.Task::(prepare) Task=`c9717ec7-16f5-4f02-8cfc-69a0dd987fa7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00798511505127', 'lastCheck': 1354451257.77001, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00420212745667', 'lastCheck': 1354451253.5354841, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00303506851196', 'lastCheck': 1354451255.364439, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00786304473877', 'lastCheck': 1354451253.5453949, 'code': 0, 'valid': True}} >Thread-28070::DEBUG::2012-12-02 14:27:43,125::task::568::TaskManager.Task::(_updateState) Task=`c9717ec7-16f5-4f02-8cfc-69a0dd987fa7`::moving from state preparing -> state finished >Thread-28070::DEBUG::2012-12-02 14:27:43,125::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28070::DEBUG::2012-12-02 14:27:43,125::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28070::DEBUG::2012-12-02 14:27:43,125::task::957::TaskManager.Task::(_decref) Task=`c9717ec7-16f5-4f02-8cfc-69a0dd987fa7`::ref 0 aborting False >Thread-28074::DEBUG::2012-12-02 14:27:48,135::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28074::DEBUG::2012-12-02 14:27:48,136::task::568::TaskManager.Task::(_updateState) Task=`50aabd28-aa54-4ad3-9000-5199a4f9df8c`::moving from state init -> state preparing >Thread-28074::INFO::2012-12-02 14:27:48,136::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28074::INFO::2012-12-02 14:27:48,136::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28074::DEBUG::2012-12-02 14:27:48,136::task::1151::TaskManager.Task::(prepare) Task=`50aabd28-aa54-4ad3-9000-5199a4f9df8c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28074::DEBUG::2012-12-02 14:27:48,136::task::568::TaskManager.Task::(_updateState) Task=`50aabd28-aa54-4ad3-9000-5199a4f9df8c`::moving from state preparing -> state finished >Thread-28074::DEBUG::2012-12-02 14:27:48,136::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28074::DEBUG::2012-12-02 14:27:48,137::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28074::DEBUG::2012-12-02 14:27:48,137::task::957::TaskManager.Task::(_decref) Task=`50aabd28-aa54-4ad3-9000-5199a4f9df8c`::ref 0 aborting False >Thread-28075::DEBUG::2012-12-02 14:27:48,139::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28075::DEBUG::2012-12-02 14:27:48,140::task::568::TaskManager.Task::(_updateState) Task=`35449458-0794-4792-a879-e9b757d9b2a1`::moving from state init -> state preparing >Thread-28075::INFO::2012-12-02 14:27:48,140::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28075::DEBUG::2012-12-02 14:27:48,140::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a2bb240-395e-42f2-b6e6-e3ff82d2790a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28075::DEBUG::2012-12-02 14:27:48,140::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28075::DEBUG::2012-12-02 14:27:48,140::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28075::DEBUG::2012-12-02 14:27:48,140::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a2bb240-395e-42f2-b6e6-e3ff82d2790a`::Granted request >Thread-28075::DEBUG::2012-12-02 14:27:48,141::task::794::TaskManager.Task::(resourceAcquired) Task=`35449458-0794-4792-a879-e9b757d9b2a1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28075::DEBUG::2012-12-02 14:27:48,141::task::957::TaskManager.Task::(_decref) Task=`35449458-0794-4792-a879-e9b757d9b2a1`::ref 1 aborting False >Thread-28075::INFO::2012-12-02 14:27:48,147::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28075::DEBUG::2012-12-02 14:27:48,147::task::1151::TaskManager.Task::(prepare) Task=`35449458-0794-4792-a879-e9b757d9b2a1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28075::DEBUG::2012-12-02 14:27:48,147::task::568::TaskManager.Task::(_updateState) Task=`35449458-0794-4792-a879-e9b757d9b2a1`::moving from state preparing -> state finished >Thread-28075::DEBUG::2012-12-02 14:27:48,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28075::DEBUG::2012-12-02 14:27:48,147::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28075::DEBUG::2012-12-02 14:27:48,147::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28075::DEBUG::2012-12-02 14:27:48,147::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28075::DEBUG::2012-12-02 14:27:48,147::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28075::DEBUG::2012-12-02 14:27:48,147::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28075::DEBUG::2012-12-02 14:27:48,147::task::957::TaskManager.Task::(_decref) Task=`35449458-0794-4792-a879-e9b757d9b2a1`::ref 0 aborting False >Thread-28078::DEBUG::2012-12-02 14:27:53,305::task::568::TaskManager.Task::(_updateState) Task=`50d431b1-80af-46af-b9d2-64dba3a61566`::moving from state init -> state preparing >Thread-28078::INFO::2012-12-02 14:27:53,305::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28078::INFO::2012-12-02 14:27:53,305::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00777697563171', 'lastCheck': 1354451267.7874701, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00364112854004', 'lastCheck': 1354451263.5550449, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00176405906677', 'lastCheck': 1354451265.375165, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828194618225', 'lastCheck': 1354451263.5658541, 'code': 0, 'valid': True}} >Thread-28078::DEBUG::2012-12-02 14:27:53,305::task::1151::TaskManager.Task::(prepare) Task=`50d431b1-80af-46af-b9d2-64dba3a61566`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00777697563171', 'lastCheck': 1354451267.7874701, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00364112854004', 'lastCheck': 1354451263.5550449, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00176405906677', 'lastCheck': 1354451265.375165, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828194618225', 'lastCheck': 1354451263.5658541, 'code': 0, 'valid': True}} >Thread-28078::DEBUG::2012-12-02 14:27:53,305::task::568::TaskManager.Task::(_updateState) Task=`50d431b1-80af-46af-b9d2-64dba3a61566`::moving from state preparing -> state finished >Thread-28078::DEBUG::2012-12-02 14:27:53,305::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28078::DEBUG::2012-12-02 14:27:53,306::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28078::DEBUG::2012-12-02 14:27:53,306::task::957::TaskManager.Task::(_decref) Task=`50d431b1-80af-46af-b9d2-64dba3a61566`::ref 0 aborting False >Thread-28082::DEBUG::2012-12-02 14:27:58,315::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28082::DEBUG::2012-12-02 14:27:58,316::task::568::TaskManager.Task::(_updateState) Task=`60c4dcec-59bb-42a2-a7dd-d15aa432ffc4`::moving from state init -> state preparing >Thread-28082::INFO::2012-12-02 14:27:58,316::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28082::INFO::2012-12-02 14:27:58,316::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28082::DEBUG::2012-12-02 14:27:58,316::task::1151::TaskManager.Task::(prepare) Task=`60c4dcec-59bb-42a2-a7dd-d15aa432ffc4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28082::DEBUG::2012-12-02 14:27:58,316::task::568::TaskManager.Task::(_updateState) Task=`60c4dcec-59bb-42a2-a7dd-d15aa432ffc4`::moving from state preparing -> state finished >Thread-28082::DEBUG::2012-12-02 14:27:58,316::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28082::DEBUG::2012-12-02 14:27:58,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28082::DEBUG::2012-12-02 14:27:58,317::task::957::TaskManager.Task::(_decref) Task=`60c4dcec-59bb-42a2-a7dd-d15aa432ffc4`::ref 0 aborting False >Thread-28083::DEBUG::2012-12-02 14:27:58,319::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28083::DEBUG::2012-12-02 14:27:58,319::task::568::TaskManager.Task::(_updateState) Task=`dd9a797b-131b-40dc-aca4-db9de49c1ee0`::moving from state init -> state preparing >Thread-28083::INFO::2012-12-02 14:27:58,320::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28083::DEBUG::2012-12-02 14:27:58,320::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a750ccb-746d-4125-ac6b-2c1e65e235da`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28083::DEBUG::2012-12-02 14:27:58,320::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28083::DEBUG::2012-12-02 14:27:58,320::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28083::DEBUG::2012-12-02 14:27:58,320::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a750ccb-746d-4125-ac6b-2c1e65e235da`::Granted request >Thread-28083::DEBUG::2012-12-02 14:27:58,321::task::794::TaskManager.Task::(resourceAcquired) Task=`dd9a797b-131b-40dc-aca4-db9de49c1ee0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28083::DEBUG::2012-12-02 14:27:58,321::task::957::TaskManager.Task::(_decref) Task=`dd9a797b-131b-40dc-aca4-db9de49c1ee0`::ref 1 aborting False >Thread-28083::INFO::2012-12-02 14:27:58,327::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28083::DEBUG::2012-12-02 14:27:58,327::task::1151::TaskManager.Task::(prepare) Task=`dd9a797b-131b-40dc-aca4-db9de49c1ee0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28083::DEBUG::2012-12-02 14:27:58,327::task::568::TaskManager.Task::(_updateState) Task=`dd9a797b-131b-40dc-aca4-db9de49c1ee0`::moving from state preparing -> state finished >Thread-28083::DEBUG::2012-12-02 14:27:58,327::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28083::DEBUG::2012-12-02 14:27:58,327::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28083::DEBUG::2012-12-02 14:27:58,327::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28083::DEBUG::2012-12-02 14:27:58,327::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28083::DEBUG::2012-12-02 14:27:58,327::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28083::DEBUG::2012-12-02 14:27:58,327::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28083::DEBUG::2012-12-02 14:27:58,327::task::957::TaskManager.Task::(_decref) Task=`dd9a797b-131b-40dc-aca4-db9de49c1ee0`::ref 0 aborting False >Thread-28086::DEBUG::2012-12-02 14:28:03,426::task::568::TaskManager.Task::(_updateState) Task=`5c6ab66a-edec-4632-8713-82039ae3371c`::moving from state init -> state preparing >Thread-28086::INFO::2012-12-02 14:28:03,426::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28086::INFO::2012-12-02 14:28:03,427::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0107889175415', 'lastCheck': 1354451277.808018, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00220704078674', 'lastCheck': 1354451273.5748529, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199389457703', 'lastCheck': 1354451275.3858969, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00800180435181', 'lastCheck': 1354451273.585398, 'code': 0, 'valid': True}} >Thread-28086::DEBUG::2012-12-02 14:28:03,427::task::1151::TaskManager.Task::(prepare) Task=`5c6ab66a-edec-4632-8713-82039ae3371c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0107889175415', 'lastCheck': 1354451277.808018, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00220704078674', 'lastCheck': 1354451273.5748529, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199389457703', 'lastCheck': 1354451275.3858969, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00800180435181', 'lastCheck': 1354451273.585398, 'code': 0, 'valid': True}} >Thread-28086::DEBUG::2012-12-02 14:28:03,427::task::568::TaskManager.Task::(_updateState) Task=`5c6ab66a-edec-4632-8713-82039ae3371c`::moving from state preparing -> state finished >Thread-28086::DEBUG::2012-12-02 14:28:03,427::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28086::DEBUG::2012-12-02 14:28:03,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28086::DEBUG::2012-12-02 14:28:03,427::task::957::TaskManager.Task::(_decref) Task=`5c6ab66a-edec-4632-8713-82039ae3371c`::ref 0 aborting False >Thread-28090::DEBUG::2012-12-02 14:28:08,511::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28090::DEBUG::2012-12-02 14:28:08,512::task::568::TaskManager.Task::(_updateState) Task=`869feadc-0079-4e6d-b47a-dddb24053a6b`::moving from state init -> state preparing >Thread-28090::INFO::2012-12-02 14:28:08,512::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28090::INFO::2012-12-02 14:28:08,512::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28090::DEBUG::2012-12-02 14:28:08,512::task::1151::TaskManager.Task::(prepare) Task=`869feadc-0079-4e6d-b47a-dddb24053a6b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28090::DEBUG::2012-12-02 14:28:08,512::task::568::TaskManager.Task::(_updateState) Task=`869feadc-0079-4e6d-b47a-dddb24053a6b`::moving from state preparing -> state finished >Thread-28090::DEBUG::2012-12-02 14:28:08,512::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28090::DEBUG::2012-12-02 14:28:08,513::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28090::DEBUG::2012-12-02 14:28:08,513::task::957::TaskManager.Task::(_decref) Task=`869feadc-0079-4e6d-b47a-dddb24053a6b`::ref 0 aborting False >Thread-28091::DEBUG::2012-12-02 14:28:08,515::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28091::DEBUG::2012-12-02 14:28:08,515::task::568::TaskManager.Task::(_updateState) Task=`a60128cd-bf0e-4747-bcc2-a9fbcb86f99e`::moving from state init -> state preparing >Thread-28091::INFO::2012-12-02 14:28:08,516::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28091::DEBUG::2012-12-02 14:28:08,516::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`09b3cef7-6482-4b69-b732-c331fbceead1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28091::DEBUG::2012-12-02 14:28:08,516::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28091::DEBUG::2012-12-02 14:28:08,516::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28091::DEBUG::2012-12-02 14:28:08,516::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`09b3cef7-6482-4b69-b732-c331fbceead1`::Granted request >Thread-28091::DEBUG::2012-12-02 14:28:08,516::task::794::TaskManager.Task::(resourceAcquired) Task=`a60128cd-bf0e-4747-bcc2-a9fbcb86f99e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28091::DEBUG::2012-12-02 14:28:08,517::task::957::TaskManager.Task::(_decref) Task=`a60128cd-bf0e-4747-bcc2-a9fbcb86f99e`::ref 1 aborting False >Thread-28091::INFO::2012-12-02 14:28:08,521::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28091::DEBUG::2012-12-02 14:28:08,521::task::1151::TaskManager.Task::(prepare) Task=`a60128cd-bf0e-4747-bcc2-a9fbcb86f99e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28091::DEBUG::2012-12-02 14:28:08,521::task::568::TaskManager.Task::(_updateState) Task=`a60128cd-bf0e-4747-bcc2-a9fbcb86f99e`::moving from state preparing -> state finished >Thread-28091::DEBUG::2012-12-02 14:28:08,521::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28091::DEBUG::2012-12-02 14:28:08,521::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28091::DEBUG::2012-12-02 14:28:08,521::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28091::DEBUG::2012-12-02 14:28:08,521::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28091::DEBUG::2012-12-02 14:28:08,522::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28091::DEBUG::2012-12-02 14:28:08,522::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28091::DEBUG::2012-12-02 14:28:08,522::task::957::TaskManager.Task::(_decref) Task=`a60128cd-bf0e-4747-bcc2-a9fbcb86f99e`::ref 0 aborting False >Thread-28094::DEBUG::2012-12-02 14:28:13,531::task::568::TaskManager.Task::(_updateState) Task=`23d61365-999d-47fc-a6c1-1a8c4c154c8a`::moving from state init -> state preparing >Thread-28094::INFO::2012-12-02 14:28:13,532::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28094::INFO::2012-12-02 14:28:13,532::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00803780555725', 'lastCheck': 1354451287.8246641, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00237393379211', 'lastCheck': 1354451283.593318, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00173592567444', 'lastCheck': 1354451285.3962791, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0090229511261', 'lastCheck': 1354451283.605206, 'code': 0, 'valid': True}} >Thread-28094::DEBUG::2012-12-02 14:28:13,532::task::1151::TaskManager.Task::(prepare) Task=`23d61365-999d-47fc-a6c1-1a8c4c154c8a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00803780555725', 'lastCheck': 1354451287.8246641, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00237393379211', 'lastCheck': 1354451283.593318, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00173592567444', 'lastCheck': 1354451285.3962791, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0090229511261', 'lastCheck': 1354451283.605206, 'code': 0, 'valid': True}} >Thread-28094::DEBUG::2012-12-02 14:28:13,532::task::568::TaskManager.Task::(_updateState) Task=`23d61365-999d-47fc-a6c1-1a8c4c154c8a`::moving from state preparing -> state finished >Thread-28094::DEBUG::2012-12-02 14:28:13,532::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28094::DEBUG::2012-12-02 14:28:13,533::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28094::DEBUG::2012-12-02 14:28:13,533::task::957::TaskManager.Task::(_decref) Task=`23d61365-999d-47fc-a6c1-1a8c4c154c8a`::ref 0 aborting False >Thread-28098::DEBUG::2012-12-02 14:28:18,651::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28098::DEBUG::2012-12-02 14:28:18,651::task::568::TaskManager.Task::(_updateState) Task=`0a150f0e-9c3d-4cec-8bf8-d62bb1c3756b`::moving from state init -> state preparing >Thread-28098::INFO::2012-12-02 14:28:18,652::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28098::INFO::2012-12-02 14:28:18,652::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28098::DEBUG::2012-12-02 14:28:18,652::task::1151::TaskManager.Task::(prepare) Task=`0a150f0e-9c3d-4cec-8bf8-d62bb1c3756b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28098::DEBUG::2012-12-02 14:28:18,652::task::568::TaskManager.Task::(_updateState) Task=`0a150f0e-9c3d-4cec-8bf8-d62bb1c3756b`::moving from state preparing -> state finished >Thread-28098::DEBUG::2012-12-02 14:28:18,652::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28098::DEBUG::2012-12-02 14:28:18,652::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28098::DEBUG::2012-12-02 14:28:18,652::task::957::TaskManager.Task::(_decref) Task=`0a150f0e-9c3d-4cec-8bf8-d62bb1c3756b`::ref 0 aborting False >Thread-28099::DEBUG::2012-12-02 14:28:18,655::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28099::DEBUG::2012-12-02 14:28:18,655::task::568::TaskManager.Task::(_updateState) Task=`d53b5391-6bc3-45f9-9867-a5fe7c78bd70`::moving from state init -> state preparing >Thread-28099::INFO::2012-12-02 14:28:18,655::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28099::DEBUG::2012-12-02 14:28:18,656::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`899fce93-df01-4c2b-9917-6b3ff16cf8cc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28099::DEBUG::2012-12-02 14:28:18,656::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28099::DEBUG::2012-12-02 14:28:18,656::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28099::DEBUG::2012-12-02 14:28:18,656::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`899fce93-df01-4c2b-9917-6b3ff16cf8cc`::Granted request >Thread-28099::DEBUG::2012-12-02 14:28:18,656::task::794::TaskManager.Task::(resourceAcquired) Task=`d53b5391-6bc3-45f9-9867-a5fe7c78bd70`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28099::DEBUG::2012-12-02 14:28:18,657::task::957::TaskManager.Task::(_decref) Task=`d53b5391-6bc3-45f9-9867-a5fe7c78bd70`::ref 1 aborting False >Thread-28099::INFO::2012-12-02 14:28:18,663::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28099::DEBUG::2012-12-02 14:28:18,663::task::1151::TaskManager.Task::(prepare) Task=`d53b5391-6bc3-45f9-9867-a5fe7c78bd70`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28099::DEBUG::2012-12-02 14:28:18,663::task::568::TaskManager.Task::(_updateState) Task=`d53b5391-6bc3-45f9-9867-a5fe7c78bd70`::moving from state preparing -> state finished >Thread-28099::DEBUG::2012-12-02 14:28:18,663::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28099::DEBUG::2012-12-02 14:28:18,664::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28099::DEBUG::2012-12-02 14:28:18,664::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28099::DEBUG::2012-12-02 14:28:18,664::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28099::DEBUG::2012-12-02 14:28:18,664::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28099::DEBUG::2012-12-02 14:28:18,664::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28099::DEBUG::2012-12-02 14:28:18,664::task::957::TaskManager.Task::(_decref) Task=`d53b5391-6bc3-45f9-9867-a5fe7c78bd70`::ref 0 aborting False >Thread-28102::DEBUG::2012-12-02 14:28:23,661::task::568::TaskManager.Task::(_updateState) Task=`ba0f9dfc-55fa-4d2d-950d-4e3ec0e37455`::moving from state init -> state preparing >Thread-28102::INFO::2012-12-02 14:28:23,661::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28102::INFO::2012-12-02 14:28:23,661::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00734400749207', 'lastCheck': 1354451297.842016, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00317478179932', 'lastCheck': 1354451303.6346381, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00352692604065', 'lastCheck': 1354451295.4103651, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0104620456696', 'lastCheck': 1354451303.6433671, 'code': 0, 'valid': True}} >Thread-28102::DEBUG::2012-12-02 14:28:23,662::task::1151::TaskManager.Task::(prepare) Task=`ba0f9dfc-55fa-4d2d-950d-4e3ec0e37455`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00734400749207', 'lastCheck': 1354451297.842016, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00317478179932', 'lastCheck': 1354451303.6346381, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00352692604065', 'lastCheck': 1354451295.4103651, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0104620456696', 'lastCheck': 1354451303.6433671, 'code': 0, 'valid': True}} >Thread-28102::DEBUG::2012-12-02 14:28:23,662::task::568::TaskManager.Task::(_updateState) Task=`ba0f9dfc-55fa-4d2d-950d-4e3ec0e37455`::moving from state preparing -> state finished >Thread-28102::DEBUG::2012-12-02 14:28:23,662::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28102::DEBUG::2012-12-02 14:28:23,662::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28102::DEBUG::2012-12-02 14:28:23,662::task::957::TaskManager.Task::(_decref) Task=`ba0f9dfc-55fa-4d2d-950d-4e3ec0e37455`::ref 0 aborting False >Thread-28106::DEBUG::2012-12-02 14:28:28,805::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28106::DEBUG::2012-12-02 14:28:28,806::task::568::TaskManager.Task::(_updateState) Task=`d709690f-8b2d-4060-89d4-f848804610fd`::moving from state init -> state preparing >Thread-28106::INFO::2012-12-02 14:28:28,806::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28106::INFO::2012-12-02 14:28:28,806::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28106::DEBUG::2012-12-02 14:28:28,806::task::1151::TaskManager.Task::(prepare) Task=`d709690f-8b2d-4060-89d4-f848804610fd`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28106::DEBUG::2012-12-02 14:28:28,806::task::568::TaskManager.Task::(_updateState) Task=`d709690f-8b2d-4060-89d4-f848804610fd`::moving from state preparing -> state finished >Thread-28106::DEBUG::2012-12-02 14:28:28,806::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28106::DEBUG::2012-12-02 14:28:28,807::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28106::DEBUG::2012-12-02 14:28:28,807::task::957::TaskManager.Task::(_decref) Task=`d709690f-8b2d-4060-89d4-f848804610fd`::ref 0 aborting False >Thread-28107::DEBUG::2012-12-02 14:28:28,809::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28107::DEBUG::2012-12-02 14:28:28,810::task::568::TaskManager.Task::(_updateState) Task=`3f208de6-af2c-4e20-a9fe-31502859a516`::moving from state init -> state preparing >Thread-28107::INFO::2012-12-02 14:28:28,810::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28107::DEBUG::2012-12-02 14:28:28,810::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5fb79701-d35e-4f38-8953-2d4a3a7fe33b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28107::DEBUG::2012-12-02 14:28:28,810::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28107::DEBUG::2012-12-02 14:28:28,810::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28107::DEBUG::2012-12-02 14:28:28,810::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5fb79701-d35e-4f38-8953-2d4a3a7fe33b`::Granted request >Thread-28107::DEBUG::2012-12-02 14:28:28,811::task::794::TaskManager.Task::(resourceAcquired) Task=`3f208de6-af2c-4e20-a9fe-31502859a516`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28107::DEBUG::2012-12-02 14:28:28,811::task::957::TaskManager.Task::(_decref) Task=`3f208de6-af2c-4e20-a9fe-31502859a516`::ref 1 aborting False >Thread-28107::INFO::2012-12-02 14:28:28,817::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28107::DEBUG::2012-12-02 14:28:28,817::task::1151::TaskManager.Task::(prepare) Task=`3f208de6-af2c-4e20-a9fe-31502859a516`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28107::DEBUG::2012-12-02 14:28:28,817::task::568::TaskManager.Task::(_updateState) Task=`3f208de6-af2c-4e20-a9fe-31502859a516`::moving from state preparing -> state finished >Thread-28107::DEBUG::2012-12-02 14:28:28,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28107::DEBUG::2012-12-02 14:28:28,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28107::DEBUG::2012-12-02 14:28:28,817::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28107::DEBUG::2012-12-02 14:28:28,817::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28107::DEBUG::2012-12-02 14:28:28,817::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28107::DEBUG::2012-12-02 14:28:28,817::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28107::DEBUG::2012-12-02 14:28:28,818::task::957::TaskManager.Task::(_decref) Task=`3f208de6-af2c-4e20-a9fe-31502859a516`::ref 0 aborting False >Thread-28110::DEBUG::2012-12-02 14:28:33,800::task::568::TaskManager.Task::(_updateState) Task=`03c4f46a-d3a7-4a9d-bf8c-eb999a5b7160`::moving from state init -> state preparing >Thread-28110::INFO::2012-12-02 14:28:33,800::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28110::INFO::2012-12-02 14:28:33,800::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00804400444031', 'lastCheck': 1354451307.8588531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00497603416443', 'lastCheck': 1354451313.6503019, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00194406509399', 'lastCheck': 1354451305.4210391, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0094199180603', 'lastCheck': 1354451313.6620359, 'code': 0, 'valid': True}} >Thread-28110::DEBUG::2012-12-02 14:28:33,800::task::1151::TaskManager.Task::(prepare) Task=`03c4f46a-d3a7-4a9d-bf8c-eb999a5b7160`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00804400444031', 'lastCheck': 1354451307.8588531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00497603416443', 'lastCheck': 1354451313.6503019, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00194406509399', 'lastCheck': 1354451305.4210391, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0094199180603', 'lastCheck': 1354451313.6620359, 'code': 0, 'valid': True}} >Thread-28110::DEBUG::2012-12-02 14:28:33,800::task::568::TaskManager.Task::(_updateState) Task=`03c4f46a-d3a7-4a9d-bf8c-eb999a5b7160`::moving from state preparing -> state finished >Thread-28110::DEBUG::2012-12-02 14:28:33,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28110::DEBUG::2012-12-02 14:28:33,801::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28110::DEBUG::2012-12-02 14:28:33,801::task::957::TaskManager.Task::(_decref) Task=`03c4f46a-d3a7-4a9d-bf8c-eb999a5b7160`::ref 0 aborting False >Thread-28114::DEBUG::2012-12-02 14:28:38,962::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28114::DEBUG::2012-12-02 14:28:38,962::task::568::TaskManager.Task::(_updateState) Task=`255113d1-fc11-45b4-b00e-9bb141a22e3e`::moving from state init -> state preparing >Thread-28114::INFO::2012-12-02 14:28:38,962::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28114::INFO::2012-12-02 14:28:38,963::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28114::DEBUG::2012-12-02 14:28:38,963::task::1151::TaskManager.Task::(prepare) Task=`255113d1-fc11-45b4-b00e-9bb141a22e3e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28114::DEBUG::2012-12-02 14:28:38,963::task::568::TaskManager.Task::(_updateState) Task=`255113d1-fc11-45b4-b00e-9bb141a22e3e`::moving from state preparing -> state finished >Thread-28114::DEBUG::2012-12-02 14:28:38,963::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28114::DEBUG::2012-12-02 14:28:38,963::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28114::DEBUG::2012-12-02 14:28:38,963::task::957::TaskManager.Task::(_decref) Task=`255113d1-fc11-45b4-b00e-9bb141a22e3e`::ref 0 aborting False >Thread-28115::DEBUG::2012-12-02 14:28:38,966::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28115::DEBUG::2012-12-02 14:28:38,966::task::568::TaskManager.Task::(_updateState) Task=`74c96cc0-9a96-45b5-b5c4-d0449ea0729c`::moving from state init -> state preparing >Thread-28115::INFO::2012-12-02 14:28:38,966::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28115::DEBUG::2012-12-02 14:28:38,966::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2dd83a17-3de2-415a-944e-6b8e1c0ed545`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28115::DEBUG::2012-12-02 14:28:38,966::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28115::DEBUG::2012-12-02 14:28:38,967::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28115::DEBUG::2012-12-02 14:28:38,967::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2dd83a17-3de2-415a-944e-6b8e1c0ed545`::Granted request >Thread-28115::DEBUG::2012-12-02 14:28:38,967::task::794::TaskManager.Task::(resourceAcquired) Task=`74c96cc0-9a96-45b5-b5c4-d0449ea0729c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28115::DEBUG::2012-12-02 14:28:38,967::task::957::TaskManager.Task::(_decref) Task=`74c96cc0-9a96-45b5-b5c4-d0449ea0729c`::ref 1 aborting False >Thread-28115::INFO::2012-12-02 14:28:38,974::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28115::DEBUG::2012-12-02 14:28:38,974::task::1151::TaskManager.Task::(prepare) Task=`74c96cc0-9a96-45b5-b5c4-d0449ea0729c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28115::DEBUG::2012-12-02 14:28:38,974::task::568::TaskManager.Task::(_updateState) Task=`74c96cc0-9a96-45b5-b5c4-d0449ea0729c`::moving from state preparing -> state finished >Thread-28115::DEBUG::2012-12-02 14:28:38,974::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28115::DEBUG::2012-12-02 14:28:38,974::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28115::DEBUG::2012-12-02 14:28:38,974::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28115::DEBUG::2012-12-02 14:28:38,975::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28115::DEBUG::2012-12-02 14:28:38,975::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28115::DEBUG::2012-12-02 14:28:38,975::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28115::DEBUG::2012-12-02 14:28:38,975::task::957::TaskManager.Task::(_decref) Task=`74c96cc0-9a96-45b5-b5c4-d0449ea0729c`::ref 0 aborting False >Thread-28118::DEBUG::2012-12-02 14:28:43,940::task::568::TaskManager.Task::(_updateState) Task=`7527d885-50de-40de-8ac3-859bfb98b9d4`::moving from state init -> state preparing >Thread-28118::INFO::2012-12-02 14:28:43,940::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28118::INFO::2012-12-02 14:28:43,940::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00850296020508', 'lastCheck': 1354451317.878634, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00412511825562', 'lastCheck': 1354451323.66927, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0028829574585', 'lastCheck': 1354451315.43325, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00850200653076', 'lastCheck': 1354451323.681978, 'code': 0, 'valid': True}} >Thread-28118::DEBUG::2012-12-02 14:28:43,940::task::1151::TaskManager.Task::(prepare) Task=`7527d885-50de-40de-8ac3-859bfb98b9d4`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00850296020508', 'lastCheck': 1354451317.878634, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00412511825562', 'lastCheck': 1354451323.66927, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0028829574585', 'lastCheck': 1354451315.43325, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00850200653076', 'lastCheck': 1354451323.681978, 'code': 0, 'valid': True}} >Thread-28118::DEBUG::2012-12-02 14:28:43,940::task::568::TaskManager.Task::(_updateState) Task=`7527d885-50de-40de-8ac3-859bfb98b9d4`::moving from state preparing -> state finished >Thread-28118::DEBUG::2012-12-02 14:28:43,940::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28118::DEBUG::2012-12-02 14:28:43,941::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28118::DEBUG::2012-12-02 14:28:43,941::task::957::TaskManager.Task::(_decref) Task=`7527d885-50de-40de-8ac3-859bfb98b9d4`::ref 0 aborting False >Thread-28122::DEBUG::2012-12-02 14:28:49,142::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28122::DEBUG::2012-12-02 14:28:49,142::task::568::TaskManager.Task::(_updateState) Task=`361e61d4-cbb0-4dbd-8059-a4c767fa4863`::moving from state init -> state preparing >Thread-28122::INFO::2012-12-02 14:28:49,143::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28122::INFO::2012-12-02 14:28:49,143::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28122::DEBUG::2012-12-02 14:28:49,143::task::1151::TaskManager.Task::(prepare) Task=`361e61d4-cbb0-4dbd-8059-a4c767fa4863`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28122::DEBUG::2012-12-02 14:28:49,143::task::568::TaskManager.Task::(_updateState) Task=`361e61d4-cbb0-4dbd-8059-a4c767fa4863`::moving from state preparing -> state finished >Thread-28122::DEBUG::2012-12-02 14:28:49,143::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28122::DEBUG::2012-12-02 14:28:49,143::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28122::DEBUG::2012-12-02 14:28:49,143::task::957::TaskManager.Task::(_decref) Task=`361e61d4-cbb0-4dbd-8059-a4c767fa4863`::ref 0 aborting False >Thread-28123::DEBUG::2012-12-02 14:28:49,146::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28123::DEBUG::2012-12-02 14:28:49,146::task::568::TaskManager.Task::(_updateState) Task=`e17c8e5d-6604-4885-be15-63697c94054c`::moving from state init -> state preparing >Thread-28123::INFO::2012-12-02 14:28:49,146::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28123::DEBUG::2012-12-02 14:28:49,147::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a362df19-5517-47a1-91d0-d9645905ce83`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28123::DEBUG::2012-12-02 14:28:49,147::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28123::DEBUG::2012-12-02 14:28:49,147::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28123::DEBUG::2012-12-02 14:28:49,147::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a362df19-5517-47a1-91d0-d9645905ce83`::Granted request >Thread-28123::DEBUG::2012-12-02 14:28:49,147::task::794::TaskManager.Task::(resourceAcquired) Task=`e17c8e5d-6604-4885-be15-63697c94054c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28123::DEBUG::2012-12-02 14:28:49,148::task::957::TaskManager.Task::(_decref) Task=`e17c8e5d-6604-4885-be15-63697c94054c`::ref 1 aborting False >Thread-28123::INFO::2012-12-02 14:28:49,154::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28123::DEBUG::2012-12-02 14:28:49,154::task::1151::TaskManager.Task::(prepare) Task=`e17c8e5d-6604-4885-be15-63697c94054c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28123::DEBUG::2012-12-02 14:28:49,154::task::568::TaskManager.Task::(_updateState) Task=`e17c8e5d-6604-4885-be15-63697c94054c`::moving from state preparing -> state finished >Thread-28123::DEBUG::2012-12-02 14:28:49,154::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28123::DEBUG::2012-12-02 14:28:49,155::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28123::DEBUG::2012-12-02 14:28:49,155::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28123::DEBUG::2012-12-02 14:28:49,155::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28123::DEBUG::2012-12-02 14:28:49,155::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28123::DEBUG::2012-12-02 14:28:49,155::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28123::DEBUG::2012-12-02 14:28:49,155::task::957::TaskManager.Task::(_decref) Task=`e17c8e5d-6604-4885-be15-63697c94054c`::ref 0 aborting False >Thread-28126::DEBUG::2012-12-02 14:28:54,088::task::568::TaskManager.Task::(_updateState) Task=`95aa9fba-e634-4ae1-b490-42436540e6b3`::moving from state init -> state preparing >Thread-28126::INFO::2012-12-02 14:28:54,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28126::INFO::2012-12-02 14:28:54,088::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00782895088196', 'lastCheck': 1354451327.8957529, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00279188156128', 'lastCheck': 1354451333.68346, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00210618972778', 'lastCheck': 1354451325.443058, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00768899917603', 'lastCheck': 1354451333.6984921, 'code': 0, 'valid': True}} >Thread-28126::DEBUG::2012-12-02 14:28:54,088::task::1151::TaskManager.Task::(prepare) Task=`95aa9fba-e634-4ae1-b490-42436540e6b3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00782895088196', 'lastCheck': 1354451327.8957529, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00279188156128', 'lastCheck': 1354451333.68346, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00210618972778', 'lastCheck': 1354451325.443058, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00768899917603', 'lastCheck': 1354451333.6984921, 'code': 0, 'valid': True}} >Thread-28126::DEBUG::2012-12-02 14:28:54,088::task::568::TaskManager.Task::(_updateState) Task=`95aa9fba-e634-4ae1-b490-42436540e6b3`::moving from state preparing -> state finished >Thread-28126::DEBUG::2012-12-02 14:28:54,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28126::DEBUG::2012-12-02 14:28:54,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28126::DEBUG::2012-12-02 14:28:54,089::task::957::TaskManager.Task::(_decref) Task=`95aa9fba-e634-4ae1-b490-42436540e6b3`::ref 0 aborting False >Thread-28130::DEBUG::2012-12-02 14:28:59,296::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28130::DEBUG::2012-12-02 14:28:59,297::task::568::TaskManager.Task::(_updateState) Task=`1615b69d-057d-48f6-835f-8c5a46161d26`::moving from state init -> state preparing >Thread-28130::INFO::2012-12-02 14:28:59,297::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28130::INFO::2012-12-02 14:28:59,297::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28130::DEBUG::2012-12-02 14:28:59,297::task::1151::TaskManager.Task::(prepare) Task=`1615b69d-057d-48f6-835f-8c5a46161d26`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28130::DEBUG::2012-12-02 14:28:59,297::task::568::TaskManager.Task::(_updateState) Task=`1615b69d-057d-48f6-835f-8c5a46161d26`::moving from state preparing -> state finished >Thread-28130::DEBUG::2012-12-02 14:28:59,297::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28130::DEBUG::2012-12-02 14:28:59,298::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28130::DEBUG::2012-12-02 14:28:59,298::task::957::TaskManager.Task::(_decref) Task=`1615b69d-057d-48f6-835f-8c5a46161d26`::ref 0 aborting False >Thread-28131::DEBUG::2012-12-02 14:28:59,300::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28131::DEBUG::2012-12-02 14:28:59,300::task::568::TaskManager.Task::(_updateState) Task=`25b2fda7-fd5a-4424-8f70-39149ca5b1e5`::moving from state init -> state preparing >Thread-28131::INFO::2012-12-02 14:28:59,301::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28131::DEBUG::2012-12-02 14:28:59,301::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c885f17c-9229-48f1-89a1-61a452e8910b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28131::DEBUG::2012-12-02 14:28:59,301::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28131::DEBUG::2012-12-02 14:28:59,301::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28131::DEBUG::2012-12-02 14:28:59,301::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c885f17c-9229-48f1-89a1-61a452e8910b`::Granted request >Thread-28131::DEBUG::2012-12-02 14:28:59,302::task::794::TaskManager.Task::(resourceAcquired) Task=`25b2fda7-fd5a-4424-8f70-39149ca5b1e5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28131::DEBUG::2012-12-02 14:28:59,302::task::957::TaskManager.Task::(_decref) Task=`25b2fda7-fd5a-4424-8f70-39149ca5b1e5`::ref 1 aborting False >Thread-28131::INFO::2012-12-02 14:28:59,307::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28131::DEBUG::2012-12-02 14:28:59,307::task::1151::TaskManager.Task::(prepare) Task=`25b2fda7-fd5a-4424-8f70-39149ca5b1e5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28131::DEBUG::2012-12-02 14:28:59,308::task::568::TaskManager.Task::(_updateState) Task=`25b2fda7-fd5a-4424-8f70-39149ca5b1e5`::moving from state preparing -> state finished >Thread-28131::DEBUG::2012-12-02 14:28:59,308::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28131::DEBUG::2012-12-02 14:28:59,308::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28131::DEBUG::2012-12-02 14:28:59,308::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28131::DEBUG::2012-12-02 14:28:59,308::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28131::DEBUG::2012-12-02 14:28:59,308::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28131::DEBUG::2012-12-02 14:28:59,308::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28131::DEBUG::2012-12-02 14:28:59,308::task::957::TaskManager.Task::(_decref) Task=`25b2fda7-fd5a-4424-8f70-39149ca5b1e5`::ref 0 aborting False >Thread-28134::DEBUG::2012-12-02 14:29:04,174::task::568::TaskManager.Task::(_updateState) Task=`6994fcc1-d019-41cd-87c0-6084879151b5`::moving from state init -> state preparing >Thread-28134::INFO::2012-12-02 14:29:04,175::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28134::INFO::2012-12-02 14:29:04,175::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0118968486786', 'lastCheck': 1354451337.919208, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00173306465149', 'lastCheck': 1354451343.7766621, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171685218811', 'lastCheck': 1354451335.452493, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00429701805115', 'lastCheck': 1354451343.777967, 'code': 0, 'valid': True}} >Thread-28134::DEBUG::2012-12-02 14:29:04,175::task::1151::TaskManager.Task::(prepare) Task=`6994fcc1-d019-41cd-87c0-6084879151b5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0118968486786', 'lastCheck': 1354451337.919208, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00173306465149', 'lastCheck': 1354451343.7766621, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171685218811', 'lastCheck': 1354451335.452493, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00429701805115', 'lastCheck': 1354451343.777967, 'code': 0, 'valid': True}} >Thread-28134::DEBUG::2012-12-02 14:29:04,175::task::568::TaskManager.Task::(_updateState) Task=`6994fcc1-d019-41cd-87c0-6084879151b5`::moving from state preparing -> state finished >Thread-28134::DEBUG::2012-12-02 14:29:04,175::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28134::DEBUG::2012-12-02 14:29:04,175::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28134::DEBUG::2012-12-02 14:29:04,175::task::957::TaskManager.Task::(_decref) Task=`6994fcc1-d019-41cd-87c0-6084879151b5`::ref 0 aborting False >Thread-28138::DEBUG::2012-12-02 14:29:09,402::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28138::DEBUG::2012-12-02 14:29:09,402::task::568::TaskManager.Task::(_updateState) Task=`4d3731ab-9ea2-43a1-b367-7965bf0f1990`::moving from state init -> state preparing >Thread-28138::INFO::2012-12-02 14:29:09,403::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28138::INFO::2012-12-02 14:29:09,403::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28138::DEBUG::2012-12-02 14:29:09,403::task::1151::TaskManager.Task::(prepare) Task=`4d3731ab-9ea2-43a1-b367-7965bf0f1990`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28138::DEBUG::2012-12-02 14:29:09,403::task::568::TaskManager.Task::(_updateState) Task=`4d3731ab-9ea2-43a1-b367-7965bf0f1990`::moving from state preparing -> state finished >Thread-28138::DEBUG::2012-12-02 14:29:09,403::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28138::DEBUG::2012-12-02 14:29:09,403::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28138::DEBUG::2012-12-02 14:29:09,403::task::957::TaskManager.Task::(_decref) Task=`4d3731ab-9ea2-43a1-b367-7965bf0f1990`::ref 0 aborting False >Thread-28139::DEBUG::2012-12-02 14:29:09,406::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28139::DEBUG::2012-12-02 14:29:09,406::task::568::TaskManager.Task::(_updateState) Task=`32f94075-8ff6-43bf-afed-ce38494678d7`::moving from state init -> state preparing >Thread-28139::INFO::2012-12-02 14:29:09,406::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28139::DEBUG::2012-12-02 14:29:09,407::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b6370dc5-e6ec-41cf-ae41-d0e3470797f7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28139::DEBUG::2012-12-02 14:29:09,407::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28139::DEBUG::2012-12-02 14:29:09,407::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28139::DEBUG::2012-12-02 14:29:09,407::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b6370dc5-e6ec-41cf-ae41-d0e3470797f7`::Granted request >Thread-28139::DEBUG::2012-12-02 14:29:09,407::task::794::TaskManager.Task::(resourceAcquired) Task=`32f94075-8ff6-43bf-afed-ce38494678d7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28139::DEBUG::2012-12-02 14:29:09,408::task::957::TaskManager.Task::(_decref) Task=`32f94075-8ff6-43bf-afed-ce38494678d7`::ref 1 aborting False >Thread-28139::INFO::2012-12-02 14:29:09,414::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28139::DEBUG::2012-12-02 14:29:09,414::task::1151::TaskManager.Task::(prepare) Task=`32f94075-8ff6-43bf-afed-ce38494678d7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28139::DEBUG::2012-12-02 14:29:09,414::task::568::TaskManager.Task::(_updateState) Task=`32f94075-8ff6-43bf-afed-ce38494678d7`::moving from state preparing -> state finished >Thread-28139::DEBUG::2012-12-02 14:29:09,414::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28139::DEBUG::2012-12-02 14:29:09,415::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28139::DEBUG::2012-12-02 14:29:09,415::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28139::DEBUG::2012-12-02 14:29:09,415::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28139::DEBUG::2012-12-02 14:29:09,415::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28139::DEBUG::2012-12-02 14:29:09,415::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28139::DEBUG::2012-12-02 14:29:09,415::task::957::TaskManager.Task::(_decref) Task=`32f94075-8ff6-43bf-afed-ce38494678d7`::ref 0 aborting False >Thread-28142::DEBUG::2012-12-02 14:29:14,317::task::568::TaskManager.Task::(_updateState) Task=`85f4291b-4a84-43df-9b45-7ca2521a94b8`::moving from state init -> state preparing >Thread-28142::INFO::2012-12-02 14:29:14,317::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28142::INFO::2012-12-02 14:29:14,317::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00772786140442', 'lastCheck': 1354451347.9362199, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00441193580627', 'lastCheck': 1354451353.794867, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00170207023621', 'lastCheck': 1354451345.4622929, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00736618041992', 'lastCheck': 1354451353.800616, 'code': 0, 'valid': True}} >Thread-28142::DEBUG::2012-12-02 14:29:14,317::task::1151::TaskManager.Task::(prepare) Task=`85f4291b-4a84-43df-9b45-7ca2521a94b8`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00772786140442', 'lastCheck': 1354451347.9362199, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00441193580627', 'lastCheck': 1354451353.794867, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00170207023621', 'lastCheck': 1354451345.4622929, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00736618041992', 'lastCheck': 1354451353.800616, 'code': 0, 'valid': True}} >Thread-28142::DEBUG::2012-12-02 14:29:14,317::task::568::TaskManager.Task::(_updateState) Task=`85f4291b-4a84-43df-9b45-7ca2521a94b8`::moving from state preparing -> state finished >Thread-28142::DEBUG::2012-12-02 14:29:14,317::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28142::DEBUG::2012-12-02 14:29:14,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28142::DEBUG::2012-12-02 14:29:14,318::task::957::TaskManager.Task::(_decref) Task=`85f4291b-4a84-43df-9b45-7ca2521a94b8`::ref 0 aborting False >Thread-28146::DEBUG::2012-12-02 14:29:19,540::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28146::DEBUG::2012-12-02 14:29:19,540::task::568::TaskManager.Task::(_updateState) Task=`b355d428-2208-4550-8318-f164bc7f600d`::moving from state init -> state preparing >Thread-28146::INFO::2012-12-02 14:29:19,540::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28146::INFO::2012-12-02 14:29:19,541::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28146::DEBUG::2012-12-02 14:29:19,541::task::1151::TaskManager.Task::(prepare) Task=`b355d428-2208-4550-8318-f164bc7f600d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28146::DEBUG::2012-12-02 14:29:19,541::task::568::TaskManager.Task::(_updateState) Task=`b355d428-2208-4550-8318-f164bc7f600d`::moving from state preparing -> state finished >Thread-28146::DEBUG::2012-12-02 14:29:19,541::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28146::DEBUG::2012-12-02 14:29:19,541::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28146::DEBUG::2012-12-02 14:29:19,541::task::957::TaskManager.Task::(_decref) Task=`b355d428-2208-4550-8318-f164bc7f600d`::ref 0 aborting False >Thread-28147::DEBUG::2012-12-02 14:29:19,544::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28147::DEBUG::2012-12-02 14:29:19,544::task::568::TaskManager.Task::(_updateState) Task=`72ff3256-927c-4d8a-b298-7eda232c009f`::moving from state init -> state preparing >Thread-28147::INFO::2012-12-02 14:29:19,545::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28147::DEBUG::2012-12-02 14:29:19,545::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`176d8e7a-1e2b-4ef5-8f73-86574a43ae38`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28147::DEBUG::2012-12-02 14:29:19,545::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28147::DEBUG::2012-12-02 14:29:19,545::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28147::DEBUG::2012-12-02 14:29:19,545::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`176d8e7a-1e2b-4ef5-8f73-86574a43ae38`::Granted request >Thread-28147::DEBUG::2012-12-02 14:29:19,546::task::794::TaskManager.Task::(resourceAcquired) Task=`72ff3256-927c-4d8a-b298-7eda232c009f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28147::DEBUG::2012-12-02 14:29:19,546::task::957::TaskManager.Task::(_decref) Task=`72ff3256-927c-4d8a-b298-7eda232c009f`::ref 1 aborting False >Thread-28147::INFO::2012-12-02 14:29:19,552::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28147::DEBUG::2012-12-02 14:29:19,552::task::1151::TaskManager.Task::(prepare) Task=`72ff3256-927c-4d8a-b298-7eda232c009f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28147::DEBUG::2012-12-02 14:29:19,552::task::568::TaskManager.Task::(_updateState) Task=`72ff3256-927c-4d8a-b298-7eda232c009f`::moving from state preparing -> state finished >Thread-28147::DEBUG::2012-12-02 14:29:19,553::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28147::DEBUG::2012-12-02 14:29:19,553::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28147::DEBUG::2012-12-02 14:29:19,553::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28147::DEBUG::2012-12-02 14:29:19,553::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28147::DEBUG::2012-12-02 14:29:19,553::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28147::DEBUG::2012-12-02 14:29:19,553::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28147::DEBUG::2012-12-02 14:29:19,553::task::957::TaskManager.Task::(_decref) Task=`72ff3256-927c-4d8a-b298-7eda232c009f`::ref 0 aborting False >Thread-28150::DEBUG::2012-12-02 14:29:24,385::task::568::TaskManager.Task::(_updateState) Task=`a858319a-86bf-428e-9784-fdc1548e3f4e`::moving from state init -> state preparing >Thread-28150::INFO::2012-12-02 14:29:24,385::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28150::INFO::2012-12-02 14:29:24,385::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00431990623474', 'lastCheck': 1354451357.94942, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00432920455933', 'lastCheck': 1354451363.813772, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00266480445862', 'lastCheck': 1354451355.4740469, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00865483283997', 'lastCheck': 1354451363.819664, 'code': 0, 'valid': True}} >Thread-28150::DEBUG::2012-12-02 14:29:24,385::task::1151::TaskManager.Task::(prepare) Task=`a858319a-86bf-428e-9784-fdc1548e3f4e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00431990623474', 'lastCheck': 1354451357.94942, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00432920455933', 'lastCheck': 1354451363.813772, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00266480445862', 'lastCheck': 1354451355.4740469, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00865483283997', 'lastCheck': 1354451363.819664, 'code': 0, 'valid': True}} >Thread-28150::DEBUG::2012-12-02 14:29:24,385::task::568::TaskManager.Task::(_updateState) Task=`a858319a-86bf-428e-9784-fdc1548e3f4e`::moving from state preparing -> state finished >Thread-28150::DEBUG::2012-12-02 14:29:24,385::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28150::DEBUG::2012-12-02 14:29:24,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28150::DEBUG::2012-12-02 14:29:24,386::task::957::TaskManager.Task::(_decref) Task=`a858319a-86bf-428e-9784-fdc1548e3f4e`::ref 0 aborting False >Thread-28154::DEBUG::2012-12-02 14:29:29,719::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28154::DEBUG::2012-12-02 14:29:29,720::task::568::TaskManager.Task::(_updateState) Task=`39f433e9-a460-426d-90ff-5c4a18e9c7b9`::moving from state init -> state preparing >Thread-28154::INFO::2012-12-02 14:29:29,720::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28154::INFO::2012-12-02 14:29:29,720::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28154::DEBUG::2012-12-02 14:29:29,720::task::1151::TaskManager.Task::(prepare) Task=`39f433e9-a460-426d-90ff-5c4a18e9c7b9`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28154::DEBUG::2012-12-02 14:29:29,721::task::568::TaskManager.Task::(_updateState) Task=`39f433e9-a460-426d-90ff-5c4a18e9c7b9`::moving from state preparing -> state finished >Thread-28154::DEBUG::2012-12-02 14:29:29,721::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28154::DEBUG::2012-12-02 14:29:29,721::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28154::DEBUG::2012-12-02 14:29:29,721::task::957::TaskManager.Task::(_decref) Task=`39f433e9-a460-426d-90ff-5c4a18e9c7b9`::ref 0 aborting False >Thread-28155::DEBUG::2012-12-02 14:29:29,723::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28155::DEBUG::2012-12-02 14:29:29,724::task::568::TaskManager.Task::(_updateState) Task=`18cc1aab-01e5-4bf9-a1e6-63df8c7ca531`::moving from state init -> state preparing >Thread-28155::INFO::2012-12-02 14:29:29,724::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28155::DEBUG::2012-12-02 14:29:29,724::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ba7123b4-6fa0-48fe-88f7-9bd8691a38a7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28155::DEBUG::2012-12-02 14:29:29,724::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28155::DEBUG::2012-12-02 14:29:29,724::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28155::DEBUG::2012-12-02 14:29:29,725::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ba7123b4-6fa0-48fe-88f7-9bd8691a38a7`::Granted request >Thread-28155::DEBUG::2012-12-02 14:29:29,725::task::794::TaskManager.Task::(resourceAcquired) Task=`18cc1aab-01e5-4bf9-a1e6-63df8c7ca531`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28155::DEBUG::2012-12-02 14:29:29,725::task::957::TaskManager.Task::(_decref) Task=`18cc1aab-01e5-4bf9-a1e6-63df8c7ca531`::ref 1 aborting False >Thread-28155::INFO::2012-12-02 14:29:29,730::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28155::DEBUG::2012-12-02 14:29:29,730::task::1151::TaskManager.Task::(prepare) Task=`18cc1aab-01e5-4bf9-a1e6-63df8c7ca531`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28155::DEBUG::2012-12-02 14:29:29,730::task::568::TaskManager.Task::(_updateState) Task=`18cc1aab-01e5-4bf9-a1e6-63df8c7ca531`::moving from state preparing -> state finished >Thread-28155::DEBUG::2012-12-02 14:29:29,730::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28155::DEBUG::2012-12-02 14:29:29,730::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28155::DEBUG::2012-12-02 14:29:29,730::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28155::DEBUG::2012-12-02 14:29:29,730::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28155::DEBUG::2012-12-02 14:29:29,730::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28155::DEBUG::2012-12-02 14:29:29,730::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28155::DEBUG::2012-12-02 14:29:29,731::task::957::TaskManager.Task::(_decref) Task=`18cc1aab-01e5-4bf9-a1e6-63df8c7ca531`::ref 0 aborting False >Thread-28158::DEBUG::2012-12-02 14:29:34,483::task::568::TaskManager.Task::(_updateState) Task=`0f612d88-f6e2-4015-87f2-7a6c8edcb06a`::moving from state init -> state preparing >Thread-28158::INFO::2012-12-02 14:29:34,483::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28158::INFO::2012-12-02 14:29:34,484::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00802087783813', 'lastCheck': 1354451367.966922, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0065450668335', 'lastCheck': 1354451373.8296199, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171399116516', 'lastCheck': 1354451365.483182, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00750303268433', 'lastCheck': 1354451373.838233, 'code': 0, 'valid': True}} >Thread-28158::DEBUG::2012-12-02 14:29:34,484::task::1151::TaskManager.Task::(prepare) Task=`0f612d88-f6e2-4015-87f2-7a6c8edcb06a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00802087783813', 'lastCheck': 1354451367.966922, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0065450668335', 'lastCheck': 1354451373.8296199, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171399116516', 'lastCheck': 1354451365.483182, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00750303268433', 'lastCheck': 1354451373.838233, 'code': 0, 'valid': True}} >Thread-28158::DEBUG::2012-12-02 14:29:34,484::task::568::TaskManager.Task::(_updateState) Task=`0f612d88-f6e2-4015-87f2-7a6c8edcb06a`::moving from state preparing -> state finished >Thread-28158::DEBUG::2012-12-02 14:29:34,484::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28158::DEBUG::2012-12-02 14:29:34,484::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28158::DEBUG::2012-12-02 14:29:34,484::task::957::TaskManager.Task::(_decref) Task=`0f612d88-f6e2-4015-87f2-7a6c8edcb06a`::ref 0 aborting False >Thread-28162::DEBUG::2012-12-02 14:29:39,884::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28162::DEBUG::2012-12-02 14:29:39,884::task::568::TaskManager.Task::(_updateState) Task=`d1fe9d9f-7a2c-4777-affa-950388eab881`::moving from state init -> state preparing >Thread-28162::INFO::2012-12-02 14:29:39,884::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28162::INFO::2012-12-02 14:29:39,884::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28162::DEBUG::2012-12-02 14:29:39,885::task::1151::TaskManager.Task::(prepare) Task=`d1fe9d9f-7a2c-4777-affa-950388eab881`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28162::DEBUG::2012-12-02 14:29:39,885::task::568::TaskManager.Task::(_updateState) Task=`d1fe9d9f-7a2c-4777-affa-950388eab881`::moving from state preparing -> state finished >Thread-28162::DEBUG::2012-12-02 14:29:39,885::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28162::DEBUG::2012-12-02 14:29:39,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28162::DEBUG::2012-12-02 14:29:39,885::task::957::TaskManager.Task::(_decref) Task=`d1fe9d9f-7a2c-4777-affa-950388eab881`::ref 0 aborting False >Thread-28163::DEBUG::2012-12-02 14:29:39,887::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28163::DEBUG::2012-12-02 14:29:39,888::task::568::TaskManager.Task::(_updateState) Task=`3fba9685-5e29-4820-9671-48adf9777be3`::moving from state init -> state preparing >Thread-28163::INFO::2012-12-02 14:29:39,888::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28163::DEBUG::2012-12-02 14:29:39,888::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d80c95e6-ebb2-4a73-bafd-981e2dc24b05`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28163::DEBUG::2012-12-02 14:29:39,888::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28163::DEBUG::2012-12-02 14:29:39,889::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28163::DEBUG::2012-12-02 14:29:39,889::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d80c95e6-ebb2-4a73-bafd-981e2dc24b05`::Granted request >Thread-28163::DEBUG::2012-12-02 14:29:39,889::task::794::TaskManager.Task::(resourceAcquired) Task=`3fba9685-5e29-4820-9671-48adf9777be3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28163::DEBUG::2012-12-02 14:29:39,889::task::957::TaskManager.Task::(_decref) Task=`3fba9685-5e29-4820-9671-48adf9777be3`::ref 1 aborting False >Thread-28163::INFO::2012-12-02 14:29:39,895::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28163::DEBUG::2012-12-02 14:29:39,895::task::1151::TaskManager.Task::(prepare) Task=`3fba9685-5e29-4820-9671-48adf9777be3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28163::DEBUG::2012-12-02 14:29:39,896::task::568::TaskManager.Task::(_updateState) Task=`3fba9685-5e29-4820-9671-48adf9777be3`::moving from state preparing -> state finished >Thread-28163::DEBUG::2012-12-02 14:29:39,896::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28163::DEBUG::2012-12-02 14:29:39,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28163::DEBUG::2012-12-02 14:29:39,896::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28163::DEBUG::2012-12-02 14:29:39,896::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28163::DEBUG::2012-12-02 14:29:39,896::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28163::DEBUG::2012-12-02 14:29:39,896::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28163::DEBUG::2012-12-02 14:29:39,896::task::957::TaskManager.Task::(_decref) Task=`3fba9685-5e29-4820-9671-48adf9777be3`::ref 0 aborting False >Thread-28166::DEBUG::2012-12-02 14:29:44,672::task::568::TaskManager.Task::(_updateState) Task=`7f4ff5c0-11cf-488d-bfac-761af32d4fc3`::moving from state init -> state preparing >Thread-28166::INFO::2012-12-02 14:29:44,673::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28166::INFO::2012-12-02 14:29:44,673::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00827097892761', 'lastCheck': 1354451377.9868059, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180101394653', 'lastCheck': 1354451383.848294, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287890434265', 'lastCheck': 1354451375.494463, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00932192802429', 'lastCheck': 1354451383.860142, 'code': 0, 'valid': True}} >Thread-28166::DEBUG::2012-12-02 14:29:44,673::task::1151::TaskManager.Task::(prepare) Task=`7f4ff5c0-11cf-488d-bfac-761af32d4fc3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00827097892761', 'lastCheck': 1354451377.9868059, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180101394653', 'lastCheck': 1354451383.848294, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287890434265', 'lastCheck': 1354451375.494463, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00932192802429', 'lastCheck': 1354451383.860142, 'code': 0, 'valid': True}} >Thread-28166::DEBUG::2012-12-02 14:29:44,673::task::568::TaskManager.Task::(_updateState) Task=`7f4ff5c0-11cf-488d-bfac-761af32d4fc3`::moving from state preparing -> state finished >Thread-28166::DEBUG::2012-12-02 14:29:44,673::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28166::DEBUG::2012-12-02 14:29:44,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28166::DEBUG::2012-12-02 14:29:44,673::task::957::TaskManager.Task::(_decref) Task=`7f4ff5c0-11cf-488d-bfac-761af32d4fc3`::ref 0 aborting False >Thread-28170::DEBUG::2012-12-02 14:29:50,037::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28170::DEBUG::2012-12-02 14:29:50,037::task::568::TaskManager.Task::(_updateState) Task=`61201299-7675-47dc-94f7-7ed7416cde0a`::moving from state init -> state preparing >Thread-28170::INFO::2012-12-02 14:29:50,037::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28170::INFO::2012-12-02 14:29:50,038::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28170::DEBUG::2012-12-02 14:29:50,038::task::1151::TaskManager.Task::(prepare) Task=`61201299-7675-47dc-94f7-7ed7416cde0a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28170::DEBUG::2012-12-02 14:29:50,038::task::568::TaskManager.Task::(_updateState) Task=`61201299-7675-47dc-94f7-7ed7416cde0a`::moving from state preparing -> state finished >Thread-28170::DEBUG::2012-12-02 14:29:50,038::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28170::DEBUG::2012-12-02 14:29:50,038::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28170::DEBUG::2012-12-02 14:29:50,038::task::957::TaskManager.Task::(_decref) Task=`61201299-7675-47dc-94f7-7ed7416cde0a`::ref 0 aborting False >Thread-28171::DEBUG::2012-12-02 14:29:50,041::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28171::DEBUG::2012-12-02 14:29:50,041::task::568::TaskManager.Task::(_updateState) Task=`a20881cf-f2c0-49d6-a8b0-dc4b207546ad`::moving from state init -> state preparing >Thread-28171::INFO::2012-12-02 14:29:50,041::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28171::DEBUG::2012-12-02 14:29:50,042::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b062525d-37a5-4171-b1bc-278db7473e26`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28171::DEBUG::2012-12-02 14:29:50,042::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28171::DEBUG::2012-12-02 14:29:50,042::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28171::DEBUG::2012-12-02 14:29:50,042::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b062525d-37a5-4171-b1bc-278db7473e26`::Granted request >Thread-28171::DEBUG::2012-12-02 14:29:50,042::task::794::TaskManager.Task::(resourceAcquired) Task=`a20881cf-f2c0-49d6-a8b0-dc4b207546ad`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28171::DEBUG::2012-12-02 14:29:50,042::task::957::TaskManager.Task::(_decref) Task=`a20881cf-f2c0-49d6-a8b0-dc4b207546ad`::ref 1 aborting False >Thread-28171::INFO::2012-12-02 14:29:50,048::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28171::DEBUG::2012-12-02 14:29:50,048::task::1151::TaskManager.Task::(prepare) Task=`a20881cf-f2c0-49d6-a8b0-dc4b207546ad`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28171::DEBUG::2012-12-02 14:29:50,048::task::568::TaskManager.Task::(_updateState) Task=`a20881cf-f2c0-49d6-a8b0-dc4b207546ad`::moving from state preparing -> state finished >Thread-28171::DEBUG::2012-12-02 14:29:50,048::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28171::DEBUG::2012-12-02 14:29:50,048::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28171::DEBUG::2012-12-02 14:29:50,048::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28171::DEBUG::2012-12-02 14:29:50,048::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28171::DEBUG::2012-12-02 14:29:50,048::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28171::DEBUG::2012-12-02 14:29:50,048::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28171::DEBUG::2012-12-02 14:29:50,049::task::957::TaskManager.Task::(_decref) Task=`a20881cf-f2c0-49d6-a8b0-dc4b207546ad`::ref 0 aborting False >Thread-28174::DEBUG::2012-12-02 14:29:54,750::task::568::TaskManager.Task::(_updateState) Task=`e1e467b2-f31f-4b10-9b83-d4d322c49bea`::moving from state init -> state preparing >Thread-28174::INFO::2012-12-02 14:29:54,750::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28174::INFO::2012-12-02 14:29:54,750::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00781011581421', 'lastCheck': 1354451388.00387, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00264191627502', 'lastCheck': 1354451393.860857, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199389457703', 'lastCheck': 1354451385.5055101, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00769519805908', 'lastCheck': 1354451393.8770919, 'code': 0, 'valid': True}} >Thread-28174::DEBUG::2012-12-02 14:29:54,750::task::1151::TaskManager.Task::(prepare) Task=`e1e467b2-f31f-4b10-9b83-d4d322c49bea`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00781011581421', 'lastCheck': 1354451388.00387, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00264191627502', 'lastCheck': 1354451393.860857, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199389457703', 'lastCheck': 1354451385.5055101, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00769519805908', 'lastCheck': 1354451393.8770919, 'code': 0, 'valid': True}} >Thread-28174::DEBUG::2012-12-02 14:29:54,750::task::568::TaskManager.Task::(_updateState) Task=`e1e467b2-f31f-4b10-9b83-d4d322c49bea`::moving from state preparing -> state finished >Thread-28174::DEBUG::2012-12-02 14:29:54,750::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28174::DEBUG::2012-12-02 14:29:54,751::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28174::DEBUG::2012-12-02 14:29:54,751::task::957::TaskManager.Task::(_decref) Task=`e1e467b2-f31f-4b10-9b83-d4d322c49bea`::ref 0 aborting False >Thread-28178::DEBUG::2012-12-02 14:30:00,184::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28178::DEBUG::2012-12-02 14:30:00,185::task::568::TaskManager.Task::(_updateState) Task=`aa141330-5f0d-4edd-a16b-845b3ec3c003`::moving from state init -> state preparing >Thread-28178::INFO::2012-12-02 14:30:00,185::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28178::INFO::2012-12-02 14:30:00,185::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28178::DEBUG::2012-12-02 14:30:00,185::task::1151::TaskManager.Task::(prepare) Task=`aa141330-5f0d-4edd-a16b-845b3ec3c003`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28178::DEBUG::2012-12-02 14:30:00,185::task::568::TaskManager.Task::(_updateState) Task=`aa141330-5f0d-4edd-a16b-845b3ec3c003`::moving from state preparing -> state finished >Thread-28178::DEBUG::2012-12-02 14:30:00,185::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28178::DEBUG::2012-12-02 14:30:00,185::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28178::DEBUG::2012-12-02 14:30:00,186::task::957::TaskManager.Task::(_decref) Task=`aa141330-5f0d-4edd-a16b-845b3ec3c003`::ref 0 aborting False >Thread-28179::DEBUG::2012-12-02 14:30:00,188::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28179::DEBUG::2012-12-02 14:30:00,188::task::568::TaskManager.Task::(_updateState) Task=`a614a08d-ada0-4500-9d20-66086f0ec312`::moving from state init -> state preparing >Thread-28179::INFO::2012-12-02 14:30:00,188::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28179::DEBUG::2012-12-02 14:30:00,189::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`99e235b2-6bb6-43f4-9be5-1267ee6250e9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28179::DEBUG::2012-12-02 14:30:00,189::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28179::DEBUG::2012-12-02 14:30:00,189::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28179::DEBUG::2012-12-02 14:30:00,189::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`99e235b2-6bb6-43f4-9be5-1267ee6250e9`::Granted request >Thread-28179::DEBUG::2012-12-02 14:30:00,189::task::794::TaskManager.Task::(resourceAcquired) Task=`a614a08d-ada0-4500-9d20-66086f0ec312`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28179::DEBUG::2012-12-02 14:30:00,189::task::957::TaskManager.Task::(_decref) Task=`a614a08d-ada0-4500-9d20-66086f0ec312`::ref 1 aborting False >Thread-28179::INFO::2012-12-02 14:30:00,193::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28179::DEBUG::2012-12-02 14:30:00,194::task::1151::TaskManager.Task::(prepare) Task=`a614a08d-ada0-4500-9d20-66086f0ec312`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28179::DEBUG::2012-12-02 14:30:00,194::task::568::TaskManager.Task::(_updateState) Task=`a614a08d-ada0-4500-9d20-66086f0ec312`::moving from state preparing -> state finished >Thread-28179::DEBUG::2012-12-02 14:30:00,194::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28179::DEBUG::2012-12-02 14:30:00,194::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28179::DEBUG::2012-12-02 14:30:00,194::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28179::DEBUG::2012-12-02 14:30:00,194::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28179::DEBUG::2012-12-02 14:30:00,194::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28179::DEBUG::2012-12-02 14:30:00,194::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28179::DEBUG::2012-12-02 14:30:00,194::task::957::TaskManager.Task::(_decref) Task=`a614a08d-ada0-4500-9d20-66086f0ec312`::ref 0 aborting False >Thread-28182::DEBUG::2012-12-02 14:30:04,846::task::568::TaskManager.Task::(_updateState) Task=`b85debdc-a551-4cf3-90bb-a3f1111398e8`::moving from state init -> state preparing >Thread-28182::INFO::2012-12-02 14:30:04,847::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28182::INFO::2012-12-02 14:30:04,847::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0246109962463', 'lastCheck': 1354451398.0508649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00188207626343', 'lastCheck': 1354451403.873199, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00179100036621', 'lastCheck': 1354451395.515902, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00918912887573', 'lastCheck': 1354451403.8951471, 'code': 0, 'valid': True}} >Thread-28182::DEBUG::2012-12-02 14:30:04,847::task::1151::TaskManager.Task::(prepare) Task=`b85debdc-a551-4cf3-90bb-a3f1111398e8`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0246109962463', 'lastCheck': 1354451398.0508649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00188207626343', 'lastCheck': 1354451403.873199, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00179100036621', 'lastCheck': 1354451395.515902, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00918912887573', 'lastCheck': 1354451403.8951471, 'code': 0, 'valid': True}} >Thread-28182::DEBUG::2012-12-02 14:30:04,847::task::568::TaskManager.Task::(_updateState) Task=`b85debdc-a551-4cf3-90bb-a3f1111398e8`::moving from state preparing -> state finished >Thread-28182::DEBUG::2012-12-02 14:30:04,847::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28182::DEBUG::2012-12-02 14:30:04,847::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28182::DEBUG::2012-12-02 14:30:04,848::task::957::TaskManager.Task::(_decref) Task=`b85debdc-a551-4cf3-90bb-a3f1111398e8`::ref 0 aborting False >Thread-28186::DEBUG::2012-12-02 14:30:10,289::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28186::DEBUG::2012-12-02 14:30:10,290::task::568::TaskManager.Task::(_updateState) Task=`c4e2ed31-cf6e-4f12-b741-b558131c6927`::moving from state init -> state preparing >Thread-28186::INFO::2012-12-02 14:30:10,290::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28186::INFO::2012-12-02 14:30:10,290::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28186::DEBUG::2012-12-02 14:30:10,290::task::1151::TaskManager.Task::(prepare) Task=`c4e2ed31-cf6e-4f12-b741-b558131c6927`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28186::DEBUG::2012-12-02 14:30:10,291::task::568::TaskManager.Task::(_updateState) Task=`c4e2ed31-cf6e-4f12-b741-b558131c6927`::moving from state preparing -> state finished >Thread-28186::DEBUG::2012-12-02 14:30:10,291::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28186::DEBUG::2012-12-02 14:30:10,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28186::DEBUG::2012-12-02 14:30:10,291::task::957::TaskManager.Task::(_decref) Task=`c4e2ed31-cf6e-4f12-b741-b558131c6927`::ref 0 aborting False >Thread-28187::DEBUG::2012-12-02 14:30:10,293::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28187::DEBUG::2012-12-02 14:30:10,294::task::568::TaskManager.Task::(_updateState) Task=`e23f63d8-3337-4492-805a-2817f70cfc23`::moving from state init -> state preparing >Thread-28187::INFO::2012-12-02 14:30:10,294::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28187::DEBUG::2012-12-02 14:30:10,294::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8b4312c9-3b6d-4ace-8bc6-090cbd48d79c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28187::DEBUG::2012-12-02 14:30:10,294::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28187::DEBUG::2012-12-02 14:30:10,295::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28187::DEBUG::2012-12-02 14:30:10,295::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8b4312c9-3b6d-4ace-8bc6-090cbd48d79c`::Granted request >Thread-28187::DEBUG::2012-12-02 14:30:10,295::task::794::TaskManager.Task::(resourceAcquired) Task=`e23f63d8-3337-4492-805a-2817f70cfc23`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28187::DEBUG::2012-12-02 14:30:10,296::task::957::TaskManager.Task::(_decref) Task=`e23f63d8-3337-4492-805a-2817f70cfc23`::ref 1 aborting False >Thread-28187::INFO::2012-12-02 14:30:10,300::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28187::DEBUG::2012-12-02 14:30:10,300::task::1151::TaskManager.Task::(prepare) Task=`e23f63d8-3337-4492-805a-2817f70cfc23`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28187::DEBUG::2012-12-02 14:30:10,300::task::568::TaskManager.Task::(_updateState) Task=`e23f63d8-3337-4492-805a-2817f70cfc23`::moving from state preparing -> state finished >Thread-28187::DEBUG::2012-12-02 14:30:10,300::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28187::DEBUG::2012-12-02 14:30:10,300::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28187::DEBUG::2012-12-02 14:30:10,300::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28187::DEBUG::2012-12-02 14:30:10,300::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28187::DEBUG::2012-12-02 14:30:10,301::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28187::DEBUG::2012-12-02 14:30:10,301::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28187::DEBUG::2012-12-02 14:30:10,301::task::957::TaskManager.Task::(_decref) Task=`e23f63d8-3337-4492-805a-2817f70cfc23`::ref 0 aborting False >Thread-28190::DEBUG::2012-12-02 14:30:14,950::task::568::TaskManager.Task::(_updateState) Task=`73807a4b-17cb-4de8-949d-d5adf9e6554c`::moving from state init -> state preparing >Thread-28190::INFO::2012-12-02 14:30:14,950::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28190::INFO::2012-12-02 14:30:14,950::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00752902030945', 'lastCheck': 1354451408.0666211, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00402092933655', 'lastCheck': 1354451413.8873811, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022509098053', 'lastCheck': 1354451405.52794, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00719404220581', 'lastCheck': 1354451413.911063, 'code': 0, 'valid': True}} >Thread-28190::DEBUG::2012-12-02 14:30:14,950::task::1151::TaskManager.Task::(prepare) Task=`73807a4b-17cb-4de8-949d-d5adf9e6554c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00752902030945', 'lastCheck': 1354451408.0666211, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00402092933655', 'lastCheck': 1354451413.8873811, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022509098053', 'lastCheck': 1354451405.52794, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00719404220581', 'lastCheck': 1354451413.911063, 'code': 0, 'valid': True}} >Thread-28190::DEBUG::2012-12-02 14:30:14,951::task::568::TaskManager.Task::(_updateState) Task=`73807a4b-17cb-4de8-949d-d5adf9e6554c`::moving from state preparing -> state finished >Thread-28190::DEBUG::2012-12-02 14:30:14,951::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28190::DEBUG::2012-12-02 14:30:14,951::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28190::DEBUG::2012-12-02 14:30:14,951::task::957::TaskManager.Task::(_decref) Task=`73807a4b-17cb-4de8-949d-d5adf9e6554c`::ref 0 aborting False >Thread-28194::DEBUG::2012-12-02 14:30:20,426::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28194::DEBUG::2012-12-02 14:30:20,426::task::568::TaskManager.Task::(_updateState) Task=`53b002c5-3fb3-405f-9380-6a859ffd62e4`::moving from state init -> state preparing >Thread-28194::INFO::2012-12-02 14:30:20,426::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28194::INFO::2012-12-02 14:30:20,427::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28194::DEBUG::2012-12-02 14:30:20,427::task::1151::TaskManager.Task::(prepare) Task=`53b002c5-3fb3-405f-9380-6a859ffd62e4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28194::DEBUG::2012-12-02 14:30:20,427::task::568::TaskManager.Task::(_updateState) Task=`53b002c5-3fb3-405f-9380-6a859ffd62e4`::moving from state preparing -> state finished >Thread-28194::DEBUG::2012-12-02 14:30:20,427::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28194::DEBUG::2012-12-02 14:30:20,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28194::DEBUG::2012-12-02 14:30:20,427::task::957::TaskManager.Task::(_decref) Task=`53b002c5-3fb3-405f-9380-6a859ffd62e4`::ref 0 aborting False >Thread-28195::DEBUG::2012-12-02 14:30:20,430::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28195::DEBUG::2012-12-02 14:30:20,430::task::568::TaskManager.Task::(_updateState) Task=`836d5d14-ca92-4950-9796-8d0d83a8629a`::moving from state init -> state preparing >Thread-28195::INFO::2012-12-02 14:30:20,430::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28195::DEBUG::2012-12-02 14:30:20,431::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e774d6d8-17ed-4b52-8cb5-8e6569a42684`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28195::DEBUG::2012-12-02 14:30:20,431::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28195::DEBUG::2012-12-02 14:30:20,431::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28195::DEBUG::2012-12-02 14:30:20,431::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e774d6d8-17ed-4b52-8cb5-8e6569a42684`::Granted request >Thread-28195::DEBUG::2012-12-02 14:30:20,431::task::794::TaskManager.Task::(resourceAcquired) Task=`836d5d14-ca92-4950-9796-8d0d83a8629a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28195::DEBUG::2012-12-02 14:30:20,431::task::957::TaskManager.Task::(_decref) Task=`836d5d14-ca92-4950-9796-8d0d83a8629a`::ref 1 aborting False >Thread-28195::INFO::2012-12-02 14:30:20,437::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28195::DEBUG::2012-12-02 14:30:20,437::task::1151::TaskManager.Task::(prepare) Task=`836d5d14-ca92-4950-9796-8d0d83a8629a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28195::DEBUG::2012-12-02 14:30:20,437::task::568::TaskManager.Task::(_updateState) Task=`836d5d14-ca92-4950-9796-8d0d83a8629a`::moving from state preparing -> state finished >Thread-28195::DEBUG::2012-12-02 14:30:20,437::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28195::DEBUG::2012-12-02 14:30:20,437::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28195::DEBUG::2012-12-02 14:30:20,437::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28195::DEBUG::2012-12-02 14:30:20,438::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28195::DEBUG::2012-12-02 14:30:20,438::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28195::DEBUG::2012-12-02 14:30:20,438::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28195::DEBUG::2012-12-02 14:30:20,438::task::957::TaskManager.Task::(_decref) Task=`836d5d14-ca92-4950-9796-8d0d83a8629a`::ref 0 aborting False >Thread-28198::DEBUG::2012-12-02 14:30:25,088::task::568::TaskManager.Task::(_updateState) Task=`7bb96e48-d1f0-4742-b2c2-b771fb7219ff`::moving from state init -> state preparing >Thread-28198::INFO::2012-12-02 14:30:25,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28198::INFO::2012-12-02 14:30:25,089::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0103669166565', 'lastCheck': 1354451418.0876889, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00256681442261', 'lastCheck': 1354451423.9021521, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253391265869', 'lastCheck': 1354451415.5405481, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00688004493713', 'lastCheck': 1354451423.926244, 'code': 0, 'valid': True}} >Thread-28198::DEBUG::2012-12-02 14:30:25,089::task::1151::TaskManager.Task::(prepare) Task=`7bb96e48-d1f0-4742-b2c2-b771fb7219ff`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0103669166565', 'lastCheck': 1354451418.0876889, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00256681442261', 'lastCheck': 1354451423.9021521, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253391265869', 'lastCheck': 1354451415.5405481, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00688004493713', 'lastCheck': 1354451423.926244, 'code': 0, 'valid': True}} >Thread-28198::DEBUG::2012-12-02 14:30:25,089::task::568::TaskManager.Task::(_updateState) Task=`7bb96e48-d1f0-4742-b2c2-b771fb7219ff`::moving from state preparing -> state finished >Thread-28198::DEBUG::2012-12-02 14:30:25,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28198::DEBUG::2012-12-02 14:30:25,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28198::DEBUG::2012-12-02 14:30:25,089::task::957::TaskManager.Task::(_decref) Task=`7bb96e48-d1f0-4742-b2c2-b771fb7219ff`::ref 0 aborting False >Thread-28202::DEBUG::2012-12-02 14:30:30,582::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28202::DEBUG::2012-12-02 14:30:30,582::task::568::TaskManager.Task::(_updateState) Task=`3ef53c7f-dd95-44b8-8e33-757c50d3bbf6`::moving from state init -> state preparing >Thread-28202::INFO::2012-12-02 14:30:30,582::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28202::INFO::2012-12-02 14:30:30,582::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28202::DEBUG::2012-12-02 14:30:30,583::task::1151::TaskManager.Task::(prepare) Task=`3ef53c7f-dd95-44b8-8e33-757c50d3bbf6`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28202::DEBUG::2012-12-02 14:30:30,583::task::568::TaskManager.Task::(_updateState) Task=`3ef53c7f-dd95-44b8-8e33-757c50d3bbf6`::moving from state preparing -> state finished >Thread-28202::DEBUG::2012-12-02 14:30:30,583::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28202::DEBUG::2012-12-02 14:30:30,583::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28202::DEBUG::2012-12-02 14:30:30,583::task::957::TaskManager.Task::(_decref) Task=`3ef53c7f-dd95-44b8-8e33-757c50d3bbf6`::ref 0 aborting False >Thread-28203::DEBUG::2012-12-02 14:30:30,586::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28203::DEBUG::2012-12-02 14:30:30,586::task::568::TaskManager.Task::(_updateState) Task=`fac699fe-b577-43d2-8800-787b3c9004c9`::moving from state init -> state preparing >Thread-28203::INFO::2012-12-02 14:30:30,586::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28203::DEBUG::2012-12-02 14:30:30,586::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7506e7c2-abae-4271-8992-4216b59c9ae3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28203::DEBUG::2012-12-02 14:30:30,586::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28203::DEBUG::2012-12-02 14:30:30,587::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28203::DEBUG::2012-12-02 14:30:30,587::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7506e7c2-abae-4271-8992-4216b59c9ae3`::Granted request >Thread-28203::DEBUG::2012-12-02 14:30:30,587::task::794::TaskManager.Task::(resourceAcquired) Task=`fac699fe-b577-43d2-8800-787b3c9004c9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28203::DEBUG::2012-12-02 14:30:30,587::task::957::TaskManager.Task::(_decref) Task=`fac699fe-b577-43d2-8800-787b3c9004c9`::ref 1 aborting False >Thread-28203::INFO::2012-12-02 14:30:30,594::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28203::DEBUG::2012-12-02 14:30:30,594::task::1151::TaskManager.Task::(prepare) Task=`fac699fe-b577-43d2-8800-787b3c9004c9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28203::DEBUG::2012-12-02 14:30:30,594::task::568::TaskManager.Task::(_updateState) Task=`fac699fe-b577-43d2-8800-787b3c9004c9`::moving from state preparing -> state finished >Thread-28203::DEBUG::2012-12-02 14:30:30,594::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28203::DEBUG::2012-12-02 14:30:30,594::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28203::DEBUG::2012-12-02 14:30:30,594::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28203::DEBUG::2012-12-02 14:30:30,595::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28203::DEBUG::2012-12-02 14:30:30,595::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28203::DEBUG::2012-12-02 14:30:30,595::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28203::DEBUG::2012-12-02 14:30:30,595::task::957::TaskManager.Task::(_decref) Task=`fac699fe-b577-43d2-8800-787b3c9004c9`::ref 0 aborting False >Thread-28206::DEBUG::2012-12-02 14:30:35,221::task::568::TaskManager.Task::(_updateState) Task=`ea0eed9b-4df3-467f-b2e6-8087768138cb`::moving from state init -> state preparing >Thread-28206::INFO::2012-12-02 14:30:35,221::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28206::INFO::2012-12-02 14:30:35,222::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00493907928467', 'lastCheck': 1354451428.101403, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00327301025391', 'lastCheck': 1354451433.91506, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00190615653992', 'lastCheck': 1354451425.5513611, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0073709487915', 'lastCheck': 1354451433.9425409, 'code': 0, 'valid': True}} >Thread-28206::DEBUG::2012-12-02 14:30:35,222::task::1151::TaskManager.Task::(prepare) Task=`ea0eed9b-4df3-467f-b2e6-8087768138cb`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00493907928467', 'lastCheck': 1354451428.101403, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00327301025391', 'lastCheck': 1354451433.91506, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00190615653992', 'lastCheck': 1354451425.5513611, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0073709487915', 'lastCheck': 1354451433.9425409, 'code': 0, 'valid': True}} >Thread-28206::DEBUG::2012-12-02 14:30:35,222::task::568::TaskManager.Task::(_updateState) Task=`ea0eed9b-4df3-467f-b2e6-8087768138cb`::moving from state preparing -> state finished >Thread-28206::DEBUG::2012-12-02 14:30:35,222::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28206::DEBUG::2012-12-02 14:30:35,222::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28206::DEBUG::2012-12-02 14:30:35,222::task::957::TaskManager.Task::(_decref) Task=`ea0eed9b-4df3-467f-b2e6-8087768138cb`::ref 0 aborting False >Thread-28210::DEBUG::2012-12-02 14:30:40,744::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28210::DEBUG::2012-12-02 14:30:40,744::task::568::TaskManager.Task::(_updateState) Task=`44868e00-0493-4ed5-a004-fd3c99615a06`::moving from state init -> state preparing >Thread-28210::INFO::2012-12-02 14:30:40,744::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28210::INFO::2012-12-02 14:30:40,744::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28210::DEBUG::2012-12-02 14:30:40,744::task::1151::TaskManager.Task::(prepare) Task=`44868e00-0493-4ed5-a004-fd3c99615a06`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28210::DEBUG::2012-12-02 14:30:40,745::task::568::TaskManager.Task::(_updateState) Task=`44868e00-0493-4ed5-a004-fd3c99615a06`::moving from state preparing -> state finished >Thread-28210::DEBUG::2012-12-02 14:30:40,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28210::DEBUG::2012-12-02 14:30:40,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28210::DEBUG::2012-12-02 14:30:40,745::task::957::TaskManager.Task::(_decref) Task=`44868e00-0493-4ed5-a004-fd3c99615a06`::ref 0 aborting False >Thread-28211::DEBUG::2012-12-02 14:30:40,747::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28211::DEBUG::2012-12-02 14:30:40,748::task::568::TaskManager.Task::(_updateState) Task=`3e00b871-acb7-4202-9653-fa3376e8ce7f`::moving from state init -> state preparing >Thread-28211::INFO::2012-12-02 14:30:40,748::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28211::DEBUG::2012-12-02 14:30:40,748::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`081fe854-9cbc-4312-a43a-b5b8214bb36c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28211::DEBUG::2012-12-02 14:30:40,748::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28211::DEBUG::2012-12-02 14:30:40,748::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28211::DEBUG::2012-12-02 14:30:40,749::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`081fe854-9cbc-4312-a43a-b5b8214bb36c`::Granted request >Thread-28211::DEBUG::2012-12-02 14:30:40,749::task::794::TaskManager.Task::(resourceAcquired) Task=`3e00b871-acb7-4202-9653-fa3376e8ce7f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28211::DEBUG::2012-12-02 14:30:40,749::task::957::TaskManager.Task::(_decref) Task=`3e00b871-acb7-4202-9653-fa3376e8ce7f`::ref 1 aborting False >Thread-28211::INFO::2012-12-02 14:30:40,755::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28211::DEBUG::2012-12-02 14:30:40,755::task::1151::TaskManager.Task::(prepare) Task=`3e00b871-acb7-4202-9653-fa3376e8ce7f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28211::DEBUG::2012-12-02 14:30:40,755::task::568::TaskManager.Task::(_updateState) Task=`3e00b871-acb7-4202-9653-fa3376e8ce7f`::moving from state preparing -> state finished >Thread-28211::DEBUG::2012-12-02 14:30:40,756::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28211::DEBUG::2012-12-02 14:30:40,756::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28211::DEBUG::2012-12-02 14:30:40,756::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28211::DEBUG::2012-12-02 14:30:40,756::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28211::DEBUG::2012-12-02 14:30:40,756::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28211::DEBUG::2012-12-02 14:30:40,756::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28211::DEBUG::2012-12-02 14:30:40,756::task::957::TaskManager.Task::(_decref) Task=`3e00b871-acb7-4202-9653-fa3376e8ce7f`::ref 0 aborting False >Thread-28214::DEBUG::2012-12-02 14:30:45,331::task::568::TaskManager.Task::(_updateState) Task=`de8deed5-b7e8-4952-9ad7-cf2642128195`::moving from state init -> state preparing >Thread-28214::INFO::2012-12-02 14:30:45,331::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28214::INFO::2012-12-02 14:30:45,331::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0039689540863', 'lastCheck': 1354451438.114181, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00197601318359', 'lastCheck': 1354451443.9304421, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0028440952301', 'lastCheck': 1354451435.563395, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00666403770447', 'lastCheck': 1354451443.959029, 'code': 0, 'valid': True}} >Thread-28214::DEBUG::2012-12-02 14:30:45,331::task::1151::TaskManager.Task::(prepare) Task=`de8deed5-b7e8-4952-9ad7-cf2642128195`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0039689540863', 'lastCheck': 1354451438.114181, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00197601318359', 'lastCheck': 1354451443.9304421, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0028440952301', 'lastCheck': 1354451435.563395, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00666403770447', 'lastCheck': 1354451443.959029, 'code': 0, 'valid': True}} >Thread-28214::DEBUG::2012-12-02 14:30:45,331::task::568::TaskManager.Task::(_updateState) Task=`de8deed5-b7e8-4952-9ad7-cf2642128195`::moving from state preparing -> state finished >Thread-28214::DEBUG::2012-12-02 14:30:45,331::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28214::DEBUG::2012-12-02 14:30:45,332::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28214::DEBUG::2012-12-02 14:30:45,332::task::957::TaskManager.Task::(_decref) Task=`de8deed5-b7e8-4952-9ad7-cf2642128195`::ref 0 aborting False >Thread-28218::DEBUG::2012-12-02 14:30:50,885::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28218::DEBUG::2012-12-02 14:30:50,885::task::568::TaskManager.Task::(_updateState) Task=`0de4120f-8ed0-466b-bf3d-fdb92db2f126`::moving from state init -> state preparing >Thread-28218::INFO::2012-12-02 14:30:50,885::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28218::INFO::2012-12-02 14:30:50,885::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28218::DEBUG::2012-12-02 14:30:50,886::task::1151::TaskManager.Task::(prepare) Task=`0de4120f-8ed0-466b-bf3d-fdb92db2f126`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28218::DEBUG::2012-12-02 14:30:50,886::task::568::TaskManager.Task::(_updateState) Task=`0de4120f-8ed0-466b-bf3d-fdb92db2f126`::moving from state preparing -> state finished >Thread-28218::DEBUG::2012-12-02 14:30:50,886::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28218::DEBUG::2012-12-02 14:30:50,886::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28218::DEBUG::2012-12-02 14:30:50,886::task::957::TaskManager.Task::(_decref) Task=`0de4120f-8ed0-466b-bf3d-fdb92db2f126`::ref 0 aborting False >Thread-28219::DEBUG::2012-12-02 14:30:50,889::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28219::DEBUG::2012-12-02 14:30:50,889::task::568::TaskManager.Task::(_updateState) Task=`b3c1698d-2e49-45f6-ad85-69c0c7d187c4`::moving from state init -> state preparing >Thread-28219::INFO::2012-12-02 14:30:50,889::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28219::DEBUG::2012-12-02 14:30:50,889::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b747a75d-fa5a-4449-8106-b2a710efb139`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28219::DEBUG::2012-12-02 14:30:50,889::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28219::DEBUG::2012-12-02 14:30:50,890::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28219::DEBUG::2012-12-02 14:30:50,890::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b747a75d-fa5a-4449-8106-b2a710efb139`::Granted request >Thread-28219::DEBUG::2012-12-02 14:30:50,890::task::794::TaskManager.Task::(resourceAcquired) Task=`b3c1698d-2e49-45f6-ad85-69c0c7d187c4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28219::DEBUG::2012-12-02 14:30:50,890::task::957::TaskManager.Task::(_decref) Task=`b3c1698d-2e49-45f6-ad85-69c0c7d187c4`::ref 1 aborting False >Thread-28219::INFO::2012-12-02 14:30:50,896::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28219::DEBUG::2012-12-02 14:30:50,896::task::1151::TaskManager.Task::(prepare) Task=`b3c1698d-2e49-45f6-ad85-69c0c7d187c4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28219::DEBUG::2012-12-02 14:30:50,896::task::568::TaskManager.Task::(_updateState) Task=`b3c1698d-2e49-45f6-ad85-69c0c7d187c4`::moving from state preparing -> state finished >Thread-28219::DEBUG::2012-12-02 14:30:50,896::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28219::DEBUG::2012-12-02 14:30:50,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28219::DEBUG::2012-12-02 14:30:50,897::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28219::DEBUG::2012-12-02 14:30:50,897::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28219::DEBUG::2012-12-02 14:30:50,897::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28219::DEBUG::2012-12-02 14:30:50,897::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28219::DEBUG::2012-12-02 14:30:50,897::task::957::TaskManager.Task::(_decref) Task=`b3c1698d-2e49-45f6-ad85-69c0c7d187c4`::ref 0 aborting False >Thread-28222::DEBUG::2012-12-02 14:30:55,438::task::568::TaskManager.Task::(_updateState) Task=`73b0130f-ccfe-4fc1-9e35-6ef9acea9daf`::moving from state init -> state preparing >Thread-28222::INFO::2012-12-02 14:30:55,438::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28222::INFO::2012-12-02 14:30:55,438::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0090229511261', 'lastCheck': 1354451448.1315429, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198197364807', 'lastCheck': 1354451453.941885, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018310546875', 'lastCheck': 1354451445.5740919, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842213630676', 'lastCheck': 1354451453.9764619, 'code': 0, 'valid': True}} >Thread-28222::DEBUG::2012-12-02 14:30:55,438::task::1151::TaskManager.Task::(prepare) Task=`73b0130f-ccfe-4fc1-9e35-6ef9acea9daf`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0090229511261', 'lastCheck': 1354451448.1315429, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198197364807', 'lastCheck': 1354451453.941885, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0018310546875', 'lastCheck': 1354451445.5740919, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842213630676', 'lastCheck': 1354451453.9764619, 'code': 0, 'valid': True}} >Thread-28222::DEBUG::2012-12-02 14:30:55,438::task::568::TaskManager.Task::(_updateState) Task=`73b0130f-ccfe-4fc1-9e35-6ef9acea9daf`::moving from state preparing -> state finished >Thread-28222::DEBUG::2012-12-02 14:30:55,439::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28222::DEBUG::2012-12-02 14:30:55,439::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28222::DEBUG::2012-12-02 14:30:55,439::task::957::TaskManager.Task::(_decref) Task=`73b0130f-ccfe-4fc1-9e35-6ef9acea9daf`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:30:55,574::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:30:55,574::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:30:55,574::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:30:55,782::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:30:55,784::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:30:55,784::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:30:55,820::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:30:55,820::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:30:55,822::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:30:55,825::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:30:55,825::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:30:55,825::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-11821::DEBUG::2012-12-02 14:30:58,131::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:30:58,132::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:30:58,132::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:30:58,338::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:30:58,339::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:30:58,340::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:30:58,375::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:30:58,375::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:30:58,385::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:30:58,388::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:30:58,388::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:30:58,388::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-28348::DEBUG::2012-12-02 14:31:01,011::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28348::DEBUG::2012-12-02 14:31:01,011::task::568::TaskManager.Task::(_updateState) Task=`f7afaddd-6fcf-4d8a-84f5-c896e3ab6e8d`::moving from state init -> state preparing >Thread-28348::INFO::2012-12-02 14:31:01,011::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28348::INFO::2012-12-02 14:31:01,012::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28348::DEBUG::2012-12-02 14:31:01,012::task::1151::TaskManager.Task::(prepare) Task=`f7afaddd-6fcf-4d8a-84f5-c896e3ab6e8d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28348::DEBUG::2012-12-02 14:31:01,012::task::568::TaskManager.Task::(_updateState) Task=`f7afaddd-6fcf-4d8a-84f5-c896e3ab6e8d`::moving from state preparing -> state finished >Thread-28348::DEBUG::2012-12-02 14:31:01,012::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28348::DEBUG::2012-12-02 14:31:01,012::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28348::DEBUG::2012-12-02 14:31:01,012::task::957::TaskManager.Task::(_decref) Task=`f7afaddd-6fcf-4d8a-84f5-c896e3ab6e8d`::ref 0 aborting False >Thread-28349::DEBUG::2012-12-02 14:31:01,015::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28349::DEBUG::2012-12-02 14:31:01,015::task::568::TaskManager.Task::(_updateState) Task=`7b520397-60ec-47e4-9e84-2cad5b98ae6a`::moving from state init -> state preparing >Thread-28349::INFO::2012-12-02 14:31:01,015::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28349::DEBUG::2012-12-02 14:31:01,016::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7f92e1b0-17b5-4be9-95f5-c4b631f08d24`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28349::DEBUG::2012-12-02 14:31:01,016::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28349::DEBUG::2012-12-02 14:31:01,016::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28349::DEBUG::2012-12-02 14:31:01,016::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7f92e1b0-17b5-4be9-95f5-c4b631f08d24`::Granted request >Thread-28349::DEBUG::2012-12-02 14:31:01,016::task::794::TaskManager.Task::(resourceAcquired) Task=`7b520397-60ec-47e4-9e84-2cad5b98ae6a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28349::DEBUG::2012-12-02 14:31:01,017::task::957::TaskManager.Task::(_decref) Task=`7b520397-60ec-47e4-9e84-2cad5b98ae6a`::ref 1 aborting False >Thread-28349::INFO::2012-12-02 14:31:01,023::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28349::DEBUG::2012-12-02 14:31:01,023::task::1151::TaskManager.Task::(prepare) Task=`7b520397-60ec-47e4-9e84-2cad5b98ae6a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28349::DEBUG::2012-12-02 14:31:01,023::task::568::TaskManager.Task::(_updateState) Task=`7b520397-60ec-47e4-9e84-2cad5b98ae6a`::moving from state preparing -> state finished >Thread-28349::DEBUG::2012-12-02 14:31:01,023::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28349::DEBUG::2012-12-02 14:31:01,023::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28349::DEBUG::2012-12-02 14:31:01,024::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28349::DEBUG::2012-12-02 14:31:01,024::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28349::DEBUG::2012-12-02 14:31:01,024::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28349::DEBUG::2012-12-02 14:31:01,024::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28349::DEBUG::2012-12-02 14:31:01,024::task::957::TaskManager.Task::(_decref) Task=`7b520397-60ec-47e4-9e84-2cad5b98ae6a`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:31:03,942::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:31:03,942::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:31:03,942::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:31:04,150::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:31:04,151::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:31:04,151::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:31:04,186::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:31:04,186::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:31:04,188::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:31:04,190::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:31:04,190::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:31:04,190::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:31:04,191::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:31:04,191::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:31:04,193::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:31:04,399::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:31:04,400::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:31:04,401::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:31:04,435::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:31:04,435::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:31:04,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:31:04,447::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:31:04,447::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:31:04,447::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-28474::DEBUG::2012-12-02 14:31:05,666::task::568::TaskManager.Task::(_updateState) Task=`2e022266-d17d-486f-ab78-473b824d6b96`::moving from state init -> state preparing >Thread-28474::INFO::2012-12-02 14:31:05,666::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28474::INFO::2012-12-02 14:31:05,666::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.006422996521', 'lastCheck': 1354451458.4041569, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00277209281921', 'lastCheck': 1354451464.4573121, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0015070438385', 'lastCheck': 1354451455.8357601, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0106470584869', 'lastCheck': 1354451464.4710441, 'code': 0, 'valid': True}} >Thread-28474::DEBUG::2012-12-02 14:31:05,667::task::1151::TaskManager.Task::(prepare) Task=`2e022266-d17d-486f-ab78-473b824d6b96`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.006422996521', 'lastCheck': 1354451458.4041569, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00277209281921', 'lastCheck': 1354451464.4573121, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0015070438385', 'lastCheck': 1354451455.8357601, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0106470584869', 'lastCheck': 1354451464.4710441, 'code': 0, 'valid': True}} >Thread-28474::DEBUG::2012-12-02 14:31:05,667::task::568::TaskManager.Task::(_updateState) Task=`2e022266-d17d-486f-ab78-473b824d6b96`::moving from state preparing -> state finished >Thread-28474::DEBUG::2012-12-02 14:31:05,667::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28474::DEBUG::2012-12-02 14:31:05,667::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28474::DEBUG::2012-12-02 14:31:05,667::task::957::TaskManager.Task::(_decref) Task=`2e022266-d17d-486f-ab78-473b824d6b96`::ref 0 aborting False >Thread-28478::DEBUG::2012-12-02 14:31:11,200::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28478::DEBUG::2012-12-02 14:31:11,200::task::568::TaskManager.Task::(_updateState) Task=`2b78d2e3-e2a7-4690-bbd8-17811d2d5866`::moving from state init -> state preparing >Thread-28478::INFO::2012-12-02 14:31:11,201::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28478::INFO::2012-12-02 14:31:11,201::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28478::DEBUG::2012-12-02 14:31:11,201::task::1151::TaskManager.Task::(prepare) Task=`2b78d2e3-e2a7-4690-bbd8-17811d2d5866`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28478::DEBUG::2012-12-02 14:31:11,201::task::568::TaskManager.Task::(_updateState) Task=`2b78d2e3-e2a7-4690-bbd8-17811d2d5866`::moving from state preparing -> state finished >Thread-28478::DEBUG::2012-12-02 14:31:11,201::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28478::DEBUG::2012-12-02 14:31:11,201::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28478::DEBUG::2012-12-02 14:31:11,201::task::957::TaskManager.Task::(_decref) Task=`2b78d2e3-e2a7-4690-bbd8-17811d2d5866`::ref 0 aborting False >Thread-28479::DEBUG::2012-12-02 14:31:11,204::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28479::DEBUG::2012-12-02 14:31:11,204::task::568::TaskManager.Task::(_updateState) Task=`a4c22398-c3cd-4d13-bf7c-198e624f94a9`::moving from state init -> state preparing >Thread-28479::INFO::2012-12-02 14:31:11,204::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28479::DEBUG::2012-12-02 14:31:11,205::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bdfb4d9a-c7ee-48ee-924f-b1cb19514205`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28479::DEBUG::2012-12-02 14:31:11,205::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28479::DEBUG::2012-12-02 14:31:11,205::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28479::DEBUG::2012-12-02 14:31:11,205::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bdfb4d9a-c7ee-48ee-924f-b1cb19514205`::Granted request >Thread-28479::DEBUG::2012-12-02 14:31:11,205::task::794::TaskManager.Task::(resourceAcquired) Task=`a4c22398-c3cd-4d13-bf7c-198e624f94a9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28479::DEBUG::2012-12-02 14:31:11,206::task::957::TaskManager.Task::(_decref) Task=`a4c22398-c3cd-4d13-bf7c-198e624f94a9`::ref 1 aborting False >Thread-28479::INFO::2012-12-02 14:31:11,210::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28479::DEBUG::2012-12-02 14:31:11,210::task::1151::TaskManager.Task::(prepare) Task=`a4c22398-c3cd-4d13-bf7c-198e624f94a9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28479::DEBUG::2012-12-02 14:31:11,210::task::568::TaskManager.Task::(_updateState) Task=`a4c22398-c3cd-4d13-bf7c-198e624f94a9`::moving from state preparing -> state finished >Thread-28479::DEBUG::2012-12-02 14:31:11,210::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28479::DEBUG::2012-12-02 14:31:11,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28479::DEBUG::2012-12-02 14:31:11,210::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28479::DEBUG::2012-12-02 14:31:11,210::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28479::DEBUG::2012-12-02 14:31:11,211::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28479::DEBUG::2012-12-02 14:31:11,211::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28479::DEBUG::2012-12-02 14:31:11,211::task::957::TaskManager.Task::(_decref) Task=`a4c22398-c3cd-4d13-bf7c-198e624f94a9`::ref 0 aborting False >Thread-28482::DEBUG::2012-12-02 14:31:15,779::task::568::TaskManager.Task::(_updateState) Task=`f353a13b-c251-4a9d-985d-bb7df849622d`::moving from state init -> state preparing >Thread-28482::INFO::2012-12-02 14:31:15,779::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28482::INFO::2012-12-02 14:31:15,779::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00834393501282', 'lastCheck': 1354451468.422168, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00358486175537', 'lastCheck': 1354451474.4789209, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00186014175415', 'lastCheck': 1354451465.84635, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00745105743408', 'lastCheck': 1354451474.488951, 'code': 0, 'valid': True}} >Thread-28482::DEBUG::2012-12-02 14:31:15,780::task::1151::TaskManager.Task::(prepare) Task=`f353a13b-c251-4a9d-985d-bb7df849622d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00834393501282', 'lastCheck': 1354451468.422168, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00358486175537', 'lastCheck': 1354451474.4789209, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00186014175415', 'lastCheck': 1354451465.84635, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00745105743408', 'lastCheck': 1354451474.488951, 'code': 0, 'valid': True}} >Thread-28482::DEBUG::2012-12-02 14:31:15,780::task::568::TaskManager.Task::(_updateState) Task=`f353a13b-c251-4a9d-985d-bb7df849622d`::moving from state preparing -> state finished >Thread-28482::DEBUG::2012-12-02 14:31:15,780::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28482::DEBUG::2012-12-02 14:31:15,780::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28482::DEBUG::2012-12-02 14:31:15,780::task::957::TaskManager.Task::(_decref) Task=`f353a13b-c251-4a9d-985d-bb7df849622d`::ref 0 aborting False >Thread-28486::DEBUG::2012-12-02 14:31:21,331::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28486::DEBUG::2012-12-02 14:31:21,331::task::568::TaskManager.Task::(_updateState) Task=`0ce74493-95c7-4499-aae2-9c816e9181ef`::moving from state init -> state preparing >Thread-28486::INFO::2012-12-02 14:31:21,331::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28486::INFO::2012-12-02 14:31:21,331::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28486::DEBUG::2012-12-02 14:31:21,332::task::1151::TaskManager.Task::(prepare) Task=`0ce74493-95c7-4499-aae2-9c816e9181ef`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28486::DEBUG::2012-12-02 14:31:21,332::task::568::TaskManager.Task::(_updateState) Task=`0ce74493-95c7-4499-aae2-9c816e9181ef`::moving from state preparing -> state finished >Thread-28486::DEBUG::2012-12-02 14:31:21,332::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28486::DEBUG::2012-12-02 14:31:21,332::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28486::DEBUG::2012-12-02 14:31:21,332::task::957::TaskManager.Task::(_decref) Task=`0ce74493-95c7-4499-aae2-9c816e9181ef`::ref 0 aborting False >Thread-28487::DEBUG::2012-12-02 14:31:21,335::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28487::DEBUG::2012-12-02 14:31:21,335::task::568::TaskManager.Task::(_updateState) Task=`c23eaef5-0286-4389-b15a-edf62fe4f206`::moving from state init -> state preparing >Thread-28487::INFO::2012-12-02 14:31:21,335::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28487::DEBUG::2012-12-02 14:31:21,335::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`14d610a5-9d22-47e3-a1f6-68ad42692a5c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28487::DEBUG::2012-12-02 14:31:21,335::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28487::DEBUG::2012-12-02 14:31:21,336::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28487::DEBUG::2012-12-02 14:31:21,336::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`14d610a5-9d22-47e3-a1f6-68ad42692a5c`::Granted request >Thread-28487::DEBUG::2012-12-02 14:31:21,336::task::794::TaskManager.Task::(resourceAcquired) Task=`c23eaef5-0286-4389-b15a-edf62fe4f206`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28487::DEBUG::2012-12-02 14:31:21,336::task::957::TaskManager.Task::(_decref) Task=`c23eaef5-0286-4389-b15a-edf62fe4f206`::ref 1 aborting False >Thread-28487::INFO::2012-12-02 14:31:21,342::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28487::DEBUG::2012-12-02 14:31:21,343::task::1151::TaskManager.Task::(prepare) Task=`c23eaef5-0286-4389-b15a-edf62fe4f206`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28487::DEBUG::2012-12-02 14:31:21,343::task::568::TaskManager.Task::(_updateState) Task=`c23eaef5-0286-4389-b15a-edf62fe4f206`::moving from state preparing -> state finished >Thread-28487::DEBUG::2012-12-02 14:31:21,343::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28487::DEBUG::2012-12-02 14:31:21,343::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28487::DEBUG::2012-12-02 14:31:21,343::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28487::DEBUG::2012-12-02 14:31:21,343::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28487::DEBUG::2012-12-02 14:31:21,344::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28487::DEBUG::2012-12-02 14:31:21,344::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28487::DEBUG::2012-12-02 14:31:21,344::task::957::TaskManager.Task::(_decref) Task=`c23eaef5-0286-4389-b15a-edf62fe4f206`::ref 0 aborting False >Thread-28490::DEBUG::2012-12-02 14:31:25,893::task::568::TaskManager.Task::(_updateState) Task=`fbc38999-e177-4dce-aa3c-6523dc4249e5`::moving from state init -> state preparing >Thread-28490::INFO::2012-12-02 14:31:25,894::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28490::INFO::2012-12-02 14:31:25,894::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0083110332489', 'lastCheck': 1354451478.439379, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191712379456', 'lastCheck': 1354451484.49949, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00187993049622', 'lastCheck': 1354451485.8692579, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00629091262817', 'lastCheck': 1354451484.5051229, 'code': 0, 'valid': True}} >Thread-28490::DEBUG::2012-12-02 14:31:25,894::task::1151::TaskManager.Task::(prepare) Task=`fbc38999-e177-4dce-aa3c-6523dc4249e5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0083110332489', 'lastCheck': 1354451478.439379, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191712379456', 'lastCheck': 1354451484.49949, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00187993049622', 'lastCheck': 1354451485.8692579, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00629091262817', 'lastCheck': 1354451484.5051229, 'code': 0, 'valid': True}} >Thread-28490::DEBUG::2012-12-02 14:31:25,894::task::568::TaskManager.Task::(_updateState) Task=`fbc38999-e177-4dce-aa3c-6523dc4249e5`::moving from state preparing -> state finished >Thread-28490::DEBUG::2012-12-02 14:31:25,894::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28490::DEBUG::2012-12-02 14:31:25,894::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28490::DEBUG::2012-12-02 14:31:25,895::task::957::TaskManager.Task::(_decref) Task=`fbc38999-e177-4dce-aa3c-6523dc4249e5`::ref 0 aborting False >Thread-28494::DEBUG::2012-12-02 14:31:31,460::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28494::DEBUG::2012-12-02 14:31:31,461::task::568::TaskManager.Task::(_updateState) Task=`908cffc3-0363-42fb-9c57-415bb04216c4`::moving from state init -> state preparing >Thread-28494::INFO::2012-12-02 14:31:31,461::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28494::INFO::2012-12-02 14:31:31,461::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28494::DEBUG::2012-12-02 14:31:31,461::task::1151::TaskManager.Task::(prepare) Task=`908cffc3-0363-42fb-9c57-415bb04216c4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28494::DEBUG::2012-12-02 14:31:31,462::task::568::TaskManager.Task::(_updateState) Task=`908cffc3-0363-42fb-9c57-415bb04216c4`::moving from state preparing -> state finished >Thread-28494::DEBUG::2012-12-02 14:31:31,462::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28494::DEBUG::2012-12-02 14:31:31,462::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28494::DEBUG::2012-12-02 14:31:31,462::task::957::TaskManager.Task::(_decref) Task=`908cffc3-0363-42fb-9c57-415bb04216c4`::ref 0 aborting False >Thread-28495::DEBUG::2012-12-02 14:31:31,464::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28495::DEBUG::2012-12-02 14:31:31,465::task::568::TaskManager.Task::(_updateState) Task=`20dce374-4960-404d-bb5e-06229d712f5f`::moving from state init -> state preparing >Thread-28495::INFO::2012-12-02 14:31:31,465::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28495::DEBUG::2012-12-02 14:31:31,465::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9319c244-253f-47ac-9d0e-39b2711d42d0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28495::DEBUG::2012-12-02 14:31:31,465::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28495::DEBUG::2012-12-02 14:31:31,465::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28495::DEBUG::2012-12-02 14:31:31,465::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9319c244-253f-47ac-9d0e-39b2711d42d0`::Granted request >Thread-28495::DEBUG::2012-12-02 14:31:31,466::task::794::TaskManager.Task::(resourceAcquired) Task=`20dce374-4960-404d-bb5e-06229d712f5f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28495::DEBUG::2012-12-02 14:31:31,466::task::957::TaskManager.Task::(_decref) Task=`20dce374-4960-404d-bb5e-06229d712f5f`::ref 1 aborting False >Thread-28495::INFO::2012-12-02 14:31:31,470::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28495::DEBUG::2012-12-02 14:31:31,470::task::1151::TaskManager.Task::(prepare) Task=`20dce374-4960-404d-bb5e-06229d712f5f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28495::DEBUG::2012-12-02 14:31:31,471::task::568::TaskManager.Task::(_updateState) Task=`20dce374-4960-404d-bb5e-06229d712f5f`::moving from state preparing -> state finished >Thread-28495::DEBUG::2012-12-02 14:31:31,471::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28495::DEBUG::2012-12-02 14:31:31,471::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28495::DEBUG::2012-12-02 14:31:31,471::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28495::DEBUG::2012-12-02 14:31:31,471::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28495::DEBUG::2012-12-02 14:31:31,471::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28495::DEBUG::2012-12-02 14:31:31,471::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28495::DEBUG::2012-12-02 14:31:31,471::task::957::TaskManager.Task::(_decref) Task=`20dce374-4960-404d-bb5e-06229d712f5f`::ref 0 aborting False >Thread-28498::DEBUG::2012-12-02 14:31:36,014::task::568::TaskManager.Task::(_updateState) Task=`1f7d56b0-4d89-4101-8a63-1443be784a92`::moving from state init -> state preparing >Thread-28498::INFO::2012-12-02 14:31:36,014::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28498::INFO::2012-12-02 14:31:36,015::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794100761414', 'lastCheck': 1354451488.4568961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00492906570435', 'lastCheck': 1354451494.516659, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00244808197021', 'lastCheck': 1354451495.881731, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0073459148407', 'lastCheck': 1354451494.5243969, 'code': 0, 'valid': True}} >Thread-28498::DEBUG::2012-12-02 14:31:36,015::task::1151::TaskManager.Task::(prepare) Task=`1f7d56b0-4d89-4101-8a63-1443be784a92`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794100761414', 'lastCheck': 1354451488.4568961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00492906570435', 'lastCheck': 1354451494.516659, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00244808197021', 'lastCheck': 1354451495.881731, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0073459148407', 'lastCheck': 1354451494.5243969, 'code': 0, 'valid': True}} >Thread-28498::DEBUG::2012-12-02 14:31:36,015::task::568::TaskManager.Task::(_updateState) Task=`1f7d56b0-4d89-4101-8a63-1443be784a92`::moving from state preparing -> state finished >Thread-28498::DEBUG::2012-12-02 14:31:36,015::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28498::DEBUG::2012-12-02 14:31:36,015::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28498::DEBUG::2012-12-02 14:31:36,015::task::957::TaskManager.Task::(_decref) Task=`1f7d56b0-4d89-4101-8a63-1443be784a92`::ref 0 aborting False >Thread-28502::DEBUG::2012-12-02 14:31:41,565::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28502::DEBUG::2012-12-02 14:31:41,566::task::568::TaskManager.Task::(_updateState) Task=`805a29fe-8243-4ca0-8fa8-027e8ff9589f`::moving from state init -> state preparing >Thread-28502::INFO::2012-12-02 14:31:41,566::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28502::INFO::2012-12-02 14:31:41,566::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28502::DEBUG::2012-12-02 14:31:41,566::task::1151::TaskManager.Task::(prepare) Task=`805a29fe-8243-4ca0-8fa8-027e8ff9589f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28502::DEBUG::2012-12-02 14:31:41,567::task::568::TaskManager.Task::(_updateState) Task=`805a29fe-8243-4ca0-8fa8-027e8ff9589f`::moving from state preparing -> state finished >Thread-28502::DEBUG::2012-12-02 14:31:41,567::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28502::DEBUG::2012-12-02 14:31:41,567::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28502::DEBUG::2012-12-02 14:31:41,567::task::957::TaskManager.Task::(_decref) Task=`805a29fe-8243-4ca0-8fa8-027e8ff9589f`::ref 0 aborting False >Thread-28503::DEBUG::2012-12-02 14:31:41,569::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28503::DEBUG::2012-12-02 14:31:41,570::task::568::TaskManager.Task::(_updateState) Task=`1812875c-8ece-40ca-ad24-12ba401b64a9`::moving from state init -> state preparing >Thread-28503::INFO::2012-12-02 14:31:41,570::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28503::DEBUG::2012-12-02 14:31:41,570::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1fac5c43-b3db-4c45-b292-c2aeb291fccf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28503::DEBUG::2012-12-02 14:31:41,570::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28503::DEBUG::2012-12-02 14:31:41,570::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28503::DEBUG::2012-12-02 14:31:41,571::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1fac5c43-b3db-4c45-b292-c2aeb291fccf`::Granted request >Thread-28503::DEBUG::2012-12-02 14:31:41,571::task::794::TaskManager.Task::(resourceAcquired) Task=`1812875c-8ece-40ca-ad24-12ba401b64a9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28503::DEBUG::2012-12-02 14:31:41,571::task::957::TaskManager.Task::(_decref) Task=`1812875c-8ece-40ca-ad24-12ba401b64a9`::ref 1 aborting False >Thread-28503::INFO::2012-12-02 14:31:41,577::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28503::DEBUG::2012-12-02 14:31:41,577::task::1151::TaskManager.Task::(prepare) Task=`1812875c-8ece-40ca-ad24-12ba401b64a9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28503::DEBUG::2012-12-02 14:31:41,577::task::568::TaskManager.Task::(_updateState) Task=`1812875c-8ece-40ca-ad24-12ba401b64a9`::moving from state preparing -> state finished >Thread-28503::DEBUG::2012-12-02 14:31:41,577::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28503::DEBUG::2012-12-02 14:31:41,577::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28503::DEBUG::2012-12-02 14:31:41,577::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28503::DEBUG::2012-12-02 14:31:41,578::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28503::DEBUG::2012-12-02 14:31:41,578::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28503::DEBUG::2012-12-02 14:31:41,578::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28503::DEBUG::2012-12-02 14:31:41,578::task::957::TaskManager.Task::(_decref) Task=`1812875c-8ece-40ca-ad24-12ba401b64a9`::ref 0 aborting False >Thread-28506::DEBUG::2012-12-02 14:31:46,104::task::568::TaskManager.Task::(_updateState) Task=`b31a9c41-2c0d-43a2-a5e1-5f821bd37d97`::moving from state init -> state preparing >Thread-28506::INFO::2012-12-02 14:31:46,104::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28506::INFO::2012-12-02 14:31:46,104::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00804781913757', 'lastCheck': 1354451498.4782059, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00415110588074', 'lastCheck': 1354451504.535538, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00176501274109', 'lastCheck': 1354451505.8915269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00813293457031', 'lastCheck': 1354451504.5449989, 'code': 0, 'valid': True}} >Thread-28506::DEBUG::2012-12-02 14:31:46,104::task::1151::TaskManager.Task::(prepare) Task=`b31a9c41-2c0d-43a2-a5e1-5f821bd37d97`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00804781913757', 'lastCheck': 1354451498.4782059, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00415110588074', 'lastCheck': 1354451504.535538, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00176501274109', 'lastCheck': 1354451505.8915269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00813293457031', 'lastCheck': 1354451504.5449989, 'code': 0, 'valid': True}} >Thread-28506::DEBUG::2012-12-02 14:31:46,105::task::568::TaskManager.Task::(_updateState) Task=`b31a9c41-2c0d-43a2-a5e1-5f821bd37d97`::moving from state preparing -> state finished >Thread-28506::DEBUG::2012-12-02 14:31:46,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28506::DEBUG::2012-12-02 14:31:46,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28506::DEBUG::2012-12-02 14:31:46,105::task::957::TaskManager.Task::(_decref) Task=`b31a9c41-2c0d-43a2-a5e1-5f821bd37d97`::ref 0 aborting False >Thread-28510::DEBUG::2012-12-02 14:31:51,722::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28510::DEBUG::2012-12-02 14:31:51,722::task::568::TaskManager.Task::(_updateState) Task=`14f4455f-8a08-44d3-a5c9-2126a00d5a98`::moving from state init -> state preparing >Thread-28510::INFO::2012-12-02 14:31:51,722::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28510::INFO::2012-12-02 14:31:51,722::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28510::DEBUG::2012-12-02 14:31:51,723::task::1151::TaskManager.Task::(prepare) Task=`14f4455f-8a08-44d3-a5c9-2126a00d5a98`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28510::DEBUG::2012-12-02 14:31:51,723::task::568::TaskManager.Task::(_updateState) Task=`14f4455f-8a08-44d3-a5c9-2126a00d5a98`::moving from state preparing -> state finished >Thread-28510::DEBUG::2012-12-02 14:31:51,723::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28510::DEBUG::2012-12-02 14:31:51,723::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28510::DEBUG::2012-12-02 14:31:51,723::task::957::TaskManager.Task::(_decref) Task=`14f4455f-8a08-44d3-a5c9-2126a00d5a98`::ref 0 aborting False >Thread-28511::DEBUG::2012-12-02 14:31:51,726::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28511::DEBUG::2012-12-02 14:31:51,726::task::568::TaskManager.Task::(_updateState) Task=`12fe77e1-74c4-45a2-97cc-4d8dab0d14ec`::moving from state init -> state preparing >Thread-28511::INFO::2012-12-02 14:31:51,726::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28511::DEBUG::2012-12-02 14:31:51,726::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b6c5dcc4-b272-40f8-ac3f-ceb5186f7447`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28511::DEBUG::2012-12-02 14:31:51,727::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28511::DEBUG::2012-12-02 14:31:51,727::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28511::DEBUG::2012-12-02 14:31:51,727::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b6c5dcc4-b272-40f8-ac3f-ceb5186f7447`::Granted request >Thread-28511::DEBUG::2012-12-02 14:31:51,727::task::794::TaskManager.Task::(resourceAcquired) Task=`12fe77e1-74c4-45a2-97cc-4d8dab0d14ec`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28511::DEBUG::2012-12-02 14:31:51,727::task::957::TaskManager.Task::(_decref) Task=`12fe77e1-74c4-45a2-97cc-4d8dab0d14ec`::ref 1 aborting False >Thread-28511::INFO::2012-12-02 14:31:51,734::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28511::DEBUG::2012-12-02 14:31:51,734::task::1151::TaskManager.Task::(prepare) Task=`12fe77e1-74c4-45a2-97cc-4d8dab0d14ec`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28511::DEBUG::2012-12-02 14:31:51,734::task::568::TaskManager.Task::(_updateState) Task=`12fe77e1-74c4-45a2-97cc-4d8dab0d14ec`::moving from state preparing -> state finished >Thread-28511::DEBUG::2012-12-02 14:31:51,734::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28511::DEBUG::2012-12-02 14:31:51,734::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28511::DEBUG::2012-12-02 14:31:51,734::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28511::DEBUG::2012-12-02 14:31:51,735::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28511::DEBUG::2012-12-02 14:31:51,735::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28511::DEBUG::2012-12-02 14:31:51,735::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28511::DEBUG::2012-12-02 14:31:51,735::task::957::TaskManager.Task::(_decref) Task=`12fe77e1-74c4-45a2-97cc-4d8dab0d14ec`::ref 0 aborting False >Thread-28514::DEBUG::2012-12-02 14:31:56,261::task::568::TaskManager.Task::(_updateState) Task=`b312c370-e6b2-47fb-9009-448a2ff018ff`::moving from state init -> state preparing >Thread-28514::INFO::2012-12-02 14:31:56,261::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28514::INFO::2012-12-02 14:31:56,261::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00814914703369', 'lastCheck': 1354451508.4961901, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00365686416626', 'lastCheck': 1354451514.5529561, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178599357605', 'lastCheck': 1354451515.9016881, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00734305381775', 'lastCheck': 1354451514.5621181, 'code': 0, 'valid': True}} >Thread-28514::DEBUG::2012-12-02 14:31:56,261::task::1151::TaskManager.Task::(prepare) Task=`b312c370-e6b2-47fb-9009-448a2ff018ff`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00814914703369', 'lastCheck': 1354451508.4961901, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00365686416626', 'lastCheck': 1354451514.5529561, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178599357605', 'lastCheck': 1354451515.9016881, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00734305381775', 'lastCheck': 1354451514.5621181, 'code': 0, 'valid': True}} >Thread-28514::DEBUG::2012-12-02 14:31:56,261::task::568::TaskManager.Task::(_updateState) Task=`b312c370-e6b2-47fb-9009-448a2ff018ff`::moving from state preparing -> state finished >Thread-28514::DEBUG::2012-12-02 14:31:56,262::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28514::DEBUG::2012-12-02 14:31:56,262::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28514::DEBUG::2012-12-02 14:31:56,262::task::957::TaskManager.Task::(_decref) Task=`b312c370-e6b2-47fb-9009-448a2ff018ff`::ref 0 aborting False >Thread-28518::DEBUG::2012-12-02 14:32:01,859::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28518::DEBUG::2012-12-02 14:32:01,859::task::568::TaskManager.Task::(_updateState) Task=`dd81a570-5144-4526-b981-cadc97187130`::moving from state init -> state preparing >Thread-28518::INFO::2012-12-02 14:32:01,859::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28518::INFO::2012-12-02 14:32:01,859::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28518::DEBUG::2012-12-02 14:32:01,859::task::1151::TaskManager.Task::(prepare) Task=`dd81a570-5144-4526-b981-cadc97187130`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28518::DEBUG::2012-12-02 14:32:01,860::task::568::TaskManager.Task::(_updateState) Task=`dd81a570-5144-4526-b981-cadc97187130`::moving from state preparing -> state finished >Thread-28518::DEBUG::2012-12-02 14:32:01,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28518::DEBUG::2012-12-02 14:32:01,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28518::DEBUG::2012-12-02 14:32:01,860::task::957::TaskManager.Task::(_decref) Task=`dd81a570-5144-4526-b981-cadc97187130`::ref 0 aborting False >Thread-28519::DEBUG::2012-12-02 14:32:01,862::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28519::DEBUG::2012-12-02 14:32:01,863::task::568::TaskManager.Task::(_updateState) Task=`f42a46a2-7cd0-405e-8308-9034dfe6084e`::moving from state init -> state preparing >Thread-28519::INFO::2012-12-02 14:32:01,863::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28519::DEBUG::2012-12-02 14:32:01,863::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c2224c34-abd5-4573-86f1-1f2becafea45`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28519::DEBUG::2012-12-02 14:32:01,863::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28519::DEBUG::2012-12-02 14:32:01,863::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28519::DEBUG::2012-12-02 14:32:01,864::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c2224c34-abd5-4573-86f1-1f2becafea45`::Granted request >Thread-28519::DEBUG::2012-12-02 14:32:01,864::task::794::TaskManager.Task::(resourceAcquired) Task=`f42a46a2-7cd0-405e-8308-9034dfe6084e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28519::DEBUG::2012-12-02 14:32:01,864::task::957::TaskManager.Task::(_decref) Task=`f42a46a2-7cd0-405e-8308-9034dfe6084e`::ref 1 aborting False >Thread-28519::INFO::2012-12-02 14:32:01,869::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28519::DEBUG::2012-12-02 14:32:01,869::task::1151::TaskManager.Task::(prepare) Task=`f42a46a2-7cd0-405e-8308-9034dfe6084e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28519::DEBUG::2012-12-02 14:32:01,869::task::568::TaskManager.Task::(_updateState) Task=`f42a46a2-7cd0-405e-8308-9034dfe6084e`::moving from state preparing -> state finished >Thread-28519::DEBUG::2012-12-02 14:32:01,869::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28519::DEBUG::2012-12-02 14:32:01,869::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28519::DEBUG::2012-12-02 14:32:01,869::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28519::DEBUG::2012-12-02 14:32:01,870::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28519::DEBUG::2012-12-02 14:32:01,870::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28519::DEBUG::2012-12-02 14:32:01,870::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28519::DEBUG::2012-12-02 14:32:01,870::task::957::TaskManager.Task::(_decref) Task=`f42a46a2-7cd0-405e-8308-9034dfe6084e`::ref 0 aborting False >Thread-28522::DEBUG::2012-12-02 14:32:06,354::task::568::TaskManager.Task::(_updateState) Task=`da5d35d8-18b7-4b33-903b-3f127ccbe153`::moving from state init -> state preparing >Thread-28522::INFO::2012-12-02 14:32:06,354::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28522::INFO::2012-12-02 14:32:06,355::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00960803031921', 'lastCheck': 1354451518.5149519, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00387716293335', 'lastCheck': 1354451524.569433, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171208381653', 'lastCheck': 1354451525.9114599, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00831198692322', 'lastCheck': 1354451524.579927, 'code': 0, 'valid': True}} >Thread-28522::DEBUG::2012-12-02 14:32:06,355::task::1151::TaskManager.Task::(prepare) Task=`da5d35d8-18b7-4b33-903b-3f127ccbe153`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00960803031921', 'lastCheck': 1354451518.5149519, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00387716293335', 'lastCheck': 1354451524.569433, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171208381653', 'lastCheck': 1354451525.9114599, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00831198692322', 'lastCheck': 1354451524.579927, 'code': 0, 'valid': True}} >Thread-28522::DEBUG::2012-12-02 14:32:06,355::task::568::TaskManager.Task::(_updateState) Task=`da5d35d8-18b7-4b33-903b-3f127ccbe153`::moving from state preparing -> state finished >Thread-28522::DEBUG::2012-12-02 14:32:06,355::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28522::DEBUG::2012-12-02 14:32:06,355::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28522::DEBUG::2012-12-02 14:32:06,355::task::957::TaskManager.Task::(_decref) Task=`da5d35d8-18b7-4b33-903b-3f127ccbe153`::ref 0 aborting False >Thread-28526::DEBUG::2012-12-02 14:32:11,997::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28526::DEBUG::2012-12-02 14:32:11,997::task::568::TaskManager.Task::(_updateState) Task=`767129ce-8761-477b-b8c5-1a44e30bdc21`::moving from state init -> state preparing >Thread-28526::INFO::2012-12-02 14:32:11,997::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28526::INFO::2012-12-02 14:32:11,998::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28526::DEBUG::2012-12-02 14:32:11,998::task::1151::TaskManager.Task::(prepare) Task=`767129ce-8761-477b-b8c5-1a44e30bdc21`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28526::DEBUG::2012-12-02 14:32:11,998::task::568::TaskManager.Task::(_updateState) Task=`767129ce-8761-477b-b8c5-1a44e30bdc21`::moving from state preparing -> state finished >Thread-28526::DEBUG::2012-12-02 14:32:11,998::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28526::DEBUG::2012-12-02 14:32:11,998::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28526::DEBUG::2012-12-02 14:32:11,998::task::957::TaskManager.Task::(_decref) Task=`767129ce-8761-477b-b8c5-1a44e30bdc21`::ref 0 aborting False >Thread-28527::DEBUG::2012-12-02 14:32:12,001::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28527::DEBUG::2012-12-02 14:32:12,001::task::568::TaskManager.Task::(_updateState) Task=`7337a044-0d9b-4f9c-a49c-d98075304ada`::moving from state init -> state preparing >Thread-28527::INFO::2012-12-02 14:32:12,001::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28527::DEBUG::2012-12-02 14:32:12,002::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d52d8b75-8858-4982-ae3b-4ce7cd2c3b24`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28527::DEBUG::2012-12-02 14:32:12,002::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28527::DEBUG::2012-12-02 14:32:12,002::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28527::DEBUG::2012-12-02 14:32:12,002::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d52d8b75-8858-4982-ae3b-4ce7cd2c3b24`::Granted request >Thread-28527::DEBUG::2012-12-02 14:32:12,002::task::794::TaskManager.Task::(resourceAcquired) Task=`7337a044-0d9b-4f9c-a49c-d98075304ada`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28527::DEBUG::2012-12-02 14:32:12,002::task::957::TaskManager.Task::(_decref) Task=`7337a044-0d9b-4f9c-a49c-d98075304ada`::ref 1 aborting False >Thread-28527::INFO::2012-12-02 14:32:12,008::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28527::DEBUG::2012-12-02 14:32:12,008::task::1151::TaskManager.Task::(prepare) Task=`7337a044-0d9b-4f9c-a49c-d98075304ada`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28527::DEBUG::2012-12-02 14:32:12,008::task::568::TaskManager.Task::(_updateState) Task=`7337a044-0d9b-4f9c-a49c-d98075304ada`::moving from state preparing -> state finished >Thread-28527::DEBUG::2012-12-02 14:32:12,008::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28527::DEBUG::2012-12-02 14:32:12,008::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28527::DEBUG::2012-12-02 14:32:12,008::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28527::DEBUG::2012-12-02 14:32:12,008::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28527::DEBUG::2012-12-02 14:32:12,008::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28527::DEBUG::2012-12-02 14:32:12,008::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28527::DEBUG::2012-12-02 14:32:12,009::task::957::TaskManager.Task::(_decref) Task=`7337a044-0d9b-4f9c-a49c-d98075304ada`::ref 0 aborting False >Thread-28530::DEBUG::2012-12-02 14:32:16,460::task::568::TaskManager.Task::(_updateState) Task=`03d671cb-d795-4372-a7c3-486c286c869d`::moving from state init -> state preparing >Thread-28530::INFO::2012-12-02 14:32:16,460::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28530::INFO::2012-12-02 14:32:16,460::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00354099273682', 'lastCheck': 1354451528.5268519, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00538992881775', 'lastCheck': 1354451534.588078, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00236701965332', 'lastCheck': 1354451535.922889, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00742602348328', 'lastCheck': 1354451534.597496, 'code': 0, 'valid': True}} >Thread-28530::DEBUG::2012-12-02 14:32:16,460::task::1151::TaskManager.Task::(prepare) Task=`03d671cb-d795-4372-a7c3-486c286c869d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00354099273682', 'lastCheck': 1354451528.5268519, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00538992881775', 'lastCheck': 1354451534.588078, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00236701965332', 'lastCheck': 1354451535.922889, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00742602348328', 'lastCheck': 1354451534.597496, 'code': 0, 'valid': True}} >Thread-28530::DEBUG::2012-12-02 14:32:16,460::task::568::TaskManager.Task::(_updateState) Task=`03d671cb-d795-4372-a7c3-486c286c869d`::moving from state preparing -> state finished >Thread-28530::DEBUG::2012-12-02 14:32:16,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28530::DEBUG::2012-12-02 14:32:16,461::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28530::DEBUG::2012-12-02 14:32:16,461::task::957::TaskManager.Task::(_decref) Task=`03d671cb-d795-4372-a7c3-486c286c869d`::ref 0 aborting False >Thread-28534::DEBUG::2012-12-02 14:32:22,127::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28534::DEBUG::2012-12-02 14:32:22,127::task::568::TaskManager.Task::(_updateState) Task=`3fd0cc78-cfce-498e-a73b-081790c42224`::moving from state init -> state preparing >Thread-28534::INFO::2012-12-02 14:32:22,127::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28534::INFO::2012-12-02 14:32:22,128::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28534::DEBUG::2012-12-02 14:32:22,128::task::1151::TaskManager.Task::(prepare) Task=`3fd0cc78-cfce-498e-a73b-081790c42224`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28534::DEBUG::2012-12-02 14:32:22,128::task::568::TaskManager.Task::(_updateState) Task=`3fd0cc78-cfce-498e-a73b-081790c42224`::moving from state preparing -> state finished >Thread-28534::DEBUG::2012-12-02 14:32:22,128::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28534::DEBUG::2012-12-02 14:32:22,128::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28534::DEBUG::2012-12-02 14:32:22,128::task::957::TaskManager.Task::(_decref) Task=`3fd0cc78-cfce-498e-a73b-081790c42224`::ref 0 aborting False >Thread-28535::DEBUG::2012-12-02 14:32:22,131::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28535::DEBUG::2012-12-02 14:32:22,131::task::568::TaskManager.Task::(_updateState) Task=`0ef4b3a4-b741-4bbf-b87b-3c7840f29816`::moving from state init -> state preparing >Thread-28535::INFO::2012-12-02 14:32:22,131::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28535::DEBUG::2012-12-02 14:32:22,131::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`16107ea5-f909-4d9e-9107-4ec9ee2833fc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28535::DEBUG::2012-12-02 14:32:22,132::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28535::DEBUG::2012-12-02 14:32:22,132::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28535::DEBUG::2012-12-02 14:32:22,132::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`16107ea5-f909-4d9e-9107-4ec9ee2833fc`::Granted request >Thread-28535::DEBUG::2012-12-02 14:32:22,132::task::794::TaskManager.Task::(resourceAcquired) Task=`0ef4b3a4-b741-4bbf-b87b-3c7840f29816`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28535::DEBUG::2012-12-02 14:32:22,132::task::957::TaskManager.Task::(_decref) Task=`0ef4b3a4-b741-4bbf-b87b-3c7840f29816`::ref 1 aborting False >Thread-28535::INFO::2012-12-02 14:32:22,138::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28535::DEBUG::2012-12-02 14:32:22,138::task::1151::TaskManager.Task::(prepare) Task=`0ef4b3a4-b741-4bbf-b87b-3c7840f29816`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28535::DEBUG::2012-12-02 14:32:22,138::task::568::TaskManager.Task::(_updateState) Task=`0ef4b3a4-b741-4bbf-b87b-3c7840f29816`::moving from state preparing -> state finished >Thread-28535::DEBUG::2012-12-02 14:32:22,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28535::DEBUG::2012-12-02 14:32:22,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28535::DEBUG::2012-12-02 14:32:22,138::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28535::DEBUG::2012-12-02 14:32:22,138::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28535::DEBUG::2012-12-02 14:32:22,138::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28535::DEBUG::2012-12-02 14:32:22,138::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28535::DEBUG::2012-12-02 14:32:22,138::task::957::TaskManager.Task::(_decref) Task=`0ef4b3a4-b741-4bbf-b87b-3c7840f29816`::ref 0 aborting False >Thread-28538::DEBUG::2012-12-02 14:32:26,523::task::568::TaskManager.Task::(_updateState) Task=`c5b315e1-9d56-4eab-a2f9-f4a0209784c3`::moving from state init -> state preparing >Thread-28538::INFO::2012-12-02 14:32:26,523::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28538::INFO::2012-12-02 14:32:26,523::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00758004188538', 'lastCheck': 1354451538.5437911, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00386905670166', 'lastCheck': 1354451544.6098821, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00161981582642', 'lastCheck': 1354451545.9323101, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00813102722168', 'lastCheck': 1354451544.616679, 'code': 0, 'valid': True}} >Thread-28538::DEBUG::2012-12-02 14:32:26,523::task::1151::TaskManager.Task::(prepare) Task=`c5b315e1-9d56-4eab-a2f9-f4a0209784c3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00758004188538', 'lastCheck': 1354451538.5437911, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00386905670166', 'lastCheck': 1354451544.6098821, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00161981582642', 'lastCheck': 1354451545.9323101, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00813102722168', 'lastCheck': 1354451544.616679, 'code': 0, 'valid': True}} >Thread-28538::DEBUG::2012-12-02 14:32:26,523::task::568::TaskManager.Task::(_updateState) Task=`c5b315e1-9d56-4eab-a2f9-f4a0209784c3`::moving from state preparing -> state finished >Thread-28538::DEBUG::2012-12-02 14:32:26,523::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28538::DEBUG::2012-12-02 14:32:26,523::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28538::DEBUG::2012-12-02 14:32:26,524::task::957::TaskManager.Task::(_decref) Task=`c5b315e1-9d56-4eab-a2f9-f4a0209784c3`::ref 0 aborting False >Thread-28542::DEBUG::2012-12-02 14:32:32,232::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28542::DEBUG::2012-12-02 14:32:32,232::task::568::TaskManager.Task::(_updateState) Task=`cdaa9156-2e4b-4b0b-9dba-f232e0b589d7`::moving from state init -> state preparing >Thread-28542::INFO::2012-12-02 14:32:32,232::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28542::INFO::2012-12-02 14:32:32,232::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28542::DEBUG::2012-12-02 14:32:32,233::task::1151::TaskManager.Task::(prepare) Task=`cdaa9156-2e4b-4b0b-9dba-f232e0b589d7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28542::DEBUG::2012-12-02 14:32:32,233::task::568::TaskManager.Task::(_updateState) Task=`cdaa9156-2e4b-4b0b-9dba-f232e0b589d7`::moving from state preparing -> state finished >Thread-28542::DEBUG::2012-12-02 14:32:32,233::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28542::DEBUG::2012-12-02 14:32:32,233::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28542::DEBUG::2012-12-02 14:32:32,233::task::957::TaskManager.Task::(_decref) Task=`cdaa9156-2e4b-4b0b-9dba-f232e0b589d7`::ref 0 aborting False >Thread-28543::DEBUG::2012-12-02 14:32:32,236::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28543::DEBUG::2012-12-02 14:32:32,236::task::568::TaskManager.Task::(_updateState) Task=`8a1b509f-dbd1-43f1-b0c3-342d9f133994`::moving from state init -> state preparing >Thread-28543::INFO::2012-12-02 14:32:32,236::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28543::DEBUG::2012-12-02 14:32:32,236::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`54a3e384-bde1-4f96-9e71-a2718eb8e08e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28543::DEBUG::2012-12-02 14:32:32,237::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28543::DEBUG::2012-12-02 14:32:32,237::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28543::DEBUG::2012-12-02 14:32:32,237::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`54a3e384-bde1-4f96-9e71-a2718eb8e08e`::Granted request >Thread-28543::DEBUG::2012-12-02 14:32:32,237::task::794::TaskManager.Task::(resourceAcquired) Task=`8a1b509f-dbd1-43f1-b0c3-342d9f133994`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28543::DEBUG::2012-12-02 14:32:32,237::task::957::TaskManager.Task::(_decref) Task=`8a1b509f-dbd1-43f1-b0c3-342d9f133994`::ref 1 aborting False >Thread-28543::INFO::2012-12-02 14:32:32,244::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28543::DEBUG::2012-12-02 14:32:32,244::task::1151::TaskManager.Task::(prepare) Task=`8a1b509f-dbd1-43f1-b0c3-342d9f133994`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28543::DEBUG::2012-12-02 14:32:32,244::task::568::TaskManager.Task::(_updateState) Task=`8a1b509f-dbd1-43f1-b0c3-342d9f133994`::moving from state preparing -> state finished >Thread-28543::DEBUG::2012-12-02 14:32:32,244::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28543::DEBUG::2012-12-02 14:32:32,244::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28543::DEBUG::2012-12-02 14:32:32,244::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28543::DEBUG::2012-12-02 14:32:32,245::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28543::DEBUG::2012-12-02 14:32:32,245::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28543::DEBUG::2012-12-02 14:32:32,245::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28543::DEBUG::2012-12-02 14:32:32,245::task::957::TaskManager.Task::(_decref) Task=`8a1b509f-dbd1-43f1-b0c3-342d9f133994`::ref 0 aborting False >Thread-28546::DEBUG::2012-12-02 14:32:36,628::task::568::TaskManager.Task::(_updateState) Task=`dfd7cb2e-a85a-4262-b9aa-5f0c9e28c9ab`::moving from state init -> state preparing >Thread-28546::INFO::2012-12-02 14:32:36,628::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28546::INFO::2012-12-02 14:32:36,628::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00850510597229', 'lastCheck': 1354451548.560647, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.006108045578', 'lastCheck': 1354451554.6283159, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00257587432861', 'lastCheck': 1354451555.944025, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00795197486877', 'lastCheck': 1354451554.6374011, 'code': 0, 'valid': True}} >Thread-28546::DEBUG::2012-12-02 14:32:36,628::task::1151::TaskManager.Task::(prepare) Task=`dfd7cb2e-a85a-4262-b9aa-5f0c9e28c9ab`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00850510597229', 'lastCheck': 1354451548.560647, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.006108045578', 'lastCheck': 1354451554.6283159, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00257587432861', 'lastCheck': 1354451555.944025, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00795197486877', 'lastCheck': 1354451554.6374011, 'code': 0, 'valid': True}} >Thread-28546::DEBUG::2012-12-02 14:32:36,628::task::568::TaskManager.Task::(_updateState) Task=`dfd7cb2e-a85a-4262-b9aa-5f0c9e28c9ab`::moving from state preparing -> state finished >Thread-28546::DEBUG::2012-12-02 14:32:36,629::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28546::DEBUG::2012-12-02 14:32:36,629::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28546::DEBUG::2012-12-02 14:32:36,629::task::957::TaskManager.Task::(_decref) Task=`dfd7cb2e-a85a-4262-b9aa-5f0c9e28c9ab`::ref 0 aborting False >Thread-28550::DEBUG::2012-12-02 14:32:42,362::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28550::DEBUG::2012-12-02 14:32:42,363::task::568::TaskManager.Task::(_updateState) Task=`745cebda-af63-4f83-a38b-68e9f6c01ab8`::moving from state init -> state preparing >Thread-28550::INFO::2012-12-02 14:32:42,363::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28550::INFO::2012-12-02 14:32:42,363::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28550::DEBUG::2012-12-02 14:32:42,363::task::1151::TaskManager.Task::(prepare) Task=`745cebda-af63-4f83-a38b-68e9f6c01ab8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28550::DEBUG::2012-12-02 14:32:42,363::task::568::TaskManager.Task::(_updateState) Task=`745cebda-af63-4f83-a38b-68e9f6c01ab8`::moving from state preparing -> state finished >Thread-28550::DEBUG::2012-12-02 14:32:42,364::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28550::DEBUG::2012-12-02 14:32:42,364::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28550::DEBUG::2012-12-02 14:32:42,364::task::957::TaskManager.Task::(_decref) Task=`745cebda-af63-4f83-a38b-68e9f6c01ab8`::ref 0 aborting False >Thread-28551::DEBUG::2012-12-02 14:32:42,366::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28551::DEBUG::2012-12-02 14:32:42,367::task::568::TaskManager.Task::(_updateState) Task=`ca4c18f8-4e4d-48b4-896a-ce28936d3ff3`::moving from state init -> state preparing >Thread-28551::INFO::2012-12-02 14:32:42,367::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28551::DEBUG::2012-12-02 14:32:42,367::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`46b8d84c-3492-41c0-a60f-578cdc78ce91`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28551::DEBUG::2012-12-02 14:32:42,367::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28551::DEBUG::2012-12-02 14:32:42,367::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28551::DEBUG::2012-12-02 14:32:42,367::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`46b8d84c-3492-41c0-a60f-578cdc78ce91`::Granted request >Thread-28551::DEBUG::2012-12-02 14:32:42,368::task::794::TaskManager.Task::(resourceAcquired) Task=`ca4c18f8-4e4d-48b4-896a-ce28936d3ff3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28551::DEBUG::2012-12-02 14:32:42,368::task::957::TaskManager.Task::(_decref) Task=`ca4c18f8-4e4d-48b4-896a-ce28936d3ff3`::ref 1 aborting False >Thread-28551::INFO::2012-12-02 14:32:42,374::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28551::DEBUG::2012-12-02 14:32:42,374::task::1151::TaskManager.Task::(prepare) Task=`ca4c18f8-4e4d-48b4-896a-ce28936d3ff3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28551::DEBUG::2012-12-02 14:32:42,375::task::568::TaskManager.Task::(_updateState) Task=`ca4c18f8-4e4d-48b4-896a-ce28936d3ff3`::moving from state preparing -> state finished >Thread-28551::DEBUG::2012-12-02 14:32:42,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28551::DEBUG::2012-12-02 14:32:42,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28551::DEBUG::2012-12-02 14:32:42,375::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28551::DEBUG::2012-12-02 14:32:42,375::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28551::DEBUG::2012-12-02 14:32:42,375::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28551::DEBUG::2012-12-02 14:32:42,375::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28551::DEBUG::2012-12-02 14:32:42,375::task::957::TaskManager.Task::(_decref) Task=`ca4c18f8-4e4d-48b4-896a-ce28936d3ff3`::ref 0 aborting False >Thread-28554::DEBUG::2012-12-02 14:32:46,719::task::568::TaskManager.Task::(_updateState) Task=`83896429-021d-4f25-90ed-ba3ab8875034`::moving from state init -> state preparing >Thread-28554::INFO::2012-12-02 14:32:46,719::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28554::INFO::2012-12-02 14:32:46,720::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00865411758423', 'lastCheck': 1354451558.5808649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0021231174469', 'lastCheck': 1354451564.6471441, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172901153564', 'lastCheck': 1354451565.953136, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00748014450073', 'lastCheck': 1354451564.6573501, 'code': 0, 'valid': True}} >Thread-28554::DEBUG::2012-12-02 14:32:46,720::task::1151::TaskManager.Task::(prepare) Task=`83896429-021d-4f25-90ed-ba3ab8875034`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00865411758423', 'lastCheck': 1354451558.5808649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0021231174469', 'lastCheck': 1354451564.6471441, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172901153564', 'lastCheck': 1354451565.953136, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00748014450073', 'lastCheck': 1354451564.6573501, 'code': 0, 'valid': True}} >Thread-28554::DEBUG::2012-12-02 14:32:46,720::task::568::TaskManager.Task::(_updateState) Task=`83896429-021d-4f25-90ed-ba3ab8875034`::moving from state preparing -> state finished >Thread-28554::DEBUG::2012-12-02 14:32:46,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28554::DEBUG::2012-12-02 14:32:46,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28554::DEBUG::2012-12-02 14:32:46,720::task::957::TaskManager.Task::(_decref) Task=`83896429-021d-4f25-90ed-ba3ab8875034`::ref 0 aborting False >Thread-28558::DEBUG::2012-12-02 14:32:52,467::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28558::DEBUG::2012-12-02 14:32:52,467::task::568::TaskManager.Task::(_updateState) Task=`d6876597-dddb-4198-9e3d-4f2715451080`::moving from state init -> state preparing >Thread-28558::INFO::2012-12-02 14:32:52,467::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28558::INFO::2012-12-02 14:32:52,467::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28558::DEBUG::2012-12-02 14:32:52,468::task::1151::TaskManager.Task::(prepare) Task=`d6876597-dddb-4198-9e3d-4f2715451080`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28558::DEBUG::2012-12-02 14:32:52,468::task::568::TaskManager.Task::(_updateState) Task=`d6876597-dddb-4198-9e3d-4f2715451080`::moving from state preparing -> state finished >Thread-28558::DEBUG::2012-12-02 14:32:52,468::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28558::DEBUG::2012-12-02 14:32:52,468::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28558::DEBUG::2012-12-02 14:32:52,468::task::957::TaskManager.Task::(_decref) Task=`d6876597-dddb-4198-9e3d-4f2715451080`::ref 0 aborting False >Thread-28559::DEBUG::2012-12-02 14:32:52,471::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28559::DEBUG::2012-12-02 14:32:52,471::task::568::TaskManager.Task::(_updateState) Task=`aca97da2-e209-4323-ad4d-6bbe4ced2fa9`::moving from state init -> state preparing >Thread-28559::INFO::2012-12-02 14:32:52,471::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28559::DEBUG::2012-12-02 14:32:52,472::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7a378093-b152-45ef-aabb-87483f84de8b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28559::DEBUG::2012-12-02 14:32:52,472::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28559::DEBUG::2012-12-02 14:32:52,472::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28559::DEBUG::2012-12-02 14:32:52,472::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7a378093-b152-45ef-aabb-87483f84de8b`::Granted request >Thread-28559::DEBUG::2012-12-02 14:32:52,472::task::794::TaskManager.Task::(resourceAcquired) Task=`aca97da2-e209-4323-ad4d-6bbe4ced2fa9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28559::DEBUG::2012-12-02 14:32:52,473::task::957::TaskManager.Task::(_decref) Task=`aca97da2-e209-4323-ad4d-6bbe4ced2fa9`::ref 1 aborting False >Thread-28559::INFO::2012-12-02 14:32:52,478::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28559::DEBUG::2012-12-02 14:32:52,478::task::1151::TaskManager.Task::(prepare) Task=`aca97da2-e209-4323-ad4d-6bbe4ced2fa9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28559::DEBUG::2012-12-02 14:32:52,478::task::568::TaskManager.Task::(_updateState) Task=`aca97da2-e209-4323-ad4d-6bbe4ced2fa9`::moving from state preparing -> state finished >Thread-28559::DEBUG::2012-12-02 14:32:52,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28559::DEBUG::2012-12-02 14:32:52,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28559::DEBUG::2012-12-02 14:32:52,478::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28559::DEBUG::2012-12-02 14:32:52,478::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28559::DEBUG::2012-12-02 14:32:52,478::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28559::DEBUG::2012-12-02 14:32:52,478::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28559::DEBUG::2012-12-02 14:32:52,479::task::957::TaskManager.Task::(_decref) Task=`aca97da2-e209-4323-ad4d-6bbe4ced2fa9`::ref 0 aborting False >Thread-28562::DEBUG::2012-12-02 14:32:56,822::task::568::TaskManager.Task::(_updateState) Task=`76291d28-f40d-4120-b61a-086ca21e00f0`::moving from state init -> state preparing >Thread-28562::INFO::2012-12-02 14:32:56,823::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28562::INFO::2012-12-02 14:32:56,823::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00849890708923', 'lastCheck': 1354451568.5984659, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266098976135', 'lastCheck': 1354451574.6639991, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00185489654541', 'lastCheck': 1354451575.9624021, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00817704200745', 'lastCheck': 1354451574.674825, 'code': 0, 'valid': True}} >Thread-28562::DEBUG::2012-12-02 14:32:56,823::task::1151::TaskManager.Task::(prepare) Task=`76291d28-f40d-4120-b61a-086ca21e00f0`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00849890708923', 'lastCheck': 1354451568.5984659, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266098976135', 'lastCheck': 1354451574.6639991, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00185489654541', 'lastCheck': 1354451575.9624021, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00817704200745', 'lastCheck': 1354451574.674825, 'code': 0, 'valid': True}} >Thread-28562::DEBUG::2012-12-02 14:32:56,823::task::568::TaskManager.Task::(_updateState) Task=`76291d28-f40d-4120-b61a-086ca21e00f0`::moving from state preparing -> state finished >Thread-28562::DEBUG::2012-12-02 14:32:56,823::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28562::DEBUG::2012-12-02 14:32:56,823::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28562::DEBUG::2012-12-02 14:32:56,824::task::957::TaskManager.Task::(_decref) Task=`76291d28-f40d-4120-b61a-086ca21e00f0`::ref 0 aborting False >Thread-28566::DEBUG::2012-12-02 14:33:02,597::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28566::DEBUG::2012-12-02 14:33:02,597::task::568::TaskManager.Task::(_updateState) Task=`966f7230-42bd-4d3c-91d6-735ced8109eb`::moving from state init -> state preparing >Thread-28566::INFO::2012-12-02 14:33:02,597::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28566::INFO::2012-12-02 14:33:02,597::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28566::DEBUG::2012-12-02 14:33:02,597::task::1151::TaskManager.Task::(prepare) Task=`966f7230-42bd-4d3c-91d6-735ced8109eb`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28566::DEBUG::2012-12-02 14:33:02,598::task::568::TaskManager.Task::(_updateState) Task=`966f7230-42bd-4d3c-91d6-735ced8109eb`::moving from state preparing -> state finished >Thread-28566::DEBUG::2012-12-02 14:33:02,598::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28566::DEBUG::2012-12-02 14:33:02,598::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28566::DEBUG::2012-12-02 14:33:02,598::task::957::TaskManager.Task::(_decref) Task=`966f7230-42bd-4d3c-91d6-735ced8109eb`::ref 0 aborting False >Thread-28567::DEBUG::2012-12-02 14:33:02,601::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28567::DEBUG::2012-12-02 14:33:02,601::task::568::TaskManager.Task::(_updateState) Task=`80ed4429-6dbf-4a85-8eca-97524c5dc44f`::moving from state init -> state preparing >Thread-28567::INFO::2012-12-02 14:33:02,601::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28567::DEBUG::2012-12-02 14:33:02,602::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`50296f8e-d5a9-45ed-b42b-3e458a0a26be`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28567::DEBUG::2012-12-02 14:33:02,602::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28567::DEBUG::2012-12-02 14:33:02,602::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28567::DEBUG::2012-12-02 14:33:02,602::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`50296f8e-d5a9-45ed-b42b-3e458a0a26be`::Granted request >Thread-28567::DEBUG::2012-12-02 14:33:02,602::task::794::TaskManager.Task::(resourceAcquired) Task=`80ed4429-6dbf-4a85-8eca-97524c5dc44f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28567::DEBUG::2012-12-02 14:33:02,602::task::957::TaskManager.Task::(_decref) Task=`80ed4429-6dbf-4a85-8eca-97524c5dc44f`::ref 1 aborting False >Thread-28567::INFO::2012-12-02 14:33:02,608::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28567::DEBUG::2012-12-02 14:33:02,608::task::1151::TaskManager.Task::(prepare) Task=`80ed4429-6dbf-4a85-8eca-97524c5dc44f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28567::DEBUG::2012-12-02 14:33:02,608::task::568::TaskManager.Task::(_updateState) Task=`80ed4429-6dbf-4a85-8eca-97524c5dc44f`::moving from state preparing -> state finished >Thread-28567::DEBUG::2012-12-02 14:33:02,608::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28567::DEBUG::2012-12-02 14:33:02,608::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28567::DEBUG::2012-12-02 14:33:02,608::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28567::DEBUG::2012-12-02 14:33:02,608::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28567::DEBUG::2012-12-02 14:33:02,608::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28567::DEBUG::2012-12-02 14:33:02,608::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28567::DEBUG::2012-12-02 14:33:02,609::task::957::TaskManager.Task::(_decref) Task=`80ed4429-6dbf-4a85-8eca-97524c5dc44f`::ref 0 aborting False >Thread-28570::DEBUG::2012-12-02 14:33:06,936::task::568::TaskManager.Task::(_updateState) Task=`f2456ee5-6f82-4a1f-b55e-79001b94a9df`::moving from state init -> state preparing >Thread-28570::INFO::2012-12-02 14:33:06,936::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28570::INFO::2012-12-02 14:33:06,936::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104629993439', 'lastCheck': 1354451578.61852, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0022120475769', 'lastCheck': 1354451584.675009, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00173187255859', 'lastCheck': 1354451585.971967, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00794196128845', 'lastCheck': 1354451584.691679, 'code': 0, 'valid': True}} >Thread-28570::DEBUG::2012-12-02 14:33:06,936::task::1151::TaskManager.Task::(prepare) Task=`f2456ee5-6f82-4a1f-b55e-79001b94a9df`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104629993439', 'lastCheck': 1354451578.61852, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0022120475769', 'lastCheck': 1354451584.675009, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00173187255859', 'lastCheck': 1354451585.971967, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00794196128845', 'lastCheck': 1354451584.691679, 'code': 0, 'valid': True}} >Thread-28570::DEBUG::2012-12-02 14:33:06,937::task::568::TaskManager.Task::(_updateState) Task=`f2456ee5-6f82-4a1f-b55e-79001b94a9df`::moving from state preparing -> state finished >Thread-28570::DEBUG::2012-12-02 14:33:06,937::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28570::DEBUG::2012-12-02 14:33:06,937::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28570::DEBUG::2012-12-02 14:33:06,937::task::957::TaskManager.Task::(_decref) Task=`f2456ee5-6f82-4a1f-b55e-79001b94a9df`::ref 0 aborting False >Thread-28574::DEBUG::2012-12-02 14:33:12,761::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28574::DEBUG::2012-12-02 14:33:12,761::task::568::TaskManager.Task::(_updateState) Task=`0a6fcc23-60bc-4b3b-82e6-950fa82f99c5`::moving from state init -> state preparing >Thread-28574::INFO::2012-12-02 14:33:12,761::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28574::INFO::2012-12-02 14:33:12,761::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28574::DEBUG::2012-12-02 14:33:12,761::task::1151::TaskManager.Task::(prepare) Task=`0a6fcc23-60bc-4b3b-82e6-950fa82f99c5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28574::DEBUG::2012-12-02 14:33:12,762::task::568::TaskManager.Task::(_updateState) Task=`0a6fcc23-60bc-4b3b-82e6-950fa82f99c5`::moving from state preparing -> state finished >Thread-28574::DEBUG::2012-12-02 14:33:12,762::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28574::DEBUG::2012-12-02 14:33:12,762::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28574::DEBUG::2012-12-02 14:33:12,762::task::957::TaskManager.Task::(_decref) Task=`0a6fcc23-60bc-4b3b-82e6-950fa82f99c5`::ref 0 aborting False >Thread-28575::DEBUG::2012-12-02 14:33:12,765::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28575::DEBUG::2012-12-02 14:33:12,765::task::568::TaskManager.Task::(_updateState) Task=`8bba0620-0dfc-4e84-8fe8-7fc5462923cc`::moving from state init -> state preparing >Thread-28575::INFO::2012-12-02 14:33:12,765::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28575::DEBUG::2012-12-02 14:33:12,765::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`30134077-d9dc-4a2e-9270-faa09f9a348f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28575::DEBUG::2012-12-02 14:33:12,766::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28575::DEBUG::2012-12-02 14:33:12,766::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28575::DEBUG::2012-12-02 14:33:12,766::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`30134077-d9dc-4a2e-9270-faa09f9a348f`::Granted request >Thread-28575::DEBUG::2012-12-02 14:33:12,766::task::794::TaskManager.Task::(resourceAcquired) Task=`8bba0620-0dfc-4e84-8fe8-7fc5462923cc`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28575::DEBUG::2012-12-02 14:33:12,766::task::957::TaskManager.Task::(_decref) Task=`8bba0620-0dfc-4e84-8fe8-7fc5462923cc`::ref 1 aborting False >Thread-28575::INFO::2012-12-02 14:33:12,773::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28575::DEBUG::2012-12-02 14:33:12,773::task::1151::TaskManager.Task::(prepare) Task=`8bba0620-0dfc-4e84-8fe8-7fc5462923cc`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28575::DEBUG::2012-12-02 14:33:12,774::task::568::TaskManager.Task::(_updateState) Task=`8bba0620-0dfc-4e84-8fe8-7fc5462923cc`::moving from state preparing -> state finished >Thread-28575::DEBUG::2012-12-02 14:33:12,774::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28575::DEBUG::2012-12-02 14:33:12,774::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28575::DEBUG::2012-12-02 14:33:12,774::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28575::DEBUG::2012-12-02 14:33:12,774::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28575::DEBUG::2012-12-02 14:33:12,774::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28575::DEBUG::2012-12-02 14:33:12,774::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28575::DEBUG::2012-12-02 14:33:12,775::task::957::TaskManager.Task::(_decref) Task=`8bba0620-0dfc-4e84-8fe8-7fc5462923cc`::ref 0 aborting False >Thread-28578::DEBUG::2012-12-02 14:33:17,131::task::568::TaskManager.Task::(_updateState) Task=`93cb8715-6317-4187-a538-26d818e7d3c3`::moving from state init -> state preparing >Thread-28578::INFO::2012-12-02 14:33:17,131::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28578::INFO::2012-12-02 14:33:17,131::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00619292259216', 'lastCheck': 1354451588.6343379, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00257897377014', 'lastCheck': 1354451594.689065, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00271201133728', 'lastCheck': 1354451595.9837151, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00780892372131', 'lastCheck': 1354451594.7079461, 'code': 0, 'valid': True}} >Thread-28578::DEBUG::2012-12-02 14:33:17,132::task::1151::TaskManager.Task::(prepare) Task=`93cb8715-6317-4187-a538-26d818e7d3c3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00619292259216', 'lastCheck': 1354451588.6343379, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00257897377014', 'lastCheck': 1354451594.689065, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00271201133728', 'lastCheck': 1354451595.9837151, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00780892372131', 'lastCheck': 1354451594.7079461, 'code': 0, 'valid': True}} >Thread-28578::DEBUG::2012-12-02 14:33:17,132::task::568::TaskManager.Task::(_updateState) Task=`93cb8715-6317-4187-a538-26d818e7d3c3`::moving from state preparing -> state finished >Thread-28578::DEBUG::2012-12-02 14:33:17,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28578::DEBUG::2012-12-02 14:33:17,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28578::DEBUG::2012-12-02 14:33:17,132::task::957::TaskManager.Task::(_decref) Task=`93cb8715-6317-4187-a538-26d818e7d3c3`::ref 0 aborting False >Thread-28582::DEBUG::2012-12-02 14:33:22,915::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28582::DEBUG::2012-12-02 14:33:22,915::task::568::TaskManager.Task::(_updateState) Task=`f7d6eed5-dca2-4494-b4c3-6d96bd03c62b`::moving from state init -> state preparing >Thread-28582::INFO::2012-12-02 14:33:22,916::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28582::INFO::2012-12-02 14:33:22,916::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28582::DEBUG::2012-12-02 14:33:22,916::task::1151::TaskManager.Task::(prepare) Task=`f7d6eed5-dca2-4494-b4c3-6d96bd03c62b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28582::DEBUG::2012-12-02 14:33:22,916::task::568::TaskManager.Task::(_updateState) Task=`f7d6eed5-dca2-4494-b4c3-6d96bd03c62b`::moving from state preparing -> state finished >Thread-28582::DEBUG::2012-12-02 14:33:22,916::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28582::DEBUG::2012-12-02 14:33:22,916::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28582::DEBUG::2012-12-02 14:33:22,916::task::957::TaskManager.Task::(_decref) Task=`f7d6eed5-dca2-4494-b4c3-6d96bd03c62b`::ref 0 aborting False >Thread-28583::DEBUG::2012-12-02 14:33:22,919::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28583::DEBUG::2012-12-02 14:33:22,919::task::568::TaskManager.Task::(_updateState) Task=`079a806f-6fc4-4c44-a4a3-f9d8eafc1972`::moving from state init -> state preparing >Thread-28583::INFO::2012-12-02 14:33:22,919::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28583::DEBUG::2012-12-02 14:33:22,920::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`013aec51-d752-4bd1-82b2-42361f2e782a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28583::DEBUG::2012-12-02 14:33:22,920::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28583::DEBUG::2012-12-02 14:33:22,920::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28583::DEBUG::2012-12-02 14:33:22,920::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`013aec51-d752-4bd1-82b2-42361f2e782a`::Granted request >Thread-28583::DEBUG::2012-12-02 14:33:22,920::task::794::TaskManager.Task::(resourceAcquired) Task=`079a806f-6fc4-4c44-a4a3-f9d8eafc1972`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28583::DEBUG::2012-12-02 14:33:22,921::task::957::TaskManager.Task::(_decref) Task=`079a806f-6fc4-4c44-a4a3-f9d8eafc1972`::ref 1 aborting False >Thread-28583::INFO::2012-12-02 14:33:22,926::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28583::DEBUG::2012-12-02 14:33:22,927::task::1151::TaskManager.Task::(prepare) Task=`079a806f-6fc4-4c44-a4a3-f9d8eafc1972`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28583::DEBUG::2012-12-02 14:33:22,927::task::568::TaskManager.Task::(_updateState) Task=`079a806f-6fc4-4c44-a4a3-f9d8eafc1972`::moving from state preparing -> state finished >Thread-28583::DEBUG::2012-12-02 14:33:22,927::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28583::DEBUG::2012-12-02 14:33:22,927::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28583::DEBUG::2012-12-02 14:33:22,927::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28583::DEBUG::2012-12-02 14:33:22,927::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28583::DEBUG::2012-12-02 14:33:22,927::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28583::DEBUG::2012-12-02 14:33:22,927::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28583::DEBUG::2012-12-02 14:33:22,927::task::957::TaskManager.Task::(_decref) Task=`079a806f-6fc4-4c44-a4a3-f9d8eafc1972`::ref 0 aborting False >Thread-28586::DEBUG::2012-12-02 14:33:27,243::task::568::TaskManager.Task::(_updateState) Task=`0a5fc22d-9aa7-459b-8409-802dc4875e71`::moving from state init -> state preparing >Thread-28586::INFO::2012-12-02 14:33:27,244::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28586::INFO::2012-12-02 14:33:27,244::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00883293151855', 'lastCheck': 1354451598.6509349, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00261187553406', 'lastCheck': 1354451604.704298, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191497802734', 'lastCheck': 1354451605.9929521, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00887012481689', 'lastCheck': 1354451604.7256789, 'code': 0, 'valid': True}} >Thread-28586::DEBUG::2012-12-02 14:33:27,244::task::1151::TaskManager.Task::(prepare) Task=`0a5fc22d-9aa7-459b-8409-802dc4875e71`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00883293151855', 'lastCheck': 1354451598.6509349, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00261187553406', 'lastCheck': 1354451604.704298, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191497802734', 'lastCheck': 1354451605.9929521, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00887012481689', 'lastCheck': 1354451604.7256789, 'code': 0, 'valid': True}} >Thread-28586::DEBUG::2012-12-02 14:33:27,244::task::568::TaskManager.Task::(_updateState) Task=`0a5fc22d-9aa7-459b-8409-802dc4875e71`::moving from state preparing -> state finished >Thread-28586::DEBUG::2012-12-02 14:33:27,244::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28586::DEBUG::2012-12-02 14:33:27,244::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28586::DEBUG::2012-12-02 14:33:27,245::task::957::TaskManager.Task::(_decref) Task=`0a5fc22d-9aa7-459b-8409-802dc4875e71`::ref 0 aborting False >Thread-28590::DEBUG::2012-12-02 14:33:33,062::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28590::DEBUG::2012-12-02 14:33:33,062::task::568::TaskManager.Task::(_updateState) Task=`585ee655-0484-406b-a3ec-4790f13bda2b`::moving from state init -> state preparing >Thread-28590::INFO::2012-12-02 14:33:33,062::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28590::INFO::2012-12-02 14:33:33,063::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28590::DEBUG::2012-12-02 14:33:33,063::task::1151::TaskManager.Task::(prepare) Task=`585ee655-0484-406b-a3ec-4790f13bda2b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28590::DEBUG::2012-12-02 14:33:33,063::task::568::TaskManager.Task::(_updateState) Task=`585ee655-0484-406b-a3ec-4790f13bda2b`::moving from state preparing -> state finished >Thread-28590::DEBUG::2012-12-02 14:33:33,063::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28590::DEBUG::2012-12-02 14:33:33,063::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28590::DEBUG::2012-12-02 14:33:33,063::task::957::TaskManager.Task::(_decref) Task=`585ee655-0484-406b-a3ec-4790f13bda2b`::ref 0 aborting False >Thread-28591::DEBUG::2012-12-02 14:33:33,066::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28591::DEBUG::2012-12-02 14:33:33,066::task::568::TaskManager.Task::(_updateState) Task=`b4170b5e-833c-4028-8f75-b67727f919d4`::moving from state init -> state preparing >Thread-28591::INFO::2012-12-02 14:33:33,066::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28591::DEBUG::2012-12-02 14:33:33,067::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`73036f82-fa7a-4bc4-9628-22c78e03e82e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28591::DEBUG::2012-12-02 14:33:33,067::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28591::DEBUG::2012-12-02 14:33:33,067::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28591::DEBUG::2012-12-02 14:33:33,067::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`73036f82-fa7a-4bc4-9628-22c78e03e82e`::Granted request >Thread-28591::DEBUG::2012-12-02 14:33:33,067::task::794::TaskManager.Task::(resourceAcquired) Task=`b4170b5e-833c-4028-8f75-b67727f919d4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28591::DEBUG::2012-12-02 14:33:33,067::task::957::TaskManager.Task::(_decref) Task=`b4170b5e-833c-4028-8f75-b67727f919d4`::ref 1 aborting False >Thread-28591::INFO::2012-12-02 14:33:33,074::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28591::DEBUG::2012-12-02 14:33:33,074::task::1151::TaskManager.Task::(prepare) Task=`b4170b5e-833c-4028-8f75-b67727f919d4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28591::DEBUG::2012-12-02 14:33:33,074::task::568::TaskManager.Task::(_updateState) Task=`b4170b5e-833c-4028-8f75-b67727f919d4`::moving from state preparing -> state finished >Thread-28591::DEBUG::2012-12-02 14:33:33,074::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28591::DEBUG::2012-12-02 14:33:33,074::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28591::DEBUG::2012-12-02 14:33:33,075::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28591::DEBUG::2012-12-02 14:33:33,075::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28591::DEBUG::2012-12-02 14:33:33,075::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28591::DEBUG::2012-12-02 14:33:33,075::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28591::DEBUG::2012-12-02 14:33:33,075::task::957::TaskManager.Task::(_decref) Task=`b4170b5e-833c-4028-8f75-b67727f919d4`::ref 0 aborting False >Thread-28594::DEBUG::2012-12-02 14:33:37,352::task::568::TaskManager.Task::(_updateState) Task=`5eb2cfb7-5009-40fc-9e5a-0e70224fd13e`::moving from state init -> state preparing >Thread-28594::INFO::2012-12-02 14:33:37,352::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28594::INFO::2012-12-02 14:33:37,353::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00881099700928', 'lastCheck': 1354451608.6686561, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00288605690002', 'lastCheck': 1354451614.7163551, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00285887718201', 'lastCheck': 1354451616.0060239, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00690889358521', 'lastCheck': 1354451614.7415769, 'code': 0, 'valid': True}} >Thread-28594::DEBUG::2012-12-02 14:33:37,353::task::1151::TaskManager.Task::(prepare) Task=`5eb2cfb7-5009-40fc-9e5a-0e70224fd13e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00881099700928', 'lastCheck': 1354451608.6686561, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00288605690002', 'lastCheck': 1354451614.7163551, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00285887718201', 'lastCheck': 1354451616.0060239, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00690889358521', 'lastCheck': 1354451614.7415769, 'code': 0, 'valid': True}} >Thread-28594::DEBUG::2012-12-02 14:33:37,353::task::568::TaskManager.Task::(_updateState) Task=`5eb2cfb7-5009-40fc-9e5a-0e70224fd13e`::moving from state preparing -> state finished >Thread-28594::DEBUG::2012-12-02 14:33:37,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28594::DEBUG::2012-12-02 14:33:37,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28594::DEBUG::2012-12-02 14:33:37,353::task::957::TaskManager.Task::(_decref) Task=`5eb2cfb7-5009-40fc-9e5a-0e70224fd13e`::ref 0 aborting False >Thread-28598::DEBUG::2012-12-02 14:33:43,225::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28598::DEBUG::2012-12-02 14:33:43,225::task::568::TaskManager.Task::(_updateState) Task=`77b6f67d-e644-4cc5-86ea-74bc11777bf7`::moving from state init -> state preparing >Thread-28598::INFO::2012-12-02 14:33:43,225::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28598::INFO::2012-12-02 14:33:43,225::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28598::DEBUG::2012-12-02 14:33:43,226::task::1151::TaskManager.Task::(prepare) Task=`77b6f67d-e644-4cc5-86ea-74bc11777bf7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28598::DEBUG::2012-12-02 14:33:43,226::task::568::TaskManager.Task::(_updateState) Task=`77b6f67d-e644-4cc5-86ea-74bc11777bf7`::moving from state preparing -> state finished >Thread-28598::DEBUG::2012-12-02 14:33:43,226::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28598::DEBUG::2012-12-02 14:33:43,226::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28598::DEBUG::2012-12-02 14:33:43,226::task::957::TaskManager.Task::(_decref) Task=`77b6f67d-e644-4cc5-86ea-74bc11777bf7`::ref 0 aborting False >Thread-28599::DEBUG::2012-12-02 14:33:43,229::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28599::DEBUG::2012-12-02 14:33:43,229::task::568::TaskManager.Task::(_updateState) Task=`bdf0fbae-463b-4e76-9be9-bb724fff59e1`::moving from state init -> state preparing >Thread-28599::INFO::2012-12-02 14:33:43,229::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28599::DEBUG::2012-12-02 14:33:43,229::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4dd0455b-3ff3-48a1-af4e-5e7a227a4795`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28599::DEBUG::2012-12-02 14:33:43,230::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28599::DEBUG::2012-12-02 14:33:43,230::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28599::DEBUG::2012-12-02 14:33:43,230::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4dd0455b-3ff3-48a1-af4e-5e7a227a4795`::Granted request >Thread-28599::DEBUG::2012-12-02 14:33:43,230::task::794::TaskManager.Task::(resourceAcquired) Task=`bdf0fbae-463b-4e76-9be9-bb724fff59e1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28599::DEBUG::2012-12-02 14:33:43,230::task::957::TaskManager.Task::(_decref) Task=`bdf0fbae-463b-4e76-9be9-bb724fff59e1`::ref 1 aborting False >Thread-28599::INFO::2012-12-02 14:33:43,236::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28599::DEBUG::2012-12-02 14:33:43,236::task::1151::TaskManager.Task::(prepare) Task=`bdf0fbae-463b-4e76-9be9-bb724fff59e1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28599::DEBUG::2012-12-02 14:33:43,236::task::568::TaskManager.Task::(_updateState) Task=`bdf0fbae-463b-4e76-9be9-bb724fff59e1`::moving from state preparing -> state finished >Thread-28599::DEBUG::2012-12-02 14:33:43,237::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28599::DEBUG::2012-12-02 14:33:43,237::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28599::DEBUG::2012-12-02 14:33:43,237::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28599::DEBUG::2012-12-02 14:33:43,237::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28599::DEBUG::2012-12-02 14:33:43,237::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28599::DEBUG::2012-12-02 14:33:43,237::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28599::DEBUG::2012-12-02 14:33:43,237::task::957::TaskManager.Task::(_decref) Task=`bdf0fbae-463b-4e76-9be9-bb724fff59e1`::ref 0 aborting False >Thread-28602::DEBUG::2012-12-02 14:33:47,440::task::568::TaskManager.Task::(_updateState) Task=`15a2221c-1170-421f-9cf1-de6b2835db1a`::moving from state init -> state preparing >Thread-28602::INFO::2012-12-02 14:33:47,440::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28602::INFO::2012-12-02 14:33:47,441::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00748991966248', 'lastCheck': 1354451618.6889241, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00240802764893', 'lastCheck': 1354451624.731451, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287985801697', 'lastCheck': 1354451626.019341, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00775599479675', 'lastCheck': 1354451624.7604451, 'code': 0, 'valid': True}} >Thread-28602::DEBUG::2012-12-02 14:33:47,441::task::1151::TaskManager.Task::(prepare) Task=`15a2221c-1170-421f-9cf1-de6b2835db1a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00748991966248', 'lastCheck': 1354451618.6889241, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00240802764893', 'lastCheck': 1354451624.731451, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00287985801697', 'lastCheck': 1354451626.019341, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00775599479675', 'lastCheck': 1354451624.7604451, 'code': 0, 'valid': True}} >Thread-28602::DEBUG::2012-12-02 14:33:47,441::task::568::TaskManager.Task::(_updateState) Task=`15a2221c-1170-421f-9cf1-de6b2835db1a`::moving from state preparing -> state finished >Thread-28602::DEBUG::2012-12-02 14:33:47,441::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28602::DEBUG::2012-12-02 14:33:47,441::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28602::DEBUG::2012-12-02 14:33:47,441::task::957::TaskManager.Task::(_decref) Task=`15a2221c-1170-421f-9cf1-de6b2835db1a`::ref 0 aborting False >Thread-28606::DEBUG::2012-12-02 14:33:53,338::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28606::DEBUG::2012-12-02 14:33:53,338::task::568::TaskManager.Task::(_updateState) Task=`49c3e903-90f3-4620-af9d-f45477da289f`::moving from state init -> state preparing >Thread-28606::INFO::2012-12-02 14:33:53,338::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28606::INFO::2012-12-02 14:33:53,339::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28606::DEBUG::2012-12-02 14:33:53,339::task::1151::TaskManager.Task::(prepare) Task=`49c3e903-90f3-4620-af9d-f45477da289f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28606::DEBUG::2012-12-02 14:33:53,339::task::568::TaskManager.Task::(_updateState) Task=`49c3e903-90f3-4620-af9d-f45477da289f`::moving from state preparing -> state finished >Thread-28606::DEBUG::2012-12-02 14:33:53,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28606::DEBUG::2012-12-02 14:33:53,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28606::DEBUG::2012-12-02 14:33:53,339::task::957::TaskManager.Task::(_decref) Task=`49c3e903-90f3-4620-af9d-f45477da289f`::ref 0 aborting False >Thread-28607::DEBUG::2012-12-02 14:33:53,342::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28607::DEBUG::2012-12-02 14:33:53,342::task::568::TaskManager.Task::(_updateState) Task=`add4a148-77da-47db-97a5-91993879f2f8`::moving from state init -> state preparing >Thread-28607::INFO::2012-12-02 14:33:53,342::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28607::DEBUG::2012-12-02 14:33:53,342::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cb93accb-fa4e-44bf-b562-68d1b0396f10`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28607::DEBUG::2012-12-02 14:33:53,343::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28607::DEBUG::2012-12-02 14:33:53,343::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28607::DEBUG::2012-12-02 14:33:53,343::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cb93accb-fa4e-44bf-b562-68d1b0396f10`::Granted request >Thread-28607::DEBUG::2012-12-02 14:33:53,343::task::794::TaskManager.Task::(resourceAcquired) Task=`add4a148-77da-47db-97a5-91993879f2f8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28607::DEBUG::2012-12-02 14:33:53,343::task::957::TaskManager.Task::(_decref) Task=`add4a148-77da-47db-97a5-91993879f2f8`::ref 1 aborting False >Thread-28607::INFO::2012-12-02 14:33:53,348::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28607::DEBUG::2012-12-02 14:33:53,348::task::1151::TaskManager.Task::(prepare) Task=`add4a148-77da-47db-97a5-91993879f2f8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28607::DEBUG::2012-12-02 14:33:53,349::task::568::TaskManager.Task::(_updateState) Task=`add4a148-77da-47db-97a5-91993879f2f8`::moving from state preparing -> state finished >Thread-28607::DEBUG::2012-12-02 14:33:53,349::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28607::DEBUG::2012-12-02 14:33:53,349::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28607::DEBUG::2012-12-02 14:33:53,349::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28607::DEBUG::2012-12-02 14:33:53,349::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28607::DEBUG::2012-12-02 14:33:53,349::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28607::DEBUG::2012-12-02 14:33:53,349::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28607::DEBUG::2012-12-02 14:33:53,349::task::957::TaskManager.Task::(_decref) Task=`add4a148-77da-47db-97a5-91993879f2f8`::ref 0 aborting False >Thread-28610::DEBUG::2012-12-02 14:33:57,502::task::568::TaskManager.Task::(_updateState) Task=`b89fc92a-d57c-4710-aa40-b500341c54b0`::moving from state init -> state preparing >Thread-28610::INFO::2012-12-02 14:33:57,502::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28610::INFO::2012-12-02 14:33:57,503::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00767397880554', 'lastCheck': 1354451628.7067499, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00200986862183', 'lastCheck': 1354451634.7428801, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00134897232056', 'lastCheck': 1354451636.0302081, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00774502754211', 'lastCheck': 1354451634.776752, 'code': 0, 'valid': True}} >Thread-28610::DEBUG::2012-12-02 14:33:57,503::task::1151::TaskManager.Task::(prepare) Task=`b89fc92a-d57c-4710-aa40-b500341c54b0`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00767397880554', 'lastCheck': 1354451628.7067499, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00200986862183', 'lastCheck': 1354451634.7428801, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00134897232056', 'lastCheck': 1354451636.0302081, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00774502754211', 'lastCheck': 1354451634.776752, 'code': 0, 'valid': True}} >Thread-28610::DEBUG::2012-12-02 14:33:57,503::task::568::TaskManager.Task::(_updateState) Task=`b89fc92a-d57c-4710-aa40-b500341c54b0`::moving from state preparing -> state finished >Thread-28610::DEBUG::2012-12-02 14:33:57,503::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28610::DEBUG::2012-12-02 14:33:57,503::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28610::DEBUG::2012-12-02 14:33:57,503::task::957::TaskManager.Task::(_decref) Task=`b89fc92a-d57c-4710-aa40-b500341c54b0`::ref 0 aborting False >Thread-28614::DEBUG::2012-12-02 14:34:03,468::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28614::DEBUG::2012-12-02 14:34:03,469::task::568::TaskManager.Task::(_updateState) Task=`e74fc840-1007-4dea-b662-ca48d5366773`::moving from state init -> state preparing >Thread-28614::INFO::2012-12-02 14:34:03,469::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28614::INFO::2012-12-02 14:34:03,469::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28614::DEBUG::2012-12-02 14:34:03,469::task::1151::TaskManager.Task::(prepare) Task=`e74fc840-1007-4dea-b662-ca48d5366773`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28614::DEBUG::2012-12-02 14:34:03,469::task::568::TaskManager.Task::(_updateState) Task=`e74fc840-1007-4dea-b662-ca48d5366773`::moving from state preparing -> state finished >Thread-28614::DEBUG::2012-12-02 14:34:03,469::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28614::DEBUG::2012-12-02 14:34:03,470::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28614::DEBUG::2012-12-02 14:34:03,470::task::957::TaskManager.Task::(_decref) Task=`e74fc840-1007-4dea-b662-ca48d5366773`::ref 0 aborting False >Thread-28615::DEBUG::2012-12-02 14:34:03,472::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28615::DEBUG::2012-12-02 14:34:03,472::task::568::TaskManager.Task::(_updateState) Task=`b8b0518d-dcd3-4663-a8e3-6dd525a527a1`::moving from state init -> state preparing >Thread-28615::INFO::2012-12-02 14:34:03,473::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28615::DEBUG::2012-12-02 14:34:03,473::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`aba85713-896a-4a3b-b3e3-c9ae3302c4b3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28615::DEBUG::2012-12-02 14:34:03,473::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28615::DEBUG::2012-12-02 14:34:03,473::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28615::DEBUG::2012-12-02 14:34:03,473::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`aba85713-896a-4a3b-b3e3-c9ae3302c4b3`::Granted request >Thread-28615::DEBUG::2012-12-02 14:34:03,474::task::794::TaskManager.Task::(resourceAcquired) Task=`b8b0518d-dcd3-4663-a8e3-6dd525a527a1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28615::DEBUG::2012-12-02 14:34:03,474::task::957::TaskManager.Task::(_decref) Task=`b8b0518d-dcd3-4663-a8e3-6dd525a527a1`::ref 1 aborting False >Thread-28615::INFO::2012-12-02 14:34:03,479::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28615::DEBUG::2012-12-02 14:34:03,479::task::1151::TaskManager.Task::(prepare) Task=`b8b0518d-dcd3-4663-a8e3-6dd525a527a1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28615::DEBUG::2012-12-02 14:34:03,479::task::568::TaskManager.Task::(_updateState) Task=`b8b0518d-dcd3-4663-a8e3-6dd525a527a1`::moving from state preparing -> state finished >Thread-28615::DEBUG::2012-12-02 14:34:03,479::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28615::DEBUG::2012-12-02 14:34:03,479::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28615::DEBUG::2012-12-02 14:34:03,479::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28615::DEBUG::2012-12-02 14:34:03,479::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28615::DEBUG::2012-12-02 14:34:03,480::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28615::DEBUG::2012-12-02 14:34:03,480::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28615::DEBUG::2012-12-02 14:34:03,480::task::957::TaskManager.Task::(_decref) Task=`b8b0518d-dcd3-4663-a8e3-6dd525a527a1`::ref 0 aborting False >Thread-28618::DEBUG::2012-12-02 14:34:07,625::task::568::TaskManager.Task::(_updateState) Task=`0d90c1bc-4a9b-4b30-bc61-a29a2a95c7a0`::moving from state init -> state preparing >Thread-28618::INFO::2012-12-02 14:34:07,625::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28618::INFO::2012-12-02 14:34:07,626::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0107569694519', 'lastCheck': 1354451638.72791, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00539398193359', 'lastCheck': 1354451644.785964, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00167512893677', 'lastCheck': 1354451646.0403709, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00900387763977', 'lastCheck': 1354451644.797859, 'code': 0, 'valid': True}} >Thread-28618::DEBUG::2012-12-02 14:34:07,626::task::1151::TaskManager.Task::(prepare) Task=`0d90c1bc-4a9b-4b30-bc61-a29a2a95c7a0`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0107569694519', 'lastCheck': 1354451638.72791, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00539398193359', 'lastCheck': 1354451644.785964, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00167512893677', 'lastCheck': 1354451646.0403709, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00900387763977', 'lastCheck': 1354451644.797859, 'code': 0, 'valid': True}} >Thread-28618::DEBUG::2012-12-02 14:34:07,626::task::568::TaskManager.Task::(_updateState) Task=`0d90c1bc-4a9b-4b30-bc61-a29a2a95c7a0`::moving from state preparing -> state finished >Thread-28618::DEBUG::2012-12-02 14:34:07,626::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28618::DEBUG::2012-12-02 14:34:07,626::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28618::DEBUG::2012-12-02 14:34:07,626::task::957::TaskManager.Task::(_decref) Task=`0d90c1bc-4a9b-4b30-bc61-a29a2a95c7a0`::ref 0 aborting False >Thread-28622::DEBUG::2012-12-02 14:34:13,573::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28622::DEBUG::2012-12-02 14:34:13,573::task::568::TaskManager.Task::(_updateState) Task=`1bf1f573-6d87-49ec-ae3d-c74c1cf253ab`::moving from state init -> state preparing >Thread-28622::INFO::2012-12-02 14:34:13,573::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28622::INFO::2012-12-02 14:34:13,573::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28622::DEBUG::2012-12-02 14:34:13,574::task::1151::TaskManager.Task::(prepare) Task=`1bf1f573-6d87-49ec-ae3d-c74c1cf253ab`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28622::DEBUG::2012-12-02 14:34:13,574::task::568::TaskManager.Task::(_updateState) Task=`1bf1f573-6d87-49ec-ae3d-c74c1cf253ab`::moving from state preparing -> state finished >Thread-28622::DEBUG::2012-12-02 14:34:13,574::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28622::DEBUG::2012-12-02 14:34:13,574::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28622::DEBUG::2012-12-02 14:34:13,574::task::957::TaskManager.Task::(_decref) Task=`1bf1f573-6d87-49ec-ae3d-c74c1cf253ab`::ref 0 aborting False >Thread-28623::DEBUG::2012-12-02 14:34:13,577::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28623::DEBUG::2012-12-02 14:34:13,577::task::568::TaskManager.Task::(_updateState) Task=`497600bb-6d6c-4d57-8bda-b15b766ca9f1`::moving from state init -> state preparing >Thread-28623::INFO::2012-12-02 14:34:13,577::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28623::DEBUG::2012-12-02 14:34:13,577::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cb1ca6ca-43b3-46fa-8d93-2cb6364d5ccc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28623::DEBUG::2012-12-02 14:34:13,577::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28623::DEBUG::2012-12-02 14:34:13,578::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28623::DEBUG::2012-12-02 14:34:13,578::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cb1ca6ca-43b3-46fa-8d93-2cb6364d5ccc`::Granted request >Thread-28623::DEBUG::2012-12-02 14:34:13,578::task::794::TaskManager.Task::(resourceAcquired) Task=`497600bb-6d6c-4d57-8bda-b15b766ca9f1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28623::DEBUG::2012-12-02 14:34:13,578::task::957::TaskManager.Task::(_decref) Task=`497600bb-6d6c-4d57-8bda-b15b766ca9f1`::ref 1 aborting False >Thread-28623::INFO::2012-12-02 14:34:13,582::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28623::DEBUG::2012-12-02 14:34:13,582::task::1151::TaskManager.Task::(prepare) Task=`497600bb-6d6c-4d57-8bda-b15b766ca9f1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28623::DEBUG::2012-12-02 14:34:13,583::task::568::TaskManager.Task::(_updateState) Task=`497600bb-6d6c-4d57-8bda-b15b766ca9f1`::moving from state preparing -> state finished >Thread-28623::DEBUG::2012-12-02 14:34:13,583::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28623::DEBUG::2012-12-02 14:34:13,583::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28623::DEBUG::2012-12-02 14:34:13,583::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28623::DEBUG::2012-12-02 14:34:13,583::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28623::DEBUG::2012-12-02 14:34:13,583::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28623::DEBUG::2012-12-02 14:34:13,583::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28623::DEBUG::2012-12-02 14:34:13,583::task::957::TaskManager.Task::(_decref) Task=`497600bb-6d6c-4d57-8bda-b15b766ca9f1`::ref 0 aborting False >Thread-28626::DEBUG::2012-12-02 14:34:17,739::task::568::TaskManager.Task::(_updateState) Task=`5e551a2a-7ce1-4e5a-8b26-88876a8f2ce1`::moving from state init -> state preparing >Thread-28626::INFO::2012-12-02 14:34:17,739::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28626::INFO::2012-12-02 14:34:17,739::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00786399841309', 'lastCheck': 1354451648.7451179, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00468182563782', 'lastCheck': 1354451654.8049409, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00233888626099', 'lastCheck': 1354451656.0539711, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072078704834', 'lastCheck': 1354451654.8139739, 'code': 0, 'valid': True}} >Thread-28626::DEBUG::2012-12-02 14:34:17,739::task::1151::TaskManager.Task::(prepare) Task=`5e551a2a-7ce1-4e5a-8b26-88876a8f2ce1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00786399841309', 'lastCheck': 1354451648.7451179, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00468182563782', 'lastCheck': 1354451654.8049409, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00233888626099', 'lastCheck': 1354451656.0539711, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072078704834', 'lastCheck': 1354451654.8139739, 'code': 0, 'valid': True}} >Thread-28626::DEBUG::2012-12-02 14:34:17,740::task::568::TaskManager.Task::(_updateState) Task=`5e551a2a-7ce1-4e5a-8b26-88876a8f2ce1`::moving from state preparing -> state finished >Thread-28626::DEBUG::2012-12-02 14:34:17,740::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28626::DEBUG::2012-12-02 14:34:17,740::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28626::DEBUG::2012-12-02 14:34:17,740::task::957::TaskManager.Task::(_decref) Task=`5e551a2a-7ce1-4e5a-8b26-88876a8f2ce1`::ref 0 aborting False >Thread-28630::DEBUG::2012-12-02 14:34:23,686::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28630::DEBUG::2012-12-02 14:34:23,686::task::568::TaskManager.Task::(_updateState) Task=`1b7e4540-7a24-45cb-a796-1fa87f36187f`::moving from state init -> state preparing >Thread-28630::INFO::2012-12-02 14:34:23,686::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28630::INFO::2012-12-02 14:34:23,687::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28630::DEBUG::2012-12-02 14:34:23,687::task::1151::TaskManager.Task::(prepare) Task=`1b7e4540-7a24-45cb-a796-1fa87f36187f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28630::DEBUG::2012-12-02 14:34:23,687::task::568::TaskManager.Task::(_updateState) Task=`1b7e4540-7a24-45cb-a796-1fa87f36187f`::moving from state preparing -> state finished >Thread-28630::DEBUG::2012-12-02 14:34:23,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28630::DEBUG::2012-12-02 14:34:23,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28630::DEBUG::2012-12-02 14:34:23,687::task::957::TaskManager.Task::(_decref) Task=`1b7e4540-7a24-45cb-a796-1fa87f36187f`::ref 0 aborting False >Thread-28631::DEBUG::2012-12-02 14:34:23,690::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28631::DEBUG::2012-12-02 14:34:23,690::task::568::TaskManager.Task::(_updateState) Task=`3d340948-577f-4685-becf-5b967ddcb2ca`::moving from state init -> state preparing >Thread-28631::INFO::2012-12-02 14:34:23,690::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28631::DEBUG::2012-12-02 14:34:23,690::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e69c627a-86d4-48c2-a437-8d868b42b58b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28631::DEBUG::2012-12-02 14:34:23,691::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28631::DEBUG::2012-12-02 14:34:23,691::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28631::DEBUG::2012-12-02 14:34:23,691::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e69c627a-86d4-48c2-a437-8d868b42b58b`::Granted request >Thread-28631::DEBUG::2012-12-02 14:34:23,691::task::794::TaskManager.Task::(resourceAcquired) Task=`3d340948-577f-4685-becf-5b967ddcb2ca`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28631::DEBUG::2012-12-02 14:34:23,691::task::957::TaskManager.Task::(_decref) Task=`3d340948-577f-4685-becf-5b967ddcb2ca`::ref 1 aborting False >Thread-28631::INFO::2012-12-02 14:34:23,697::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28631::DEBUG::2012-12-02 14:34:23,697::task::1151::TaskManager.Task::(prepare) Task=`3d340948-577f-4685-becf-5b967ddcb2ca`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28631::DEBUG::2012-12-02 14:34:23,697::task::568::TaskManager.Task::(_updateState) Task=`3d340948-577f-4685-becf-5b967ddcb2ca`::moving from state preparing -> state finished >Thread-28631::DEBUG::2012-12-02 14:34:23,697::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28631::DEBUG::2012-12-02 14:34:23,698::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28631::DEBUG::2012-12-02 14:34:23,698::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28631::DEBUG::2012-12-02 14:34:23,698::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28631::DEBUG::2012-12-02 14:34:23,698::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28631::DEBUG::2012-12-02 14:34:23,698::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28631::DEBUG::2012-12-02 14:34:23,698::task::957::TaskManager.Task::(_decref) Task=`3d340948-577f-4685-becf-5b967ddcb2ca`::ref 0 aborting False >Thread-28634::DEBUG::2012-12-02 14:34:27,887::task::568::TaskManager.Task::(_updateState) Task=`3d7180a5-99ed-43de-a7cf-85f55f608f63`::moving from state init -> state preparing >Thread-28634::INFO::2012-12-02 14:34:27,887::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28634::INFO::2012-12-02 14:34:27,888::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00722503662109', 'lastCheck': 1354451658.7620499, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180792808533', 'lastCheck': 1354451664.825839, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00229096412659', 'lastCheck': 1354451666.0661681, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0083909034729', 'lastCheck': 1354451664.8324299, 'code': 0, 'valid': True}} >Thread-28634::DEBUG::2012-12-02 14:34:27,888::task::1151::TaskManager.Task::(prepare) Task=`3d7180a5-99ed-43de-a7cf-85f55f608f63`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00722503662109', 'lastCheck': 1354451658.7620499, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180792808533', 'lastCheck': 1354451664.825839, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00229096412659', 'lastCheck': 1354451666.0661681, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0083909034729', 'lastCheck': 1354451664.8324299, 'code': 0, 'valid': True}} >Thread-28634::DEBUG::2012-12-02 14:34:27,888::task::568::TaskManager.Task::(_updateState) Task=`3d7180a5-99ed-43de-a7cf-85f55f608f63`::moving from state preparing -> state finished >Thread-28634::DEBUG::2012-12-02 14:34:27,888::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28634::DEBUG::2012-12-02 14:34:27,888::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28634::DEBUG::2012-12-02 14:34:27,888::task::957::TaskManager.Task::(_decref) Task=`3d7180a5-99ed-43de-a7cf-85f55f608f63`::ref 0 aborting False >Thread-28638::DEBUG::2012-12-02 14:34:33,841::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28638::DEBUG::2012-12-02 14:34:33,841::task::568::TaskManager.Task::(_updateState) Task=`9f59f52c-e833-4ddb-809c-948f16d14e73`::moving from state init -> state preparing >Thread-28638::INFO::2012-12-02 14:34:33,841::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28638::INFO::2012-12-02 14:34:33,841::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28638::DEBUG::2012-12-02 14:34:33,842::task::1151::TaskManager.Task::(prepare) Task=`9f59f52c-e833-4ddb-809c-948f16d14e73`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28638::DEBUG::2012-12-02 14:34:33,842::task::568::TaskManager.Task::(_updateState) Task=`9f59f52c-e833-4ddb-809c-948f16d14e73`::moving from state preparing -> state finished >Thread-28638::DEBUG::2012-12-02 14:34:33,842::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28638::DEBUG::2012-12-02 14:34:33,842::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28638::DEBUG::2012-12-02 14:34:33,842::task::957::TaskManager.Task::(_decref) Task=`9f59f52c-e833-4ddb-809c-948f16d14e73`::ref 0 aborting False >Thread-28639::DEBUG::2012-12-02 14:34:33,845::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28639::DEBUG::2012-12-02 14:34:33,845::task::568::TaskManager.Task::(_updateState) Task=`b165fc5b-195f-4928-9a5d-0f8b4cca33be`::moving from state init -> state preparing >Thread-28639::INFO::2012-12-02 14:34:33,845::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28639::DEBUG::2012-12-02 14:34:33,846::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`378fd08c-4e9a-470c-a4ac-c59e410a9308`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28639::DEBUG::2012-12-02 14:34:33,846::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28639::DEBUG::2012-12-02 14:34:33,846::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28639::DEBUG::2012-12-02 14:34:33,846::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`378fd08c-4e9a-470c-a4ac-c59e410a9308`::Granted request >Thread-28639::DEBUG::2012-12-02 14:34:33,846::task::794::TaskManager.Task::(resourceAcquired) Task=`b165fc5b-195f-4928-9a5d-0f8b4cca33be`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28639::DEBUG::2012-12-02 14:34:33,846::task::957::TaskManager.Task::(_decref) Task=`b165fc5b-195f-4928-9a5d-0f8b4cca33be`::ref 1 aborting False >Thread-28639::INFO::2012-12-02 14:34:33,853::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28639::DEBUG::2012-12-02 14:34:33,853::task::1151::TaskManager.Task::(prepare) Task=`b165fc5b-195f-4928-9a5d-0f8b4cca33be`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28639::DEBUG::2012-12-02 14:34:33,853::task::568::TaskManager.Task::(_updateState) Task=`b165fc5b-195f-4928-9a5d-0f8b4cca33be`::moving from state preparing -> state finished >Thread-28639::DEBUG::2012-12-02 14:34:33,853::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28639::DEBUG::2012-12-02 14:34:33,853::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28639::DEBUG::2012-12-02 14:34:33,854::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28639::DEBUG::2012-12-02 14:34:33,854::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28639::DEBUG::2012-12-02 14:34:33,854::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28639::DEBUG::2012-12-02 14:34:33,854::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28639::DEBUG::2012-12-02 14:34:33,854::task::957::TaskManager.Task::(_decref) Task=`b165fc5b-195f-4928-9a5d-0f8b4cca33be`::ref 0 aborting False >Thread-28642::DEBUG::2012-12-02 14:34:38,008::task::568::TaskManager.Task::(_updateState) Task=`53a94df0-dca6-400e-8636-eeae0de8f14e`::moving from state init -> state preparing >Thread-28642::INFO::2012-12-02 14:34:38,009::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28642::INFO::2012-12-02 14:34:38,009::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0128710269928', 'lastCheck': 1354451668.783653, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0039050579071', 'lastCheck': 1354451674.8408389, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00292205810547', 'lastCheck': 1354451676.0788, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00784707069397', 'lastCheck': 1354451674.848742, 'code': 0, 'valid': True}} >Thread-28642::DEBUG::2012-12-02 14:34:38,009::task::1151::TaskManager.Task::(prepare) Task=`53a94df0-dca6-400e-8636-eeae0de8f14e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0128710269928', 'lastCheck': 1354451668.783653, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0039050579071', 'lastCheck': 1354451674.8408389, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00292205810547', 'lastCheck': 1354451676.0788, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00784707069397', 'lastCheck': 1354451674.848742, 'code': 0, 'valid': True}} >Thread-28642::DEBUG::2012-12-02 14:34:38,009::task::568::TaskManager.Task::(_updateState) Task=`53a94df0-dca6-400e-8636-eeae0de8f14e`::moving from state preparing -> state finished >Thread-28642::DEBUG::2012-12-02 14:34:38,009::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28642::DEBUG::2012-12-02 14:34:38,009::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28642::DEBUG::2012-12-02 14:34:38,010::task::957::TaskManager.Task::(_decref) Task=`53a94df0-dca6-400e-8636-eeae0de8f14e`::ref 0 aborting False >Thread-28646::DEBUG::2012-12-02 14:34:44,037::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28646::DEBUG::2012-12-02 14:34:44,037::task::568::TaskManager.Task::(_updateState) Task=`28f4908e-2b0d-42a1-b78e-222d38a18af6`::moving from state init -> state preparing >Thread-28646::INFO::2012-12-02 14:34:44,038::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28646::INFO::2012-12-02 14:34:44,038::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28646::DEBUG::2012-12-02 14:34:44,038::task::1151::TaskManager.Task::(prepare) Task=`28f4908e-2b0d-42a1-b78e-222d38a18af6`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28646::DEBUG::2012-12-02 14:34:44,038::task::568::TaskManager.Task::(_updateState) Task=`28f4908e-2b0d-42a1-b78e-222d38a18af6`::moving from state preparing -> state finished >Thread-28646::DEBUG::2012-12-02 14:34:44,038::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28646::DEBUG::2012-12-02 14:34:44,038::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28646::DEBUG::2012-12-02 14:34:44,038::task::957::TaskManager.Task::(_decref) Task=`28f4908e-2b0d-42a1-b78e-222d38a18af6`::ref 0 aborting False >Thread-28647::DEBUG::2012-12-02 14:34:44,041::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28647::DEBUG::2012-12-02 14:34:44,041::task::568::TaskManager.Task::(_updateState) Task=`8e869898-8584-45a3-8b7d-f3cefaa5b80f`::moving from state init -> state preparing >Thread-28647::INFO::2012-12-02 14:34:44,041::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28647::DEBUG::2012-12-02 14:34:44,042::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`479e4379-057c-4d34-9bd3-79efc22e7aa7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28647::DEBUG::2012-12-02 14:34:44,042::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28647::DEBUG::2012-12-02 14:34:44,042::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28647::DEBUG::2012-12-02 14:34:44,042::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`479e4379-057c-4d34-9bd3-79efc22e7aa7`::Granted request >Thread-28647::DEBUG::2012-12-02 14:34:44,043::task::794::TaskManager.Task::(resourceAcquired) Task=`8e869898-8584-45a3-8b7d-f3cefaa5b80f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28647::DEBUG::2012-12-02 14:34:44,043::task::957::TaskManager.Task::(_decref) Task=`8e869898-8584-45a3-8b7d-f3cefaa5b80f`::ref 1 aborting False >Thread-28647::INFO::2012-12-02 14:34:44,048::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28647::DEBUG::2012-12-02 14:34:44,048::task::1151::TaskManager.Task::(prepare) Task=`8e869898-8584-45a3-8b7d-f3cefaa5b80f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28647::DEBUG::2012-12-02 14:34:44,049::task::568::TaskManager.Task::(_updateState) Task=`8e869898-8584-45a3-8b7d-f3cefaa5b80f`::moving from state preparing -> state finished >Thread-28647::DEBUG::2012-12-02 14:34:44,049::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28647::DEBUG::2012-12-02 14:34:44,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28647::DEBUG::2012-12-02 14:34:44,049::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28647::DEBUG::2012-12-02 14:34:44,049::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28647::DEBUG::2012-12-02 14:34:44,049::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28647::DEBUG::2012-12-02 14:34:44,049::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28647::DEBUG::2012-12-02 14:34:44,049::task::957::TaskManager.Task::(_decref) Task=`8e869898-8584-45a3-8b7d-f3cefaa5b80f`::ref 0 aborting False >Thread-28650::DEBUG::2012-12-02 14:34:48,103::task::568::TaskManager.Task::(_updateState) Task=`5de4aa85-93f6-40db-b2a4-3682dc639ebf`::moving from state init -> state preparing >Thread-28650::INFO::2012-12-02 14:34:48,104::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28650::INFO::2012-12-02 14:34:48,104::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0107400417328', 'lastCheck': 1354451678.8046451, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00461196899414', 'lastCheck': 1354451684.8597741, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00205111503601', 'lastCheck': 1354451686.0895, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00749897956848', 'lastCheck': 1354451684.86836, 'code': 0, 'valid': True}} >Thread-28650::DEBUG::2012-12-02 14:34:48,104::task::1151::TaskManager.Task::(prepare) Task=`5de4aa85-93f6-40db-b2a4-3682dc639ebf`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0107400417328', 'lastCheck': 1354451678.8046451, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00461196899414', 'lastCheck': 1354451684.8597741, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00205111503601', 'lastCheck': 1354451686.0895, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00749897956848', 'lastCheck': 1354451684.86836, 'code': 0, 'valid': True}} >Thread-28650::DEBUG::2012-12-02 14:34:48,104::task::568::TaskManager.Task::(_updateState) Task=`5de4aa85-93f6-40db-b2a4-3682dc639ebf`::moving from state preparing -> state finished >Thread-28650::DEBUG::2012-12-02 14:34:48,104::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28650::DEBUG::2012-12-02 14:34:48,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28650::DEBUG::2012-12-02 14:34:48,105::task::957::TaskManager.Task::(_decref) Task=`5de4aa85-93f6-40db-b2a4-3682dc639ebf`::ref 0 aborting False >Thread-28654::DEBUG::2012-12-02 14:34:54,182::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28654::DEBUG::2012-12-02 14:34:54,182::task::568::TaskManager.Task::(_updateState) Task=`7fc87625-571b-4952-bdef-afb616b4d060`::moving from state init -> state preparing >Thread-28654::INFO::2012-12-02 14:34:54,182::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28654::INFO::2012-12-02 14:34:54,183::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28654::DEBUG::2012-12-02 14:34:54,183::task::1151::TaskManager.Task::(prepare) Task=`7fc87625-571b-4952-bdef-afb616b4d060`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28654::DEBUG::2012-12-02 14:34:54,183::task::568::TaskManager.Task::(_updateState) Task=`7fc87625-571b-4952-bdef-afb616b4d060`::moving from state preparing -> state finished >Thread-28654::DEBUG::2012-12-02 14:34:54,183::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28654::DEBUG::2012-12-02 14:34:54,183::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28654::DEBUG::2012-12-02 14:34:54,183::task::957::TaskManager.Task::(_decref) Task=`7fc87625-571b-4952-bdef-afb616b4d060`::ref 0 aborting False >Thread-28655::DEBUG::2012-12-02 14:34:54,186::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28655::DEBUG::2012-12-02 14:34:54,187::task::568::TaskManager.Task::(_updateState) Task=`9c8db866-56d6-4470-9ef1-85003b7d1d34`::moving from state init -> state preparing >Thread-28655::INFO::2012-12-02 14:34:54,187::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28655::DEBUG::2012-12-02 14:34:54,187::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8e9ca929-2cd8-4ec5-8b5a-5bb4582d349e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28655::DEBUG::2012-12-02 14:34:54,188::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28655::DEBUG::2012-12-02 14:34:54,188::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28655::DEBUG::2012-12-02 14:34:54,188::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8e9ca929-2cd8-4ec5-8b5a-5bb4582d349e`::Granted request >Thread-28655::DEBUG::2012-12-02 14:34:54,188::task::794::TaskManager.Task::(resourceAcquired) Task=`9c8db866-56d6-4470-9ef1-85003b7d1d34`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28655::DEBUG::2012-12-02 14:34:54,188::task::957::TaskManager.Task::(_decref) Task=`9c8db866-56d6-4470-9ef1-85003b7d1d34`::ref 1 aborting False >Thread-28655::INFO::2012-12-02 14:34:54,195::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28655::DEBUG::2012-12-02 14:34:54,195::task::1151::TaskManager.Task::(prepare) Task=`9c8db866-56d6-4470-9ef1-85003b7d1d34`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28655::DEBUG::2012-12-02 14:34:54,195::task::568::TaskManager.Task::(_updateState) Task=`9c8db866-56d6-4470-9ef1-85003b7d1d34`::moving from state preparing -> state finished >Thread-28655::DEBUG::2012-12-02 14:34:54,195::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28655::DEBUG::2012-12-02 14:34:54,195::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28655::DEBUG::2012-12-02 14:34:54,196::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28655::DEBUG::2012-12-02 14:34:54,196::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28655::DEBUG::2012-12-02 14:34:54,196::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28655::DEBUG::2012-12-02 14:34:54,196::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28655::DEBUG::2012-12-02 14:34:54,196::task::957::TaskManager.Task::(_decref) Task=`9c8db866-56d6-4470-9ef1-85003b7d1d34`::ref 0 aborting False >Thread-28658::DEBUG::2012-12-02 14:34:58,197::task::568::TaskManager.Task::(_updateState) Task=`5119b456-8bf3-47e1-8e47-1a107064e5fb`::moving from state init -> state preparing >Thread-28658::INFO::2012-12-02 14:34:58,197::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28658::INFO::2012-12-02 14:34:58,198::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00962615013123', 'lastCheck': 1354451688.82406, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0017831325531', 'lastCheck': 1354451694.952415, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00195693969727', 'lastCheck': 1354451696.1029661, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00348305702209', 'lastCheck': 1354451694.953362, 'code': 0, 'valid': True}} >Thread-28658::DEBUG::2012-12-02 14:34:58,198::task::1151::TaskManager.Task::(prepare) Task=`5119b456-8bf3-47e1-8e47-1a107064e5fb`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00962615013123', 'lastCheck': 1354451688.82406, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0017831325531', 'lastCheck': 1354451694.952415, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00195693969727', 'lastCheck': 1354451696.1029661, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00348305702209', 'lastCheck': 1354451694.953362, 'code': 0, 'valid': True}} >Thread-28658::DEBUG::2012-12-02 14:34:58,198::task::568::TaskManager.Task::(_updateState) Task=`5119b456-8bf3-47e1-8e47-1a107064e5fb`::moving from state preparing -> state finished >Thread-28658::DEBUG::2012-12-02 14:34:58,198::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28658::DEBUG::2012-12-02 14:34:58,198::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28658::DEBUG::2012-12-02 14:34:58,198::task::957::TaskManager.Task::(_decref) Task=`5119b456-8bf3-47e1-8e47-1a107064e5fb`::ref 0 aborting False >Thread-28663::DEBUG::2012-12-02 14:35:04,287::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28663::DEBUG::2012-12-02 14:35:04,287::task::568::TaskManager.Task::(_updateState) Task=`c4a47b75-dc96-4baf-8951-338e74ebbb72`::moving from state init -> state preparing >Thread-28663::INFO::2012-12-02 14:35:04,287::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28663::INFO::2012-12-02 14:35:04,287::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28663::DEBUG::2012-12-02 14:35:04,287::task::1151::TaskManager.Task::(prepare) Task=`c4a47b75-dc96-4baf-8951-338e74ebbb72`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28663::DEBUG::2012-12-02 14:35:04,288::task::568::TaskManager.Task::(_updateState) Task=`c4a47b75-dc96-4baf-8951-338e74ebbb72`::moving from state preparing -> state finished >Thread-28663::DEBUG::2012-12-02 14:35:04,288::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28663::DEBUG::2012-12-02 14:35:04,288::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28663::DEBUG::2012-12-02 14:35:04,288::task::957::TaskManager.Task::(_decref) Task=`c4a47b75-dc96-4baf-8951-338e74ebbb72`::ref 0 aborting False >Thread-28664::DEBUG::2012-12-02 14:35:04,291::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28664::DEBUG::2012-12-02 14:35:04,291::task::568::TaskManager.Task::(_updateState) Task=`28043c3e-69df-46f1-90cf-45a99f175bd5`::moving from state init -> state preparing >Thread-28664::INFO::2012-12-02 14:35:04,291::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28664::DEBUG::2012-12-02 14:35:04,291::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f502d990-8672-4626-aabd-f4e81103bc4a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28664::DEBUG::2012-12-02 14:35:04,291::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28664::DEBUG::2012-12-02 14:35:04,292::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28664::DEBUG::2012-12-02 14:35:04,292::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f502d990-8672-4626-aabd-f4e81103bc4a`::Granted request >Thread-28664::DEBUG::2012-12-02 14:35:04,292::task::794::TaskManager.Task::(resourceAcquired) Task=`28043c3e-69df-46f1-90cf-45a99f175bd5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28664::DEBUG::2012-12-02 14:35:04,292::task::957::TaskManager.Task::(_decref) Task=`28043c3e-69df-46f1-90cf-45a99f175bd5`::ref 1 aborting False >Thread-28664::INFO::2012-12-02 14:35:04,297::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28664::DEBUG::2012-12-02 14:35:04,298::task::1151::TaskManager.Task::(prepare) Task=`28043c3e-69df-46f1-90cf-45a99f175bd5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28664::DEBUG::2012-12-02 14:35:04,298::task::568::TaskManager.Task::(_updateState) Task=`28043c3e-69df-46f1-90cf-45a99f175bd5`::moving from state preparing -> state finished >Thread-28664::DEBUG::2012-12-02 14:35:04,298::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28664::DEBUG::2012-12-02 14:35:04,298::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28664::DEBUG::2012-12-02 14:35:04,298::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28664::DEBUG::2012-12-02 14:35:04,298::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28664::DEBUG::2012-12-02 14:35:04,298::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28664::DEBUG::2012-12-02 14:35:04,298::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28664::DEBUG::2012-12-02 14:35:04,298::task::957::TaskManager.Task::(_decref) Task=`28043c3e-69df-46f1-90cf-45a99f175bd5`::ref 0 aborting False >Thread-28666::DEBUG::2012-12-02 14:35:08,286::task::568::TaskManager.Task::(_updateState) Task=`7818ebb6-ae18-4e2d-8c82-2ba53287a227`::moving from state init -> state preparing >Thread-28666::INFO::2012-12-02 14:35:08,287::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28666::INFO::2012-12-02 14:35:08,287::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0115959644318', 'lastCheck': 1354451698.8442531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00234603881836', 'lastCheck': 1354451704.968946, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00327110290527', 'lastCheck': 1354451706.1209991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821495056152', 'lastCheck': 1354451704.9754341, 'code': 0, 'valid': True}} >Thread-28666::DEBUG::2012-12-02 14:35:08,287::task::1151::TaskManager.Task::(prepare) Task=`7818ebb6-ae18-4e2d-8c82-2ba53287a227`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0115959644318', 'lastCheck': 1354451698.8442531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00234603881836', 'lastCheck': 1354451704.968946, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00327110290527', 'lastCheck': 1354451706.1209991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821495056152', 'lastCheck': 1354451704.9754341, 'code': 0, 'valid': True}} >Thread-28666::DEBUG::2012-12-02 14:35:08,287::task::568::TaskManager.Task::(_updateState) Task=`7818ebb6-ae18-4e2d-8c82-2ba53287a227`::moving from state preparing -> state finished >Thread-28666::DEBUG::2012-12-02 14:35:08,287::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28666::DEBUG::2012-12-02 14:35:08,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28666::DEBUG::2012-12-02 14:35:08,288::task::957::TaskManager.Task::(_decref) Task=`7818ebb6-ae18-4e2d-8c82-2ba53287a227`::ref 0 aborting False >Thread-28671::DEBUG::2012-12-02 14:35:14,391::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28671::DEBUG::2012-12-02 14:35:14,391::task::568::TaskManager.Task::(_updateState) Task=`a50c6ef6-25e1-4a29-9632-37f82ead7084`::moving from state init -> state preparing >Thread-28671::INFO::2012-12-02 14:35:14,392::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28671::INFO::2012-12-02 14:35:14,392::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28671::DEBUG::2012-12-02 14:35:14,392::task::1151::TaskManager.Task::(prepare) Task=`a50c6ef6-25e1-4a29-9632-37f82ead7084`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28671::DEBUG::2012-12-02 14:35:14,392::task::568::TaskManager.Task::(_updateState) Task=`a50c6ef6-25e1-4a29-9632-37f82ead7084`::moving from state preparing -> state finished >Thread-28671::DEBUG::2012-12-02 14:35:14,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28671::DEBUG::2012-12-02 14:35:14,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28671::DEBUG::2012-12-02 14:35:14,392::task::957::TaskManager.Task::(_decref) Task=`a50c6ef6-25e1-4a29-9632-37f82ead7084`::ref 0 aborting False >Thread-28672::DEBUG::2012-12-02 14:35:14,395::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28672::DEBUG::2012-12-02 14:35:14,395::task::568::TaskManager.Task::(_updateState) Task=`ae5737d9-9375-45fe-8283-a11e1468635e`::moving from state init -> state preparing >Thread-28672::INFO::2012-12-02 14:35:14,395::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28672::DEBUG::2012-12-02 14:35:14,396::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8d044eb8-201d-443d-be02-202c267e362c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28672::DEBUG::2012-12-02 14:35:14,396::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28672::DEBUG::2012-12-02 14:35:14,396::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28672::DEBUG::2012-12-02 14:35:14,396::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8d044eb8-201d-443d-be02-202c267e362c`::Granted request >Thread-28672::DEBUG::2012-12-02 14:35:14,396::task::794::TaskManager.Task::(resourceAcquired) Task=`ae5737d9-9375-45fe-8283-a11e1468635e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28672::DEBUG::2012-12-02 14:35:14,396::task::957::TaskManager.Task::(_decref) Task=`ae5737d9-9375-45fe-8283-a11e1468635e`::ref 1 aborting False >Thread-28672::INFO::2012-12-02 14:35:14,401::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28672::DEBUG::2012-12-02 14:35:14,401::task::1151::TaskManager.Task::(prepare) Task=`ae5737d9-9375-45fe-8283-a11e1468635e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28672::DEBUG::2012-12-02 14:35:14,401::task::568::TaskManager.Task::(_updateState) Task=`ae5737d9-9375-45fe-8283-a11e1468635e`::moving from state preparing -> state finished >Thread-28672::DEBUG::2012-12-02 14:35:14,401::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28672::DEBUG::2012-12-02 14:35:14,401::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28672::DEBUG::2012-12-02 14:35:14,401::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28672::DEBUG::2012-12-02 14:35:14,401::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28672::DEBUG::2012-12-02 14:35:14,401::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28672::DEBUG::2012-12-02 14:35:14,401::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28672::DEBUG::2012-12-02 14:35:14,401::task::957::TaskManager.Task::(_decref) Task=`ae5737d9-9375-45fe-8283-a11e1468635e`::ref 0 aborting False >Thread-28674::DEBUG::2012-12-02 14:35:18,399::task::568::TaskManager.Task::(_updateState) Task=`b480dd90-c730-447a-abba-dfa7b7543411`::moving from state init -> state preparing >Thread-28674::INFO::2012-12-02 14:35:18,399::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28674::INFO::2012-12-02 14:35:18,399::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830101966858', 'lastCheck': 1354451708.8611121, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00477409362793', 'lastCheck': 1354451714.984695, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00543999671936', 'lastCheck': 1354451716.137234, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00715303421021', 'lastCheck': 1354451714.991688, 'code': 0, 'valid': True}} >Thread-28674::DEBUG::2012-12-02 14:35:18,399::task::1151::TaskManager.Task::(prepare) Task=`b480dd90-c730-447a-abba-dfa7b7543411`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830101966858', 'lastCheck': 1354451708.8611121, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00477409362793', 'lastCheck': 1354451714.984695, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00543999671936', 'lastCheck': 1354451716.137234, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00715303421021', 'lastCheck': 1354451714.991688, 'code': 0, 'valid': True}} >Thread-28674::DEBUG::2012-12-02 14:35:18,399::task::568::TaskManager.Task::(_updateState) Task=`b480dd90-c730-447a-abba-dfa7b7543411`::moving from state preparing -> state finished >Thread-28674::DEBUG::2012-12-02 14:35:18,399::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28674::DEBUG::2012-12-02 14:35:18,400::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28674::DEBUG::2012-12-02 14:35:18,400::task::957::TaskManager.Task::(_decref) Task=`b480dd90-c730-447a-abba-dfa7b7543411`::ref 0 aborting False >Thread-28678::DEBUG::2012-12-02 14:35:24,538::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28678::DEBUG::2012-12-02 14:35:24,538::task::568::TaskManager.Task::(_updateState) Task=`6214dc0f-d71b-4e27-a8a0-411f1a851f5c`::moving from state init -> state preparing >Thread-28678::INFO::2012-12-02 14:35:24,538::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28678::INFO::2012-12-02 14:35:24,538::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28678::DEBUG::2012-12-02 14:35:24,539::task::1151::TaskManager.Task::(prepare) Task=`6214dc0f-d71b-4e27-a8a0-411f1a851f5c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28678::DEBUG::2012-12-02 14:35:24,539::task::568::TaskManager.Task::(_updateState) Task=`6214dc0f-d71b-4e27-a8a0-411f1a851f5c`::moving from state preparing -> state finished >Thread-28678::DEBUG::2012-12-02 14:35:24,539::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28678::DEBUG::2012-12-02 14:35:24,539::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28678::DEBUG::2012-12-02 14:35:24,539::task::957::TaskManager.Task::(_decref) Task=`6214dc0f-d71b-4e27-a8a0-411f1a851f5c`::ref 0 aborting False >Thread-28679::DEBUG::2012-12-02 14:35:24,542::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28679::DEBUG::2012-12-02 14:35:24,542::task::568::TaskManager.Task::(_updateState) Task=`3bcf767c-678e-4022-a914-6238829eb8cb`::moving from state init -> state preparing >Thread-28679::INFO::2012-12-02 14:35:24,542::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28679::DEBUG::2012-12-02 14:35:24,542::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b3905820-82a3-46bd-93c8-fb77b5492987`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28679::DEBUG::2012-12-02 14:35:24,543::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28679::DEBUG::2012-12-02 14:35:24,543::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28679::DEBUG::2012-12-02 14:35:24,543::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b3905820-82a3-46bd-93c8-fb77b5492987`::Granted request >Thread-28679::DEBUG::2012-12-02 14:35:24,543::task::794::TaskManager.Task::(resourceAcquired) Task=`3bcf767c-678e-4022-a914-6238829eb8cb`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28679::DEBUG::2012-12-02 14:35:24,543::task::957::TaskManager.Task::(_decref) Task=`3bcf767c-678e-4022-a914-6238829eb8cb`::ref 1 aborting False >Thread-28679::INFO::2012-12-02 14:35:24,548::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28679::DEBUG::2012-12-02 14:35:24,548::task::1151::TaskManager.Task::(prepare) Task=`3bcf767c-678e-4022-a914-6238829eb8cb`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28679::DEBUG::2012-12-02 14:35:24,548::task::568::TaskManager.Task::(_updateState) Task=`3bcf767c-678e-4022-a914-6238829eb8cb`::moving from state preparing -> state finished >Thread-28679::DEBUG::2012-12-02 14:35:24,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28679::DEBUG::2012-12-02 14:35:24,549::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28679::DEBUG::2012-12-02 14:35:24,549::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28679::DEBUG::2012-12-02 14:35:24,549::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28679::DEBUG::2012-12-02 14:35:24,549::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28679::DEBUG::2012-12-02 14:35:24,549::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28679::DEBUG::2012-12-02 14:35:24,549::task::957::TaskManager.Task::(_decref) Task=`3bcf767c-678e-4022-a914-6238829eb8cb`::ref 0 aborting False >Thread-28682::DEBUG::2012-12-02 14:35:28,603::task::568::TaskManager.Task::(_updateState) Task=`e9e247c9-0a93-4322-93ba-3f6ac2a27a1a`::moving from state init -> state preparing >Thread-28682::INFO::2012-12-02 14:35:28,603::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28682::INFO::2012-12-02 14:35:28,603::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00759506225586', 'lastCheck': 1354451718.8776569, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00430297851562', 'lastCheck': 1354451725.0051229, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00180697441101', 'lastCheck': 1354451726.1495991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0111289024353', 'lastCheck': 1354451725.014219, 'code': 0, 'valid': True}} >Thread-28682::DEBUG::2012-12-02 14:35:28,604::task::1151::TaskManager.Task::(prepare) Task=`e9e247c9-0a93-4322-93ba-3f6ac2a27a1a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00759506225586', 'lastCheck': 1354451718.8776569, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00430297851562', 'lastCheck': 1354451725.0051229, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00180697441101', 'lastCheck': 1354451726.1495991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0111289024353', 'lastCheck': 1354451725.014219, 'code': 0, 'valid': True}} >Thread-28682::DEBUG::2012-12-02 14:35:28,604::task::568::TaskManager.Task::(_updateState) Task=`e9e247c9-0a93-4322-93ba-3f6ac2a27a1a`::moving from state preparing -> state finished >Thread-28682::DEBUG::2012-12-02 14:35:28,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28682::DEBUG::2012-12-02 14:35:28,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28682::DEBUG::2012-12-02 14:35:28,604::task::957::TaskManager.Task::(_decref) Task=`e9e247c9-0a93-4322-93ba-3f6ac2a27a1a`::ref 0 aborting False >Thread-28686::DEBUG::2012-12-02 14:35:34,684::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28686::DEBUG::2012-12-02 14:35:34,685::task::568::TaskManager.Task::(_updateState) Task=`0542c912-1b97-4169-89b1-192aa7ca3806`::moving from state init -> state preparing >Thread-28686::INFO::2012-12-02 14:35:34,685::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28686::INFO::2012-12-02 14:35:34,685::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28686::DEBUG::2012-12-02 14:35:34,685::task::1151::TaskManager.Task::(prepare) Task=`0542c912-1b97-4169-89b1-192aa7ca3806`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28686::DEBUG::2012-12-02 14:35:34,685::task::568::TaskManager.Task::(_updateState) Task=`0542c912-1b97-4169-89b1-192aa7ca3806`::moving from state preparing -> state finished >Thread-28686::DEBUG::2012-12-02 14:35:34,685::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28686::DEBUG::2012-12-02 14:35:34,685::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28686::DEBUG::2012-12-02 14:35:34,686::task::957::TaskManager.Task::(_decref) Task=`0542c912-1b97-4169-89b1-192aa7ca3806`::ref 0 aborting False >Thread-28687::DEBUG::2012-12-02 14:35:34,688::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28687::DEBUG::2012-12-02 14:35:34,688::task::568::TaskManager.Task::(_updateState) Task=`2ddaea8a-9f2b-4c4d-9f19-513d0aeb1977`::moving from state init -> state preparing >Thread-28687::INFO::2012-12-02 14:35:34,688::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28687::DEBUG::2012-12-02 14:35:34,688::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bbd613a2-6973-4baa-8258-f45af420da21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28687::DEBUG::2012-12-02 14:35:34,689::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28687::DEBUG::2012-12-02 14:35:34,689::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28687::DEBUG::2012-12-02 14:35:34,689::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bbd613a2-6973-4baa-8258-f45af420da21`::Granted request >Thread-28687::DEBUG::2012-12-02 14:35:34,689::task::794::TaskManager.Task::(resourceAcquired) Task=`2ddaea8a-9f2b-4c4d-9f19-513d0aeb1977`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28687::DEBUG::2012-12-02 14:35:34,689::task::957::TaskManager.Task::(_decref) Task=`2ddaea8a-9f2b-4c4d-9f19-513d0aeb1977`::ref 1 aborting False >Thread-28687::INFO::2012-12-02 14:35:34,693::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28687::DEBUG::2012-12-02 14:35:34,693::task::1151::TaskManager.Task::(prepare) Task=`2ddaea8a-9f2b-4c4d-9f19-513d0aeb1977`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28687::DEBUG::2012-12-02 14:35:34,694::task::568::TaskManager.Task::(_updateState) Task=`2ddaea8a-9f2b-4c4d-9f19-513d0aeb1977`::moving from state preparing -> state finished >Thread-28687::DEBUG::2012-12-02 14:35:34,694::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28687::DEBUG::2012-12-02 14:35:34,694::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28687::DEBUG::2012-12-02 14:35:34,694::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28687::DEBUG::2012-12-02 14:35:34,694::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28687::DEBUG::2012-12-02 14:35:34,694::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28687::DEBUG::2012-12-02 14:35:34,694::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28687::DEBUG::2012-12-02 14:35:34,694::task::957::TaskManager.Task::(_decref) Task=`2ddaea8a-9f2b-4c4d-9f19-513d0aeb1977`::ref 0 aborting False >Thread-28690::DEBUG::2012-12-02 14:35:38,710::task::568::TaskManager.Task::(_updateState) Task=`e7786b88-0f3f-49e0-b831-19c57a6d646e`::moving from state init -> state preparing >Thread-28690::INFO::2012-12-02 14:35:38,710::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28690::INFO::2012-12-02 14:35:38,710::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00982093811035', 'lastCheck': 1354451728.8972459, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00476908683777', 'lastCheck': 1354451735.021354, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199317932129', 'lastCheck': 1354451736.2008159, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00726103782654', 'lastCheck': 1354451735.0306039, 'code': 0, 'valid': True}} >Thread-28690::DEBUG::2012-12-02 14:35:38,710::task::1151::TaskManager.Task::(prepare) Task=`e7786b88-0f3f-49e0-b831-19c57a6d646e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00982093811035', 'lastCheck': 1354451728.8972459, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00476908683777', 'lastCheck': 1354451735.021354, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199317932129', 'lastCheck': 1354451736.2008159, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00726103782654', 'lastCheck': 1354451735.0306039, 'code': 0, 'valid': True}} >Thread-28690::DEBUG::2012-12-02 14:35:38,710::task::568::TaskManager.Task::(_updateState) Task=`e7786b88-0f3f-49e0-b831-19c57a6d646e`::moving from state preparing -> state finished >Thread-28690::DEBUG::2012-12-02 14:35:38,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28690::DEBUG::2012-12-02 14:35:38,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28690::DEBUG::2012-12-02 14:35:38,711::task::957::TaskManager.Task::(_decref) Task=`e7786b88-0f3f-49e0-b831-19c57a6d646e`::ref 0 aborting False >Thread-28695::DEBUG::2012-12-02 14:35:44,831::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28695::DEBUG::2012-12-02 14:35:44,831::task::568::TaskManager.Task::(_updateState) Task=`75643194-a40d-4f98-ab02-477b91d88c42`::moving from state init -> state preparing >Thread-28695::INFO::2012-12-02 14:35:44,832::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28695::INFO::2012-12-02 14:35:44,832::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28695::DEBUG::2012-12-02 14:35:44,832::task::1151::TaskManager.Task::(prepare) Task=`75643194-a40d-4f98-ab02-477b91d88c42`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28695::DEBUG::2012-12-02 14:35:44,832::task::568::TaskManager.Task::(_updateState) Task=`75643194-a40d-4f98-ab02-477b91d88c42`::moving from state preparing -> state finished >Thread-28695::DEBUG::2012-12-02 14:35:44,832::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28695::DEBUG::2012-12-02 14:35:44,832::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28695::DEBUG::2012-12-02 14:35:44,832::task::957::TaskManager.Task::(_decref) Task=`75643194-a40d-4f98-ab02-477b91d88c42`::ref 0 aborting False >Thread-28696::DEBUG::2012-12-02 14:35:44,835::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28696::DEBUG::2012-12-02 14:35:44,835::task::568::TaskManager.Task::(_updateState) Task=`30f8eb0c-39f1-43be-bbde-f2399592323e`::moving from state init -> state preparing >Thread-28696::INFO::2012-12-02 14:35:44,835::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28696::DEBUG::2012-12-02 14:35:44,836::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d4243a01-2dd5-42d2-a303-7e5172fa1085`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28696::DEBUG::2012-12-02 14:35:44,836::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28696::DEBUG::2012-12-02 14:35:44,836::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28696::DEBUG::2012-12-02 14:35:44,836::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d4243a01-2dd5-42d2-a303-7e5172fa1085`::Granted request >Thread-28696::DEBUG::2012-12-02 14:35:44,836::task::794::TaskManager.Task::(resourceAcquired) Task=`30f8eb0c-39f1-43be-bbde-f2399592323e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28696::DEBUG::2012-12-02 14:35:44,837::task::957::TaskManager.Task::(_decref) Task=`30f8eb0c-39f1-43be-bbde-f2399592323e`::ref 1 aborting False >Thread-28696::INFO::2012-12-02 14:35:44,843::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28696::DEBUG::2012-12-02 14:35:44,843::task::1151::TaskManager.Task::(prepare) Task=`30f8eb0c-39f1-43be-bbde-f2399592323e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28696::DEBUG::2012-12-02 14:35:44,843::task::568::TaskManager.Task::(_updateState) Task=`30f8eb0c-39f1-43be-bbde-f2399592323e`::moving from state preparing -> state finished >Thread-28696::DEBUG::2012-12-02 14:35:44,843::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28696::DEBUG::2012-12-02 14:35:44,844::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28696::DEBUG::2012-12-02 14:35:44,844::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28696::DEBUG::2012-12-02 14:35:44,844::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28696::DEBUG::2012-12-02 14:35:44,844::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28696::DEBUG::2012-12-02 14:35:44,844::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28696::DEBUG::2012-12-02 14:35:44,844::task::957::TaskManager.Task::(_decref) Task=`30f8eb0c-39f1-43be-bbde-f2399592323e`::ref 0 aborting False >Thread-28698::DEBUG::2012-12-02 14:35:48,772::task::568::TaskManager.Task::(_updateState) Task=`ff197f9c-c2b3-4989-94fc-2da85b1bf8c5`::moving from state init -> state preparing >Thread-28698::INFO::2012-12-02 14:35:48,772::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28698::INFO::2012-12-02 14:35:48,772::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00841808319092', 'lastCheck': 1354451738.9178531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00156021118164', 'lastCheck': 1354451745.0383179, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00175619125366', 'lastCheck': 1354451746.213083, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00405502319336', 'lastCheck': 1354451745.045224, 'code': 0, 'valid': True}} >Thread-28698::DEBUG::2012-12-02 14:35:48,772::task::1151::TaskManager.Task::(prepare) Task=`ff197f9c-c2b3-4989-94fc-2da85b1bf8c5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00841808319092', 'lastCheck': 1354451738.9178531, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00156021118164', 'lastCheck': 1354451745.0383179, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00175619125366', 'lastCheck': 1354451746.213083, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00405502319336', 'lastCheck': 1354451745.045224, 'code': 0, 'valid': True}} >Thread-28698::DEBUG::2012-12-02 14:35:48,773::task::568::TaskManager.Task::(_updateState) Task=`ff197f9c-c2b3-4989-94fc-2da85b1bf8c5`::moving from state preparing -> state finished >Thread-28698::DEBUG::2012-12-02 14:35:48,773::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28698::DEBUG::2012-12-02 14:35:48,773::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28698::DEBUG::2012-12-02 14:35:48,773::task::957::TaskManager.Task::(_decref) Task=`ff197f9c-c2b3-4989-94fc-2da85b1bf8c5`::ref 0 aborting False >Thread-28703::DEBUG::2012-12-02 14:35:55,053::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28703::DEBUG::2012-12-02 14:35:55,053::task::568::TaskManager.Task::(_updateState) Task=`0fe44129-b231-4fde-a3a2-fd73212dddfd`::moving from state init -> state preparing >Thread-28703::INFO::2012-12-02 14:35:55,053::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28703::INFO::2012-12-02 14:35:55,053::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28703::DEBUG::2012-12-02 14:35:55,053::task::1151::TaskManager.Task::(prepare) Task=`0fe44129-b231-4fde-a3a2-fd73212dddfd`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28703::DEBUG::2012-12-02 14:35:55,054::task::568::TaskManager.Task::(_updateState) Task=`0fe44129-b231-4fde-a3a2-fd73212dddfd`::moving from state preparing -> state finished >Thread-28703::DEBUG::2012-12-02 14:35:55,054::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28703::DEBUG::2012-12-02 14:35:55,054::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28703::DEBUG::2012-12-02 14:35:55,054::task::957::TaskManager.Task::(_decref) Task=`0fe44129-b231-4fde-a3a2-fd73212dddfd`::ref 0 aborting False >Thread-28704::DEBUG::2012-12-02 14:35:55,057::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28704::DEBUG::2012-12-02 14:35:55,057::task::568::TaskManager.Task::(_updateState) Task=`f7d2cbf2-5934-4529-b900-e8f908094b9e`::moving from state init -> state preparing >Thread-28704::INFO::2012-12-02 14:35:55,057::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28704::DEBUG::2012-12-02 14:35:55,057::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`af2497b3-8680-4c75-99b8-ef86fee61d24`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28704::DEBUG::2012-12-02 14:35:55,057::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28704::DEBUG::2012-12-02 14:35:55,057::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28704::DEBUG::2012-12-02 14:35:55,057::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`af2497b3-8680-4c75-99b8-ef86fee61d24`::Granted request >Thread-28704::DEBUG::2012-12-02 14:35:55,058::task::794::TaskManager.Task::(resourceAcquired) Task=`f7d2cbf2-5934-4529-b900-e8f908094b9e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28704::DEBUG::2012-12-02 14:35:55,058::task::957::TaskManager.Task::(_decref) Task=`f7d2cbf2-5934-4529-b900-e8f908094b9e`::ref 1 aborting False >Thread-28704::INFO::2012-12-02 14:35:55,063::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28704::DEBUG::2012-12-02 14:35:55,063::task::1151::TaskManager.Task::(prepare) Task=`f7d2cbf2-5934-4529-b900-e8f908094b9e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28704::DEBUG::2012-12-02 14:35:55,063::task::568::TaskManager.Task::(_updateState) Task=`f7d2cbf2-5934-4529-b900-e8f908094b9e`::moving from state preparing -> state finished >Thread-28704::DEBUG::2012-12-02 14:35:55,063::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28704::DEBUG::2012-12-02 14:35:55,063::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28704::DEBUG::2012-12-02 14:35:55,063::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28704::DEBUG::2012-12-02 14:35:55,064::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28704::DEBUG::2012-12-02 14:35:55,064::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28704::DEBUG::2012-12-02 14:35:55,064::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28704::DEBUG::2012-12-02 14:35:55,064::task::957::TaskManager.Task::(_decref) Task=`f7d2cbf2-5934-4529-b900-e8f908094b9e`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:35:56,213::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:35:56,213::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:35:56,213::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:35:56,425::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:35:56,426::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:35:56,426::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:35:56,461::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:35:56,462::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:35:56,464::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:35:56,467::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:35:56,468::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:35:56,468::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-28767::DEBUG::2012-12-02 14:35:58,920::task::568::TaskManager.Task::(_updateState) Task=`490e08d0-ba6a-4d5d-b955-0bf161045d20`::moving from state init -> state preparing >Thread-28767::INFO::2012-12-02 14:35:58,921::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28767::INFO::2012-12-02 14:35:58,921::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00745010375977', 'lastCheck': 1354451748.9351051, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00305509567261', 'lastCheck': 1354451755.0531271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0023889541626', 'lastCheck': 1354451756.4792769, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00844311714172', 'lastCheck': 1354451755.0639491, 'code': 0, 'valid': True}} >Thread-28767::DEBUG::2012-12-02 14:35:58,921::task::1151::TaskManager.Task::(prepare) Task=`490e08d0-ba6a-4d5d-b955-0bf161045d20`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00745010375977', 'lastCheck': 1354451748.9351051, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00305509567261', 'lastCheck': 1354451755.0531271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0023889541626', 'lastCheck': 1354451756.4792769, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00844311714172', 'lastCheck': 1354451755.0639491, 'code': 0, 'valid': True}} >Thread-28767::DEBUG::2012-12-02 14:35:58,921::task::568::TaskManager.Task::(_updateState) Task=`490e08d0-ba6a-4d5d-b955-0bf161045d20`::moving from state preparing -> state finished >Thread-28767::DEBUG::2012-12-02 14:35:58,921::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28767::DEBUG::2012-12-02 14:35:58,921::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28767::DEBUG::2012-12-02 14:35:58,922::task::957::TaskManager.Task::(_decref) Task=`490e08d0-ba6a-4d5d-b955-0bf161045d20`::ref 0 aborting False >Thread-11821::DEBUG::2012-12-02 14:35:58,935::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:35:58,935::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:35:58,936::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:35:59,145::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:35:59,147::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:35:59,147::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:35:59,181::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:35:59,182::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:35:59,191::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:35:59,194::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:35:59,194::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:35:59,194::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-11822::DEBUG::2012-12-02 14:36:05,053::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:36:05,053::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:36:05,054::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-28833::DEBUG::2012-12-02 14:36:05,191::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28833::DEBUG::2012-12-02 14:36:05,192::task::568::TaskManager.Task::(_updateState) Task=`02696fe0-a751-4592-8d16-255321b7705a`::moving from state init -> state preparing >Thread-28833::INFO::2012-12-02 14:36:05,192::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-11822::DEBUG::2012-12-02 14:36:05,262::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:36:05,263::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:36:05,263::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:36:05,297::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:36:05,297::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:36:05,300::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:36:05,303::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:36:05,303::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:36:05,303::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:36:05,303::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-28833::INFO::2012-12-02 14:36:05,303::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-11946::DEBUG::2012-12-02 14:36:05,304::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-28833::DEBUG::2012-12-02 14:36:05,306::task::1151::TaskManager.Task::(prepare) Task=`02696fe0-a751-4592-8d16-255321b7705a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-11946::DEBUG::2012-12-02 14:36:05,308::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-28833::DEBUG::2012-12-02 14:36:05,309::task::568::TaskManager.Task::(_updateState) Task=`02696fe0-a751-4592-8d16-255321b7705a`::moving from state preparing -> state finished >Thread-28833::DEBUG::2012-12-02 14:36:05,316::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28833::DEBUG::2012-12-02 14:36:05,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28833::DEBUG::2012-12-02 14:36:05,317::task::957::TaskManager.Task::(_decref) Task=`02696fe0-a751-4592-8d16-255321b7705a`::ref 0 aborting False >Thread-28895::DEBUG::2012-12-02 14:36:05,321::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28895::DEBUG::2012-12-02 14:36:05,321::task::568::TaskManager.Task::(_updateState) Task=`5601b1ad-e3be-480b-a3b0-14a2592279ed`::moving from state init -> state preparing >Thread-28895::INFO::2012-12-02 14:36:05,321::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28895::DEBUG::2012-12-02 14:36:05,322::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2f80095e-0604-43f1-9b7d-6571d92f7237`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28895::DEBUG::2012-12-02 14:36:05,322::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28895::DEBUG::2012-12-02 14:36:05,322::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28895::DEBUG::2012-12-02 14:36:05,322::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2f80095e-0604-43f1-9b7d-6571d92f7237`::Granted request >Thread-28895::DEBUG::2012-12-02 14:36:05,322::task::794::TaskManager.Task::(resourceAcquired) Task=`5601b1ad-e3be-480b-a3b0-14a2592279ed`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28895::DEBUG::2012-12-02 14:36:05,322::task::957::TaskManager.Task::(_decref) Task=`5601b1ad-e3be-480b-a3b0-14a2592279ed`::ref 1 aborting False >Thread-11946::DEBUG::2012-12-02 14:36:05,517::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:36:05,519::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:36:05,519::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:36:05,553::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:36:05,553::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:36:05,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:36:05,565::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:36:05,565::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:36:05,565::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-28895::INFO::2012-12-02 14:36:05,581::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28895::DEBUG::2012-12-02 14:36:05,582::task::1151::TaskManager.Task::(prepare) Task=`5601b1ad-e3be-480b-a3b0-14a2592279ed`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28895::DEBUG::2012-12-02 14:36:05,582::task::568::TaskManager.Task::(_updateState) Task=`5601b1ad-e3be-480b-a3b0-14a2592279ed`::moving from state preparing -> state finished >Thread-28895::DEBUG::2012-12-02 14:36:05,582::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28895::DEBUG::2012-12-02 14:36:05,582::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28895::DEBUG::2012-12-02 14:36:05,583::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28895::DEBUG::2012-12-02 14:36:05,583::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28895::DEBUG::2012-12-02 14:36:05,583::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28895::DEBUG::2012-12-02 14:36:05,583::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28895::DEBUG::2012-12-02 14:36:05,583::task::957::TaskManager.Task::(_decref) Task=`5601b1ad-e3be-480b-a3b0-14a2592279ed`::ref 0 aborting False >Thread-28958::DEBUG::2012-12-02 14:36:09,015::task::568::TaskManager.Task::(_updateState) Task=`8295e20b-0964-4143-b3ec-2ca998dc538b`::moving from state init -> state preparing >Thread-28958::INFO::2012-12-02 14:36:09,015::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28958::INFO::2012-12-02 14:36:09,016::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00602507591248', 'lastCheck': 1354451759.2098229, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00328207015991', 'lastCheck': 1354451765.5830951, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0017261505127', 'lastCheck': 1354451766.4899099, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00866913795471', 'lastCheck': 1354451765.592042, 'code': 0, 'valid': True}} >Thread-28958::DEBUG::2012-12-02 14:36:09,016::task::1151::TaskManager.Task::(prepare) Task=`8295e20b-0964-4143-b3ec-2ca998dc538b`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00602507591248', 'lastCheck': 1354451759.2098229, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00328207015991', 'lastCheck': 1354451765.5830951, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0017261505127', 'lastCheck': 1354451766.4899099, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00866913795471', 'lastCheck': 1354451765.592042, 'code': 0, 'valid': True}} >Thread-28958::DEBUG::2012-12-02 14:36:09,016::task::568::TaskManager.Task::(_updateState) Task=`8295e20b-0964-4143-b3ec-2ca998dc538b`::moving from state preparing -> state finished >Thread-28958::DEBUG::2012-12-02 14:36:09,016::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28958::DEBUG::2012-12-02 14:36:09,016::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28958::DEBUG::2012-12-02 14:36:09,016::task::957::TaskManager.Task::(_decref) Task=`8295e20b-0964-4143-b3ec-2ca998dc538b`::ref 0 aborting False >Thread-28963::DEBUG::2012-12-02 14:36:15,709::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28963::DEBUG::2012-12-02 14:36:15,709::task::568::TaskManager.Task::(_updateState) Task=`ad6670c9-09cb-490a-9efc-d50a08785944`::moving from state init -> state preparing >Thread-28963::INFO::2012-12-02 14:36:15,709::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28963::INFO::2012-12-02 14:36:15,710::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28963::DEBUG::2012-12-02 14:36:15,710::task::1151::TaskManager.Task::(prepare) Task=`ad6670c9-09cb-490a-9efc-d50a08785944`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28963::DEBUG::2012-12-02 14:36:15,710::task::568::TaskManager.Task::(_updateState) Task=`ad6670c9-09cb-490a-9efc-d50a08785944`::moving from state preparing -> state finished >Thread-28963::DEBUG::2012-12-02 14:36:15,710::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28963::DEBUG::2012-12-02 14:36:15,710::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28963::DEBUG::2012-12-02 14:36:15,710::task::957::TaskManager.Task::(_decref) Task=`ad6670c9-09cb-490a-9efc-d50a08785944`::ref 0 aborting False >Thread-28964::DEBUG::2012-12-02 14:36:15,713::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28964::DEBUG::2012-12-02 14:36:15,713::task::568::TaskManager.Task::(_updateState) Task=`b650f4bd-b475-4b6e-bd24-e88bc3fac5f7`::moving from state init -> state preparing >Thread-28964::INFO::2012-12-02 14:36:15,713::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28964::DEBUG::2012-12-02 14:36:15,713::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5b7e1005-c355-4b07-a139-af857cd03484`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28964::DEBUG::2012-12-02 14:36:15,714::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28964::DEBUG::2012-12-02 14:36:15,714::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28964::DEBUG::2012-12-02 14:36:15,714::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5b7e1005-c355-4b07-a139-af857cd03484`::Granted request >Thread-28964::DEBUG::2012-12-02 14:36:15,714::task::794::TaskManager.Task::(resourceAcquired) Task=`b650f4bd-b475-4b6e-bd24-e88bc3fac5f7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28964::DEBUG::2012-12-02 14:36:15,714::task::957::TaskManager.Task::(_decref) Task=`b650f4bd-b475-4b6e-bd24-e88bc3fac5f7`::ref 1 aborting False >Thread-28964::INFO::2012-12-02 14:36:15,719::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28964::DEBUG::2012-12-02 14:36:15,719::task::1151::TaskManager.Task::(prepare) Task=`b650f4bd-b475-4b6e-bd24-e88bc3fac5f7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28964::DEBUG::2012-12-02 14:36:15,719::task::568::TaskManager.Task::(_updateState) Task=`b650f4bd-b475-4b6e-bd24-e88bc3fac5f7`::moving from state preparing -> state finished >Thread-28964::DEBUG::2012-12-02 14:36:15,719::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28964::DEBUG::2012-12-02 14:36:15,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28964::DEBUG::2012-12-02 14:36:15,720::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28964::DEBUG::2012-12-02 14:36:15,720::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28964::DEBUG::2012-12-02 14:36:15,720::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28964::DEBUG::2012-12-02 14:36:15,720::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28964::DEBUG::2012-12-02 14:36:15,720::task::957::TaskManager.Task::(_decref) Task=`b650f4bd-b475-4b6e-bd24-e88bc3fac5f7`::ref 0 aborting False >Thread-28966::DEBUG::2012-12-02 14:36:19,162::task::568::TaskManager.Task::(_updateState) Task=`263fff1d-a4c3-4b3c-8b5e-dd3f6ed1cc45`::moving from state init -> state preparing >Thread-28966::INFO::2012-12-02 14:36:19,162::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28966::INFO::2012-12-02 14:36:19,163::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00732421875', 'lastCheck': 1354451769.227241, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00574517250061', 'lastCheck': 1354451775.600004, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183892250061', 'lastCheck': 1354451776.5360351, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00753021240234', 'lastCheck': 1354451775.609256, 'code': 0, 'valid': True}} >Thread-28966::DEBUG::2012-12-02 14:36:19,163::task::1151::TaskManager.Task::(prepare) Task=`263fff1d-a4c3-4b3c-8b5e-dd3f6ed1cc45`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00732421875', 'lastCheck': 1354451769.227241, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00574517250061', 'lastCheck': 1354451775.600004, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183892250061', 'lastCheck': 1354451776.5360351, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00753021240234', 'lastCheck': 1354451775.609256, 'code': 0, 'valid': True}} >Thread-28966::DEBUG::2012-12-02 14:36:19,163::task::568::TaskManager.Task::(_updateState) Task=`263fff1d-a4c3-4b3c-8b5e-dd3f6ed1cc45`::moving from state preparing -> state finished >Thread-28966::DEBUG::2012-12-02 14:36:19,163::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28966::DEBUG::2012-12-02 14:36:19,163::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28966::DEBUG::2012-12-02 14:36:19,163::task::957::TaskManager.Task::(_decref) Task=`263fff1d-a4c3-4b3c-8b5e-dd3f6ed1cc45`::ref 0 aborting False >Thread-28971::DEBUG::2012-12-02 14:36:25,872::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28971::DEBUG::2012-12-02 14:36:25,873::task::568::TaskManager.Task::(_updateState) Task=`01dcc40d-19aa-4454-abce-1421e07d1a4f`::moving from state init -> state preparing >Thread-28971::INFO::2012-12-02 14:36:25,873::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28971::INFO::2012-12-02 14:36:25,873::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28971::DEBUG::2012-12-02 14:36:25,873::task::1151::TaskManager.Task::(prepare) Task=`01dcc40d-19aa-4454-abce-1421e07d1a4f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28971::DEBUG::2012-12-02 14:36:25,873::task::568::TaskManager.Task::(_updateState) Task=`01dcc40d-19aa-4454-abce-1421e07d1a4f`::moving from state preparing -> state finished >Thread-28971::DEBUG::2012-12-02 14:36:25,873::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28971::DEBUG::2012-12-02 14:36:25,874::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28971::DEBUG::2012-12-02 14:36:25,874::task::957::TaskManager.Task::(_decref) Task=`01dcc40d-19aa-4454-abce-1421e07d1a4f`::ref 0 aborting False >Thread-28972::DEBUG::2012-12-02 14:36:25,876::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28972::DEBUG::2012-12-02 14:36:25,876::task::568::TaskManager.Task::(_updateState) Task=`5854e053-e1cf-45d9-8ab5-17dfd109e909`::moving from state init -> state preparing >Thread-28972::INFO::2012-12-02 14:36:25,877::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28972::DEBUG::2012-12-02 14:36:25,877::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`266aa92f-0f42-436e-b958-d1c03e941eb3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28972::DEBUG::2012-12-02 14:36:25,877::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28972::DEBUG::2012-12-02 14:36:25,877::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28972::DEBUG::2012-12-02 14:36:25,877::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`266aa92f-0f42-436e-b958-d1c03e941eb3`::Granted request >Thread-28972::DEBUG::2012-12-02 14:36:25,878::task::794::TaskManager.Task::(resourceAcquired) Task=`5854e053-e1cf-45d9-8ab5-17dfd109e909`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28972::DEBUG::2012-12-02 14:36:25,878::task::957::TaskManager.Task::(_decref) Task=`5854e053-e1cf-45d9-8ab5-17dfd109e909`::ref 1 aborting False >Thread-28972::INFO::2012-12-02 14:36:25,884::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28972::DEBUG::2012-12-02 14:36:25,884::task::1151::TaskManager.Task::(prepare) Task=`5854e053-e1cf-45d9-8ab5-17dfd109e909`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28972::DEBUG::2012-12-02 14:36:25,884::task::568::TaskManager.Task::(_updateState) Task=`5854e053-e1cf-45d9-8ab5-17dfd109e909`::moving from state preparing -> state finished >Thread-28972::DEBUG::2012-12-02 14:36:25,885::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28972::DEBUG::2012-12-02 14:36:25,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28972::DEBUG::2012-12-02 14:36:25,885::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28972::DEBUG::2012-12-02 14:36:25,885::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28972::DEBUG::2012-12-02 14:36:25,885::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28972::DEBUG::2012-12-02 14:36:25,885::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28972::DEBUG::2012-12-02 14:36:25,885::task::957::TaskManager.Task::(_decref) Task=`5854e053-e1cf-45d9-8ab5-17dfd109e909`::ref 0 aborting False >Thread-28974::DEBUG::2012-12-02 14:36:29,226::task::568::TaskManager.Task::(_updateState) Task=`ceed46a9-31de-436d-a068-bfe7a547e8a7`::moving from state init -> state preparing >Thread-28974::INFO::2012-12-02 14:36:29,226::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28974::INFO::2012-12-02 14:36:29,226::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00888681411743', 'lastCheck': 1354451779.245568, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.004310131073', 'lastCheck': 1354451785.6226029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00210285186768', 'lastCheck': 1354451786.547085, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00752210617065', 'lastCheck': 1354451785.6293571, 'code': 0, 'valid': True}} >Thread-28974::DEBUG::2012-12-02 14:36:29,226::task::1151::TaskManager.Task::(prepare) Task=`ceed46a9-31de-436d-a068-bfe7a547e8a7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00888681411743', 'lastCheck': 1354451779.245568, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.004310131073', 'lastCheck': 1354451785.6226029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00210285186768', 'lastCheck': 1354451786.547085, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00752210617065', 'lastCheck': 1354451785.6293571, 'code': 0, 'valid': True}} >Thread-28974::DEBUG::2012-12-02 14:36:29,226::task::568::TaskManager.Task::(_updateState) Task=`ceed46a9-31de-436d-a068-bfe7a547e8a7`::moving from state preparing -> state finished >Thread-28974::DEBUG::2012-12-02 14:36:29,226::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28974::DEBUG::2012-12-02 14:36:29,227::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28974::DEBUG::2012-12-02 14:36:29,227::task::957::TaskManager.Task::(_decref) Task=`ceed46a9-31de-436d-a068-bfe7a547e8a7`::ref 0 aborting False >Thread-28979::DEBUG::2012-12-02 14:36:36,044::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28979::DEBUG::2012-12-02 14:36:36,044::task::568::TaskManager.Task::(_updateState) Task=`7aecf7b4-ba6d-43e0-826f-e1e100b98ce4`::moving from state init -> state preparing >Thread-28979::INFO::2012-12-02 14:36:36,044::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28979::INFO::2012-12-02 14:36:36,044::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28979::DEBUG::2012-12-02 14:36:36,045::task::1151::TaskManager.Task::(prepare) Task=`7aecf7b4-ba6d-43e0-826f-e1e100b98ce4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28979::DEBUG::2012-12-02 14:36:36,045::task::568::TaskManager.Task::(_updateState) Task=`7aecf7b4-ba6d-43e0-826f-e1e100b98ce4`::moving from state preparing -> state finished >Thread-28979::DEBUG::2012-12-02 14:36:36,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28979::DEBUG::2012-12-02 14:36:36,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28979::DEBUG::2012-12-02 14:36:36,045::task::957::TaskManager.Task::(_decref) Task=`7aecf7b4-ba6d-43e0-826f-e1e100b98ce4`::ref 0 aborting False >Thread-28980::DEBUG::2012-12-02 14:36:36,048::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28980::DEBUG::2012-12-02 14:36:36,048::task::568::TaskManager.Task::(_updateState) Task=`f1a31068-ab82-41d1-a309-1cedb73a39e1`::moving from state init -> state preparing >Thread-28980::INFO::2012-12-02 14:36:36,048::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28980::DEBUG::2012-12-02 14:36:36,048::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`069cf671-3dad-4a19-847d-94cf1a605cb6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28980::DEBUG::2012-12-02 14:36:36,048::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28980::DEBUG::2012-12-02 14:36:36,048::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28980::DEBUG::2012-12-02 14:36:36,048::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`069cf671-3dad-4a19-847d-94cf1a605cb6`::Granted request >Thread-28980::DEBUG::2012-12-02 14:36:36,049::task::794::TaskManager.Task::(resourceAcquired) Task=`f1a31068-ab82-41d1-a309-1cedb73a39e1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28980::DEBUG::2012-12-02 14:36:36,049::task::957::TaskManager.Task::(_decref) Task=`f1a31068-ab82-41d1-a309-1cedb73a39e1`::ref 1 aborting False >Thread-28980::INFO::2012-12-02 14:36:36,053::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28980::DEBUG::2012-12-02 14:36:36,053::task::1151::TaskManager.Task::(prepare) Task=`f1a31068-ab82-41d1-a309-1cedb73a39e1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28980::DEBUG::2012-12-02 14:36:36,053::task::568::TaskManager.Task::(_updateState) Task=`f1a31068-ab82-41d1-a309-1cedb73a39e1`::moving from state preparing -> state finished >Thread-28980::DEBUG::2012-12-02 14:36:36,053::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28980::DEBUG::2012-12-02 14:36:36,053::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28980::DEBUG::2012-12-02 14:36:36,054::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28980::DEBUG::2012-12-02 14:36:36,054::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28980::DEBUG::2012-12-02 14:36:36,054::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28980::DEBUG::2012-12-02 14:36:36,054::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28980::DEBUG::2012-12-02 14:36:36,054::task::957::TaskManager.Task::(_decref) Task=`f1a31068-ab82-41d1-a309-1cedb73a39e1`::ref 0 aborting False >Thread-28982::DEBUG::2012-12-02 14:36:39,320::task::568::TaskManager.Task::(_updateState) Task=`24e76368-b6ef-421b-ab9e-af816456e983`::moving from state init -> state preparing >Thread-28982::INFO::2012-12-02 14:36:39,321::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28982::INFO::2012-12-02 14:36:39,321::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007817029953', 'lastCheck': 1354451799.283247, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00485014915466', 'lastCheck': 1354451795.6375101, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00242209434509', 'lastCheck': 1354451796.5594499, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00705099105835', 'lastCheck': 1354451795.6453791, 'code': 0, 'valid': True}} >Thread-28982::DEBUG::2012-12-02 14:36:39,321::task::1151::TaskManager.Task::(prepare) Task=`24e76368-b6ef-421b-ab9e-af816456e983`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.007817029953', 'lastCheck': 1354451799.283247, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00485014915466', 'lastCheck': 1354451795.6375101, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00242209434509', 'lastCheck': 1354451796.5594499, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00705099105835', 'lastCheck': 1354451795.6453791, 'code': 0, 'valid': True}} >Thread-28982::DEBUG::2012-12-02 14:36:39,321::task::568::TaskManager.Task::(_updateState) Task=`24e76368-b6ef-421b-ab9e-af816456e983`::moving from state preparing -> state finished >Thread-28982::DEBUG::2012-12-02 14:36:39,321::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28982::DEBUG::2012-12-02 14:36:39,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28982::DEBUG::2012-12-02 14:36:39,322::task::957::TaskManager.Task::(_decref) Task=`24e76368-b6ef-421b-ab9e-af816456e983`::ref 0 aborting False >Thread-28987::DEBUG::2012-12-02 14:36:46,149::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28987::DEBUG::2012-12-02 14:36:46,150::task::568::TaskManager.Task::(_updateState) Task=`a7874d40-3cbf-4ce4-9e84-af333db575a6`::moving from state init -> state preparing >Thread-28987::INFO::2012-12-02 14:36:46,150::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28987::INFO::2012-12-02 14:36:46,150::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28987::DEBUG::2012-12-02 14:36:46,150::task::1151::TaskManager.Task::(prepare) Task=`a7874d40-3cbf-4ce4-9e84-af333db575a6`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28987::DEBUG::2012-12-02 14:36:46,150::task::568::TaskManager.Task::(_updateState) Task=`a7874d40-3cbf-4ce4-9e84-af333db575a6`::moving from state preparing -> state finished >Thread-28987::DEBUG::2012-12-02 14:36:46,150::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28987::DEBUG::2012-12-02 14:36:46,150::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28987::DEBUG::2012-12-02 14:36:46,151::task::957::TaskManager.Task::(_decref) Task=`a7874d40-3cbf-4ce4-9e84-af333db575a6`::ref 0 aborting False >Thread-28988::DEBUG::2012-12-02 14:36:46,153::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28988::DEBUG::2012-12-02 14:36:46,153::task::568::TaskManager.Task::(_updateState) Task=`2ea2662e-e026-4f04-b39d-32217977d2a6`::moving from state init -> state preparing >Thread-28988::INFO::2012-12-02 14:36:46,153::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28988::DEBUG::2012-12-02 14:36:46,154::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`31133761-78e7-46a2-8d98-8f2c78cb4b0d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28988::DEBUG::2012-12-02 14:36:46,154::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28988::DEBUG::2012-12-02 14:36:46,154::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28988::DEBUG::2012-12-02 14:36:46,154::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`31133761-78e7-46a2-8d98-8f2c78cb4b0d`::Granted request >Thread-28988::DEBUG::2012-12-02 14:36:46,154::task::794::TaskManager.Task::(resourceAcquired) Task=`2ea2662e-e026-4f04-b39d-32217977d2a6`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28988::DEBUG::2012-12-02 14:36:46,155::task::957::TaskManager.Task::(_decref) Task=`2ea2662e-e026-4f04-b39d-32217977d2a6`::ref 1 aborting False >Thread-28988::INFO::2012-12-02 14:36:46,159::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28988::DEBUG::2012-12-02 14:36:46,159::task::1151::TaskManager.Task::(prepare) Task=`2ea2662e-e026-4f04-b39d-32217977d2a6`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28988::DEBUG::2012-12-02 14:36:46,159::task::568::TaskManager.Task::(_updateState) Task=`2ea2662e-e026-4f04-b39d-32217977d2a6`::moving from state preparing -> state finished >Thread-28988::DEBUG::2012-12-02 14:36:46,159::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28988::DEBUG::2012-12-02 14:36:46,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28988::DEBUG::2012-12-02 14:36:46,159::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28988::DEBUG::2012-12-02 14:36:46,160::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28988::DEBUG::2012-12-02 14:36:46,160::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28988::DEBUG::2012-12-02 14:36:46,160::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28988::DEBUG::2012-12-02 14:36:46,160::task::957::TaskManager.Task::(_decref) Task=`2ea2662e-e026-4f04-b39d-32217977d2a6`::ref 0 aborting False >Thread-28990::DEBUG::2012-12-02 14:36:49,385::task::568::TaskManager.Task::(_updateState) Task=`d499d0ed-6ce2-47b1-b140-adfb8f3aadde`::moving from state init -> state preparing >Thread-28990::INFO::2012-12-02 14:36:49,385::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28990::INFO::2012-12-02 14:36:49,385::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00342702865601', 'lastCheck': 1354451809.2951651, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00405812263489', 'lastCheck': 1354451805.656038, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171709060669', 'lastCheck': 1354451806.5697689, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00781106948853', 'lastCheck': 1354451805.6663511, 'code': 0, 'valid': True}} >Thread-28990::DEBUG::2012-12-02 14:36:49,385::task::1151::TaskManager.Task::(prepare) Task=`d499d0ed-6ce2-47b1-b140-adfb8f3aadde`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00342702865601', 'lastCheck': 1354451809.2951651, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00405812263489', 'lastCheck': 1354451805.656038, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171709060669', 'lastCheck': 1354451806.5697689, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00781106948853', 'lastCheck': 1354451805.6663511, 'code': 0, 'valid': True}} >Thread-28990::DEBUG::2012-12-02 14:36:49,385::task::568::TaskManager.Task::(_updateState) Task=`d499d0ed-6ce2-47b1-b140-adfb8f3aadde`::moving from state preparing -> state finished >Thread-28990::DEBUG::2012-12-02 14:36:49,385::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28990::DEBUG::2012-12-02 14:36:49,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28990::DEBUG::2012-12-02 14:36:49,386::task::957::TaskManager.Task::(_decref) Task=`d499d0ed-6ce2-47b1-b140-adfb8f3aadde`::ref 0 aborting False >Thread-28995::DEBUG::2012-12-02 14:36:56,279::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28995::DEBUG::2012-12-02 14:36:56,280::task::568::TaskManager.Task::(_updateState) Task=`ce295c32-84a0-460b-8fa4-f97850ddb931`::moving from state init -> state preparing >Thread-28995::INFO::2012-12-02 14:36:56,280::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28995::INFO::2012-12-02 14:36:56,280::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28995::DEBUG::2012-12-02 14:36:56,280::task::1151::TaskManager.Task::(prepare) Task=`ce295c32-84a0-460b-8fa4-f97850ddb931`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-28995::DEBUG::2012-12-02 14:36:56,280::task::568::TaskManager.Task::(_updateState) Task=`ce295c32-84a0-460b-8fa4-f97850ddb931`::moving from state preparing -> state finished >Thread-28995::DEBUG::2012-12-02 14:36:56,281::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28995::DEBUG::2012-12-02 14:36:56,281::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28995::DEBUG::2012-12-02 14:36:56,281::task::957::TaskManager.Task::(_decref) Task=`ce295c32-84a0-460b-8fa4-f97850ddb931`::ref 0 aborting False >Thread-28996::DEBUG::2012-12-02 14:36:56,283::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-28996::DEBUG::2012-12-02 14:36:56,283::task::568::TaskManager.Task::(_updateState) Task=`da52bc9a-2f47-41d7-a006-183e8bcd3094`::moving from state init -> state preparing >Thread-28996::INFO::2012-12-02 14:36:56,284::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-28996::DEBUG::2012-12-02 14:36:56,284::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f0088cef-4758-41a7-9453-0c42a24714d4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-28996::DEBUG::2012-12-02 14:36:56,284::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-28996::DEBUG::2012-12-02 14:36:56,284::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-28996::DEBUG::2012-12-02 14:36:56,284::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f0088cef-4758-41a7-9453-0c42a24714d4`::Granted request >Thread-28996::DEBUG::2012-12-02 14:36:56,285::task::794::TaskManager.Task::(resourceAcquired) Task=`da52bc9a-2f47-41d7-a006-183e8bcd3094`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-28996::DEBUG::2012-12-02 14:36:56,285::task::957::TaskManager.Task::(_decref) Task=`da52bc9a-2f47-41d7-a006-183e8bcd3094`::ref 1 aborting False >Thread-28996::INFO::2012-12-02 14:36:56,290::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28996::DEBUG::2012-12-02 14:36:56,290::task::1151::TaskManager.Task::(prepare) Task=`da52bc9a-2f47-41d7-a006-183e8bcd3094`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-28996::DEBUG::2012-12-02 14:36:56,290::task::568::TaskManager.Task::(_updateState) Task=`da52bc9a-2f47-41d7-a006-183e8bcd3094`::moving from state preparing -> state finished >Thread-28996::DEBUG::2012-12-02 14:36:56,290::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-28996::DEBUG::2012-12-02 14:36:56,290::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28996::DEBUG::2012-12-02 14:36:56,290::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-28996::DEBUG::2012-12-02 14:36:56,290::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-28996::DEBUG::2012-12-02 14:36:56,290::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-28996::DEBUG::2012-12-02 14:36:56,290::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-28996::DEBUG::2012-12-02 14:36:56,290::task::957::TaskManager.Task::(_decref) Task=`da52bc9a-2f47-41d7-a006-183e8bcd3094`::ref 0 aborting False >Thread-28998::DEBUG::2012-12-02 14:36:59,448::task::568::TaskManager.Task::(_updateState) Task=`33fdb2fa-3a42-4bee-8c58-d9bda81143e9`::moving from state init -> state preparing >Thread-28998::INFO::2012-12-02 14:36:59,449::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-28998::INFO::2012-12-02 14:36:59,449::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0102410316467', 'lastCheck': 1354451819.3149769, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206899642944', 'lastCheck': 1354451815.6740661, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00222301483154', 'lastCheck': 1354451816.5811329, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00875997543335', 'lastCheck': 1354451815.685082, 'code': 0, 'valid': True}} >Thread-28998::DEBUG::2012-12-02 14:36:59,449::task::1151::TaskManager.Task::(prepare) Task=`33fdb2fa-3a42-4bee-8c58-d9bda81143e9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0102410316467', 'lastCheck': 1354451819.3149769, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206899642944', 'lastCheck': 1354451815.6740661, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00222301483154', 'lastCheck': 1354451816.5811329, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00875997543335', 'lastCheck': 1354451815.685082, 'code': 0, 'valid': True}} >Thread-28998::DEBUG::2012-12-02 14:36:59,449::task::568::TaskManager.Task::(_updateState) Task=`33fdb2fa-3a42-4bee-8c58-d9bda81143e9`::moving from state preparing -> state finished >Thread-28998::DEBUG::2012-12-02 14:36:59,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-28998::DEBUG::2012-12-02 14:36:59,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-28998::DEBUG::2012-12-02 14:36:59,450::task::957::TaskManager.Task::(_decref) Task=`33fdb2fa-3a42-4bee-8c58-d9bda81143e9`::ref 0 aborting False >Thread-29003::DEBUG::2012-12-02 14:37:06,416::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29003::DEBUG::2012-12-02 14:37:06,417::task::568::TaskManager.Task::(_updateState) Task=`ce35f49b-1bc7-4379-a5b2-b900f41e6fb5`::moving from state init -> state preparing >Thread-29003::INFO::2012-12-02 14:37:06,417::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29003::INFO::2012-12-02 14:37:06,417::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29003::DEBUG::2012-12-02 14:37:06,417::task::1151::TaskManager.Task::(prepare) Task=`ce35f49b-1bc7-4379-a5b2-b900f41e6fb5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29003::DEBUG::2012-12-02 14:37:06,417::task::568::TaskManager.Task::(_updateState) Task=`ce35f49b-1bc7-4379-a5b2-b900f41e6fb5`::moving from state preparing -> state finished >Thread-29003::DEBUG::2012-12-02 14:37:06,418::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29003::DEBUG::2012-12-02 14:37:06,418::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29003::DEBUG::2012-12-02 14:37:06,418::task::957::TaskManager.Task::(_decref) Task=`ce35f49b-1bc7-4379-a5b2-b900f41e6fb5`::ref 0 aborting False >Thread-29004::DEBUG::2012-12-02 14:37:06,420::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29004::DEBUG::2012-12-02 14:37:06,421::task::568::TaskManager.Task::(_updateState) Task=`42f6cf55-6f79-4779-8bae-29fd87b9edcd`::moving from state init -> state preparing >Thread-29004::INFO::2012-12-02 14:37:06,421::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29004::DEBUG::2012-12-02 14:37:06,421::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`77f58256-a763-409b-b316-24c805c8b7b8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29004::DEBUG::2012-12-02 14:37:06,421::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29004::DEBUG::2012-12-02 14:37:06,422::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29004::DEBUG::2012-12-02 14:37:06,422::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`77f58256-a763-409b-b316-24c805c8b7b8`::Granted request >Thread-29004::DEBUG::2012-12-02 14:37:06,422::task::794::TaskManager.Task::(resourceAcquired) Task=`42f6cf55-6f79-4779-8bae-29fd87b9edcd`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29004::DEBUG::2012-12-02 14:37:06,422::task::957::TaskManager.Task::(_decref) Task=`42f6cf55-6f79-4779-8bae-29fd87b9edcd`::ref 1 aborting False >Thread-29004::INFO::2012-12-02 14:37:06,427::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29004::DEBUG::2012-12-02 14:37:06,428::task::1151::TaskManager.Task::(prepare) Task=`42f6cf55-6f79-4779-8bae-29fd87b9edcd`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29004::DEBUG::2012-12-02 14:37:06,428::task::568::TaskManager.Task::(_updateState) Task=`42f6cf55-6f79-4779-8bae-29fd87b9edcd`::moving from state preparing -> state finished >Thread-29004::DEBUG::2012-12-02 14:37:06,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29004::DEBUG::2012-12-02 14:37:06,428::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29004::DEBUG::2012-12-02 14:37:06,428::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29004::DEBUG::2012-12-02 14:37:06,428::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29004::DEBUG::2012-12-02 14:37:06,428::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29004::DEBUG::2012-12-02 14:37:06,428::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29004::DEBUG::2012-12-02 14:37:06,428::task::957::TaskManager.Task::(_decref) Task=`42f6cf55-6f79-4779-8bae-29fd87b9edcd`::ref 0 aborting False >Thread-29006::DEBUG::2012-12-02 14:37:09,564::task::568::TaskManager.Task::(_updateState) Task=`217da143-41e4-4865-af51-e56a3566f5e3`::moving from state init -> state preparing >Thread-29006::INFO::2012-12-02 14:37:09,564::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29006::INFO::2012-12-02 14:37:09,564::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00681495666504', 'lastCheck': 1354451829.3316381, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00259494781494', 'lastCheck': 1354451825.6856639, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00165820121765', 'lastCheck': 1354451826.5905709, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00873994827271', 'lastCheck': 1354451825.7028711, 'code': 0, 'valid': True}} >Thread-29006::DEBUG::2012-12-02 14:37:09,564::task::1151::TaskManager.Task::(prepare) Task=`217da143-41e4-4865-af51-e56a3566f5e3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00681495666504', 'lastCheck': 1354451829.3316381, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00259494781494', 'lastCheck': 1354451825.6856639, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00165820121765', 'lastCheck': 1354451826.5905709, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00873994827271', 'lastCheck': 1354451825.7028711, 'code': 0, 'valid': True}} >Thread-29006::DEBUG::2012-12-02 14:37:09,564::task::568::TaskManager.Task::(_updateState) Task=`217da143-41e4-4865-af51-e56a3566f5e3`::moving from state preparing -> state finished >Thread-29006::DEBUG::2012-12-02 14:37:09,564::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29006::DEBUG::2012-12-02 14:37:09,565::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29006::DEBUG::2012-12-02 14:37:09,565::task::957::TaskManager.Task::(_decref) Task=`217da143-41e4-4865-af51-e56a3566f5e3`::ref 0 aborting False >Thread-29011::DEBUG::2012-12-02 14:37:16,556::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29011::DEBUG::2012-12-02 14:37:16,556::task::568::TaskManager.Task::(_updateState) Task=`5e46c379-c51a-4479-b2d7-569f44380209`::moving from state init -> state preparing >Thread-29011::INFO::2012-12-02 14:37:16,556::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29011::INFO::2012-12-02 14:37:16,556::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29011::DEBUG::2012-12-02 14:37:16,557::task::1151::TaskManager.Task::(prepare) Task=`5e46c379-c51a-4479-b2d7-569f44380209`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29011::DEBUG::2012-12-02 14:37:16,557::task::568::TaskManager.Task::(_updateState) Task=`5e46c379-c51a-4479-b2d7-569f44380209`::moving from state preparing -> state finished >Thread-29011::DEBUG::2012-12-02 14:37:16,557::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29011::DEBUG::2012-12-02 14:37:16,557::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29011::DEBUG::2012-12-02 14:37:16,557::task::957::TaskManager.Task::(_decref) Task=`5e46c379-c51a-4479-b2d7-569f44380209`::ref 0 aborting False >Thread-29012::DEBUG::2012-12-02 14:37:16,560::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29012::DEBUG::2012-12-02 14:37:16,560::task::568::TaskManager.Task::(_updateState) Task=`9220196d-579e-41ee-b76c-0f0e9f9ca52e`::moving from state init -> state preparing >Thread-29012::INFO::2012-12-02 14:37:16,560::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29012::DEBUG::2012-12-02 14:37:16,560::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`13d27f0e-42d6-4ba0-80ae-9c917c39ebb6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29012::DEBUG::2012-12-02 14:37:16,561::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29012::DEBUG::2012-12-02 14:37:16,561::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29012::DEBUG::2012-12-02 14:37:16,561::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`13d27f0e-42d6-4ba0-80ae-9c917c39ebb6`::Granted request >Thread-29012::DEBUG::2012-12-02 14:37:16,561::task::794::TaskManager.Task::(resourceAcquired) Task=`9220196d-579e-41ee-b76c-0f0e9f9ca52e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29012::DEBUG::2012-12-02 14:37:16,561::task::957::TaskManager.Task::(_decref) Task=`9220196d-579e-41ee-b76c-0f0e9f9ca52e`::ref 1 aborting False >Thread-29012::INFO::2012-12-02 14:37:16,567::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29012::DEBUG::2012-12-02 14:37:16,567::task::1151::TaskManager.Task::(prepare) Task=`9220196d-579e-41ee-b76c-0f0e9f9ca52e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29012::DEBUG::2012-12-02 14:37:16,567::task::568::TaskManager.Task::(_updateState) Task=`9220196d-579e-41ee-b76c-0f0e9f9ca52e`::moving from state preparing -> state finished >Thread-29012::DEBUG::2012-12-02 14:37:16,567::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29012::DEBUG::2012-12-02 14:37:16,567::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29012::DEBUG::2012-12-02 14:37:16,567::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29012::DEBUG::2012-12-02 14:37:16,567::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29012::DEBUG::2012-12-02 14:37:16,568::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29012::DEBUG::2012-12-02 14:37:16,568::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29012::DEBUG::2012-12-02 14:37:16,568::task::957::TaskManager.Task::(_decref) Task=`9220196d-579e-41ee-b76c-0f0e9f9ca52e`::ref 0 aborting False >Thread-29014::DEBUG::2012-12-02 14:37:19,703::task::568::TaskManager.Task::(_updateState) Task=`ce85db60-36ff-4058-ad17-d5db91768596`::moving from state init -> state preparing >Thread-29014::INFO::2012-12-02 14:37:19,704::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29014::INFO::2012-12-02 14:37:19,704::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00798988342285', 'lastCheck': 1354451839.3496161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00340604782104', 'lastCheck': 1354451835.700119, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00301599502563', 'lastCheck': 1354451836.6031129, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0069408416748', 'lastCheck': 1354451835.7186301, 'code': 0, 'valid': True}} >Thread-29014::DEBUG::2012-12-02 14:37:19,704::task::1151::TaskManager.Task::(prepare) Task=`ce85db60-36ff-4058-ad17-d5db91768596`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00798988342285', 'lastCheck': 1354451839.3496161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00340604782104', 'lastCheck': 1354451835.700119, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00301599502563', 'lastCheck': 1354451836.6031129, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0069408416748', 'lastCheck': 1354451835.7186301, 'code': 0, 'valid': True}} >Thread-29014::DEBUG::2012-12-02 14:37:19,704::task::568::TaskManager.Task::(_updateState) Task=`ce85db60-36ff-4058-ad17-d5db91768596`::moving from state preparing -> state finished >Thread-29014::DEBUG::2012-12-02 14:37:19,704::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29014::DEBUG::2012-12-02 14:37:19,704::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29014::DEBUG::2012-12-02 14:37:19,705::task::957::TaskManager.Task::(_decref) Task=`ce85db60-36ff-4058-ad17-d5db91768596`::ref 0 aborting False >Thread-29019::DEBUG::2012-12-02 14:37:26,710::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29019::DEBUG::2012-12-02 14:37:26,710::task::568::TaskManager.Task::(_updateState) Task=`41dccb96-ad72-48dc-8acc-36b4dcab0637`::moving from state init -> state preparing >Thread-29019::INFO::2012-12-02 14:37:26,710::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29019::INFO::2012-12-02 14:37:26,710::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29019::DEBUG::2012-12-02 14:37:26,710::task::1151::TaskManager.Task::(prepare) Task=`41dccb96-ad72-48dc-8acc-36b4dcab0637`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29019::DEBUG::2012-12-02 14:37:26,711::task::568::TaskManager.Task::(_updateState) Task=`41dccb96-ad72-48dc-8acc-36b4dcab0637`::moving from state preparing -> state finished >Thread-29019::DEBUG::2012-12-02 14:37:26,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29019::DEBUG::2012-12-02 14:37:26,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29019::DEBUG::2012-12-02 14:37:26,711::task::957::TaskManager.Task::(_decref) Task=`41dccb96-ad72-48dc-8acc-36b4dcab0637`::ref 0 aborting False >Thread-29020::DEBUG::2012-12-02 14:37:26,714::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29020::DEBUG::2012-12-02 14:37:26,714::task::568::TaskManager.Task::(_updateState) Task=`ccc02520-1702-4127-98bb-3c416f62428a`::moving from state init -> state preparing >Thread-29020::INFO::2012-12-02 14:37:26,714::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29020::DEBUG::2012-12-02 14:37:26,714::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ef6da27b-a1d6-405c-ba97-974c30a24a0d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29020::DEBUG::2012-12-02 14:37:26,714::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29020::DEBUG::2012-12-02 14:37:26,715::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29020::DEBUG::2012-12-02 14:37:26,715::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ef6da27b-a1d6-405c-ba97-974c30a24a0d`::Granted request >Thread-29020::DEBUG::2012-12-02 14:37:26,715::task::794::TaskManager.Task::(resourceAcquired) Task=`ccc02520-1702-4127-98bb-3c416f62428a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29020::DEBUG::2012-12-02 14:37:26,715::task::957::TaskManager.Task::(_decref) Task=`ccc02520-1702-4127-98bb-3c416f62428a`::ref 1 aborting False >Thread-29020::INFO::2012-12-02 14:37:26,720::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29020::DEBUG::2012-12-02 14:37:26,720::task::1151::TaskManager.Task::(prepare) Task=`ccc02520-1702-4127-98bb-3c416f62428a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29020::DEBUG::2012-12-02 14:37:26,720::task::568::TaskManager.Task::(_updateState) Task=`ccc02520-1702-4127-98bb-3c416f62428a`::moving from state preparing -> state finished >Thread-29020::DEBUG::2012-12-02 14:37:26,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29020::DEBUG::2012-12-02 14:37:26,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29020::DEBUG::2012-12-02 14:37:26,720::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29020::DEBUG::2012-12-02 14:37:26,720::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29020::DEBUG::2012-12-02 14:37:26,720::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29020::DEBUG::2012-12-02 14:37:26,720::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29020::DEBUG::2012-12-02 14:37:26,720::task::957::TaskManager.Task::(_decref) Task=`ccc02520-1702-4127-98bb-3c416f62428a`::ref 0 aborting False >Thread-29022::DEBUG::2012-12-02 14:37:29,790::task::568::TaskManager.Task::(_updateState) Task=`71a3f994-125c-42fd-93ff-6935f754cecc`::moving from state init -> state preparing >Thread-29022::INFO::2012-12-02 14:37:29,790::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29022::INFO::2012-12-02 14:37:29,790::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00808691978455', 'lastCheck': 1354451849.3672161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00205707550049', 'lastCheck': 1354451845.715765, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0017421245575', 'lastCheck': 1354451846.6124079, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00851202011108', 'lastCheck': 1354451845.7367511, 'code': 0, 'valid': True}} >Thread-29022::DEBUG::2012-12-02 14:37:29,791::task::1151::TaskManager.Task::(prepare) Task=`71a3f994-125c-42fd-93ff-6935f754cecc`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00808691978455', 'lastCheck': 1354451849.3672161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00205707550049', 'lastCheck': 1354451845.715765, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0017421245575', 'lastCheck': 1354451846.6124079, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00851202011108', 'lastCheck': 1354451845.7367511, 'code': 0, 'valid': True}} >Thread-29022::DEBUG::2012-12-02 14:37:29,791::task::568::TaskManager.Task::(_updateState) Task=`71a3f994-125c-42fd-93ff-6935f754cecc`::moving from state preparing -> state finished >Thread-29022::DEBUG::2012-12-02 14:37:29,791::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29022::DEBUG::2012-12-02 14:37:29,791::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29022::DEBUG::2012-12-02 14:37:29,791::task::957::TaskManager.Task::(_decref) Task=`71a3f994-125c-42fd-93ff-6935f754cecc`::ref 0 aborting False >Thread-29027::DEBUG::2012-12-02 14:37:36,849::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29027::DEBUG::2012-12-02 14:37:36,850::task::568::TaskManager.Task::(_updateState) Task=`2595d632-4ce4-4f28-a7d7-9be317a5fffb`::moving from state init -> state preparing >Thread-29027::INFO::2012-12-02 14:37:36,850::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29027::INFO::2012-12-02 14:37:36,850::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29027::DEBUG::2012-12-02 14:37:36,850::task::1151::TaskManager.Task::(prepare) Task=`2595d632-4ce4-4f28-a7d7-9be317a5fffb`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29027::DEBUG::2012-12-02 14:37:36,850::task::568::TaskManager.Task::(_updateState) Task=`2595d632-4ce4-4f28-a7d7-9be317a5fffb`::moving from state preparing -> state finished >Thread-29027::DEBUG::2012-12-02 14:37:36,851::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29027::DEBUG::2012-12-02 14:37:36,851::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29027::DEBUG::2012-12-02 14:37:36,851::task::957::TaskManager.Task::(_decref) Task=`2595d632-4ce4-4f28-a7d7-9be317a5fffb`::ref 0 aborting False >Thread-29028::DEBUG::2012-12-02 14:37:36,853::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29028::DEBUG::2012-12-02 14:37:36,853::task::568::TaskManager.Task::(_updateState) Task=`23d2ad8b-f81c-45e3-a391-df22867caeb9`::moving from state init -> state preparing >Thread-29028::INFO::2012-12-02 14:37:36,854::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29028::DEBUG::2012-12-02 14:37:36,854::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3bcc0a79-43b3-49dc-92b6-3506139357b2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29028::DEBUG::2012-12-02 14:37:36,854::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29028::DEBUG::2012-12-02 14:37:36,854::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29028::DEBUG::2012-12-02 14:37:36,854::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3bcc0a79-43b3-49dc-92b6-3506139357b2`::Granted request >Thread-29028::DEBUG::2012-12-02 14:37:36,855::task::794::TaskManager.Task::(resourceAcquired) Task=`23d2ad8b-f81c-45e3-a391-df22867caeb9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29028::DEBUG::2012-12-02 14:37:36,855::task::957::TaskManager.Task::(_decref) Task=`23d2ad8b-f81c-45e3-a391-df22867caeb9`::ref 1 aborting False >Thread-29028::INFO::2012-12-02 14:37:36,860::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29028::DEBUG::2012-12-02 14:37:36,860::task::1151::TaskManager.Task::(prepare) Task=`23d2ad8b-f81c-45e3-a391-df22867caeb9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29028::DEBUG::2012-12-02 14:37:36,860::task::568::TaskManager.Task::(_updateState) Task=`23d2ad8b-f81c-45e3-a391-df22867caeb9`::moving from state preparing -> state finished >Thread-29028::DEBUG::2012-12-02 14:37:36,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29028::DEBUG::2012-12-02 14:37:36,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29028::DEBUG::2012-12-02 14:37:36,860::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29028::DEBUG::2012-12-02 14:37:36,860::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29028::DEBUG::2012-12-02 14:37:36,860::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29028::DEBUG::2012-12-02 14:37:36,860::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29028::DEBUG::2012-12-02 14:37:36,860::task::957::TaskManager.Task::(_decref) Task=`23d2ad8b-f81c-45e3-a391-df22867caeb9`::ref 0 aborting False >Thread-29030::DEBUG::2012-12-02 14:37:39,920::task::568::TaskManager.Task::(_updateState) Task=`e7f975d4-6d1b-4f24-bf3d-19a7ece09375`::moving from state init -> state preparing >Thread-29030::INFO::2012-12-02 14:37:39,921::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29030::INFO::2012-12-02 14:37:39,921::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00853800773621', 'lastCheck': 1354451859.38765, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00311803817749', 'lastCheck': 1354451855.7285659, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00277185440063', 'lastCheck': 1354451856.6237619, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00697994232178', 'lastCheck': 1354451855.752445, 'code': 0, 'valid': True}} >Thread-29030::DEBUG::2012-12-02 14:37:39,921::task::1151::TaskManager.Task::(prepare) Task=`e7f975d4-6d1b-4f24-bf3d-19a7ece09375`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00853800773621', 'lastCheck': 1354451859.38765, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00311803817749', 'lastCheck': 1354451855.7285659, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00277185440063', 'lastCheck': 1354451856.6237619, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00697994232178', 'lastCheck': 1354451855.752445, 'code': 0, 'valid': True}} >Thread-29030::DEBUG::2012-12-02 14:37:39,921::task::568::TaskManager.Task::(_updateState) Task=`e7f975d4-6d1b-4f24-bf3d-19a7ece09375`::moving from state preparing -> state finished >Thread-29030::DEBUG::2012-12-02 14:37:39,921::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29030::DEBUG::2012-12-02 14:37:39,922::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29030::DEBUG::2012-12-02 14:37:39,922::task::957::TaskManager.Task::(_decref) Task=`e7f975d4-6d1b-4f24-bf3d-19a7ece09375`::ref 0 aborting False >Thread-29035::DEBUG::2012-12-02 14:37:46,987::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29035::DEBUG::2012-12-02 14:37:46,987::task::568::TaskManager.Task::(_updateState) Task=`a9070eea-74d3-4fce-94c2-20c094024c84`::moving from state init -> state preparing >Thread-29035::INFO::2012-12-02 14:37:46,987::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29035::INFO::2012-12-02 14:37:46,987::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29035::DEBUG::2012-12-02 14:37:46,987::task::1151::TaskManager.Task::(prepare) Task=`a9070eea-74d3-4fce-94c2-20c094024c84`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29035::DEBUG::2012-12-02 14:37:46,988::task::568::TaskManager.Task::(_updateState) Task=`a9070eea-74d3-4fce-94c2-20c094024c84`::moving from state preparing -> state finished >Thread-29035::DEBUG::2012-12-02 14:37:46,988::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29035::DEBUG::2012-12-02 14:37:46,988::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29035::DEBUG::2012-12-02 14:37:46,988::task::957::TaskManager.Task::(_decref) Task=`a9070eea-74d3-4fce-94c2-20c094024c84`::ref 0 aborting False >Thread-29036::DEBUG::2012-12-02 14:37:46,991::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29036::DEBUG::2012-12-02 14:37:46,991::task::568::TaskManager.Task::(_updateState) Task=`4a5f1fea-04d9-464e-85f8-0ba4f18d7f27`::moving from state init -> state preparing >Thread-29036::INFO::2012-12-02 14:37:46,991::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29036::DEBUG::2012-12-02 14:37:46,991::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bc820912-5254-4af1-8301-ce333a5c7f9c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29036::DEBUG::2012-12-02 14:37:46,991::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29036::DEBUG::2012-12-02 14:37:46,992::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29036::DEBUG::2012-12-02 14:37:46,992::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`bc820912-5254-4af1-8301-ce333a5c7f9c`::Granted request >Thread-29036::DEBUG::2012-12-02 14:37:46,992::task::794::TaskManager.Task::(resourceAcquired) Task=`4a5f1fea-04d9-464e-85f8-0ba4f18d7f27`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29036::DEBUG::2012-12-02 14:37:46,992::task::957::TaskManager.Task::(_decref) Task=`4a5f1fea-04d9-464e-85f8-0ba4f18d7f27`::ref 1 aborting False >Thread-29036::INFO::2012-12-02 14:37:46,997::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29036::DEBUG::2012-12-02 14:37:46,998::task::1151::TaskManager.Task::(prepare) Task=`4a5f1fea-04d9-464e-85f8-0ba4f18d7f27`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29036::DEBUG::2012-12-02 14:37:46,998::task::568::TaskManager.Task::(_updateState) Task=`4a5f1fea-04d9-464e-85f8-0ba4f18d7f27`::moving from state preparing -> state finished >Thread-29036::DEBUG::2012-12-02 14:37:46,998::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29036::DEBUG::2012-12-02 14:37:46,998::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29036::DEBUG::2012-12-02 14:37:46,998::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29036::DEBUG::2012-12-02 14:37:46,998::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29036::DEBUG::2012-12-02 14:37:46,998::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29036::DEBUG::2012-12-02 14:37:46,998::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29036::DEBUG::2012-12-02 14:37:46,998::task::957::TaskManager.Task::(_decref) Task=`4a5f1fea-04d9-464e-85f8-0ba4f18d7f27`::ref 0 aborting False >Thread-29038::DEBUG::2012-12-02 14:37:50,024::task::568::TaskManager.Task::(_updateState) Task=`c5a4be77-a171-4d87-907a-940a9042b16d`::moving from state init -> state preparing >Thread-29038::INFO::2012-12-02 14:37:50,024::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29038::INFO::2012-12-02 14:37:50,025::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00692796707153', 'lastCheck': 1354451869.4043219, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00228905677795', 'lastCheck': 1354451865.744334, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178599357605', 'lastCheck': 1354451866.633431, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821185112', 'lastCheck': 1354451865.7709939, 'code': 0, 'valid': True}} >Thread-29038::DEBUG::2012-12-02 14:37:50,025::task::1151::TaskManager.Task::(prepare) Task=`c5a4be77-a171-4d87-907a-940a9042b16d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00692796707153', 'lastCheck': 1354451869.4043219, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00228905677795', 'lastCheck': 1354451865.744334, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178599357605', 'lastCheck': 1354451866.633431, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821185112', 'lastCheck': 1354451865.7709939, 'code': 0, 'valid': True}} >Thread-29038::DEBUG::2012-12-02 14:37:50,025::task::568::TaskManager.Task::(_updateState) Task=`c5a4be77-a171-4d87-907a-940a9042b16d`::moving from state preparing -> state finished >Thread-29038::DEBUG::2012-12-02 14:37:50,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29038::DEBUG::2012-12-02 14:37:50,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29038::DEBUG::2012-12-02 14:37:50,025::task::957::TaskManager.Task::(_decref) Task=`c5a4be77-a171-4d87-907a-940a9042b16d`::ref 0 aborting False >Thread-29043::DEBUG::2012-12-02 14:37:57,158::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29043::DEBUG::2012-12-02 14:37:57,158::task::568::TaskManager.Task::(_updateState) Task=`fe2bf4b3-8d6e-4cca-8882-303f19a77fc4`::moving from state init -> state preparing >Thread-29043::INFO::2012-12-02 14:37:57,158::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29043::INFO::2012-12-02 14:37:57,158::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29043::DEBUG::2012-12-02 14:37:57,159::task::1151::TaskManager.Task::(prepare) Task=`fe2bf4b3-8d6e-4cca-8882-303f19a77fc4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29043::DEBUG::2012-12-02 14:37:57,159::task::568::TaskManager.Task::(_updateState) Task=`fe2bf4b3-8d6e-4cca-8882-303f19a77fc4`::moving from state preparing -> state finished >Thread-29043::DEBUG::2012-12-02 14:37:57,159::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29043::DEBUG::2012-12-02 14:37:57,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29043::DEBUG::2012-12-02 14:37:57,159::task::957::TaskManager.Task::(_decref) Task=`fe2bf4b3-8d6e-4cca-8882-303f19a77fc4`::ref 0 aborting False >Thread-29044::DEBUG::2012-12-02 14:37:57,162::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29044::DEBUG::2012-12-02 14:37:57,162::task::568::TaskManager.Task::(_updateState) Task=`6731a018-7ca2-4c00-9a86-ef63361c64a2`::moving from state init -> state preparing >Thread-29044::INFO::2012-12-02 14:37:57,162::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29044::DEBUG::2012-12-02 14:37:57,162::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3f41a222-5fe5-4643-be8c-09ffd8bdc071`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29044::DEBUG::2012-12-02 14:37:57,162::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29044::DEBUG::2012-12-02 14:37:57,163::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29044::DEBUG::2012-12-02 14:37:57,163::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3f41a222-5fe5-4643-be8c-09ffd8bdc071`::Granted request >Thread-29044::DEBUG::2012-12-02 14:37:57,163::task::794::TaskManager.Task::(resourceAcquired) Task=`6731a018-7ca2-4c00-9a86-ef63361c64a2`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29044::DEBUG::2012-12-02 14:37:57,163::task::957::TaskManager.Task::(_decref) Task=`6731a018-7ca2-4c00-9a86-ef63361c64a2`::ref 1 aborting False >Thread-29044::INFO::2012-12-02 14:37:57,168::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29044::DEBUG::2012-12-02 14:37:57,168::task::1151::TaskManager.Task::(prepare) Task=`6731a018-7ca2-4c00-9a86-ef63361c64a2`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29044::DEBUG::2012-12-02 14:37:57,168::task::568::TaskManager.Task::(_updateState) Task=`6731a018-7ca2-4c00-9a86-ef63361c64a2`::moving from state preparing -> state finished >Thread-29044::DEBUG::2012-12-02 14:37:57,168::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29044::DEBUG::2012-12-02 14:37:57,169::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29044::DEBUG::2012-12-02 14:37:57,169::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29044::DEBUG::2012-12-02 14:37:57,169::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29044::DEBUG::2012-12-02 14:37:57,169::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29044::DEBUG::2012-12-02 14:37:57,169::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29044::DEBUG::2012-12-02 14:37:57,169::task::957::TaskManager.Task::(_decref) Task=`6731a018-7ca2-4c00-9a86-ef63361c64a2`::ref 0 aborting False >Thread-29046::DEBUG::2012-12-02 14:38:00,113::task::568::TaskManager.Task::(_updateState) Task=`34b0b8c7-1066-42a0-b3dc-15b884b4a10a`::moving from state init -> state preparing >Thread-29046::INFO::2012-12-02 14:38:00,114::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29046::INFO::2012-12-02 14:38:00,114::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00877785682678', 'lastCheck': 1354451879.4206901, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00182795524597', 'lastCheck': 1354451875.7543969, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00205302238464', 'lastCheck': 1354451876.6429369, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00702500343323', 'lastCheck': 1354451875.787678, 'code': 0, 'valid': True}} >Thread-29046::DEBUG::2012-12-02 14:38:00,114::task::1151::TaskManager.Task::(prepare) Task=`34b0b8c7-1066-42a0-b3dc-15b884b4a10a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00877785682678', 'lastCheck': 1354451879.4206901, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00182795524597', 'lastCheck': 1354451875.7543969, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00205302238464', 'lastCheck': 1354451876.6429369, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00702500343323', 'lastCheck': 1354451875.787678, 'code': 0, 'valid': True}} >Thread-29046::DEBUG::2012-12-02 14:38:00,114::task::568::TaskManager.Task::(_updateState) Task=`34b0b8c7-1066-42a0-b3dc-15b884b4a10a`::moving from state preparing -> state finished >Thread-29046::DEBUG::2012-12-02 14:38:00,114::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29046::DEBUG::2012-12-02 14:38:00,114::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29046::DEBUG::2012-12-02 14:38:00,115::task::957::TaskManager.Task::(_decref) Task=`34b0b8c7-1066-42a0-b3dc-15b884b4a10a`::ref 0 aborting False >Thread-29051::DEBUG::2012-12-02 14:38:07,304::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29051::DEBUG::2012-12-02 14:38:07,304::task::568::TaskManager.Task::(_updateState) Task=`6b329e89-5269-44bb-bb8b-1c1a358d1585`::moving from state init -> state preparing >Thread-29051::INFO::2012-12-02 14:38:07,304::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29051::INFO::2012-12-02 14:38:07,304::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29051::DEBUG::2012-12-02 14:38:07,305::task::1151::TaskManager.Task::(prepare) Task=`6b329e89-5269-44bb-bb8b-1c1a358d1585`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29051::DEBUG::2012-12-02 14:38:07,305::task::568::TaskManager.Task::(_updateState) Task=`6b329e89-5269-44bb-bb8b-1c1a358d1585`::moving from state preparing -> state finished >Thread-29051::DEBUG::2012-12-02 14:38:07,305::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29051::DEBUG::2012-12-02 14:38:07,305::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29051::DEBUG::2012-12-02 14:38:07,305::task::957::TaskManager.Task::(_decref) Task=`6b329e89-5269-44bb-bb8b-1c1a358d1585`::ref 0 aborting False >Thread-29052::DEBUG::2012-12-02 14:38:07,308::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29052::DEBUG::2012-12-02 14:38:07,308::task::568::TaskManager.Task::(_updateState) Task=`2f026eec-bb76-4240-96db-abf3bec40db4`::moving from state init -> state preparing >Thread-29052::INFO::2012-12-02 14:38:07,308::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29052::DEBUG::2012-12-02 14:38:07,308::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e603aeb4-997d-48d9-8182-a11d04c0ce62`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29052::DEBUG::2012-12-02 14:38:07,308::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29052::DEBUG::2012-12-02 14:38:07,308::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29052::DEBUG::2012-12-02 14:38:07,308::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e603aeb4-997d-48d9-8182-a11d04c0ce62`::Granted request >Thread-29052::DEBUG::2012-12-02 14:38:07,309::task::794::TaskManager.Task::(resourceAcquired) Task=`2f026eec-bb76-4240-96db-abf3bec40db4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29052::DEBUG::2012-12-02 14:38:07,309::task::957::TaskManager.Task::(_decref) Task=`2f026eec-bb76-4240-96db-abf3bec40db4`::ref 1 aborting False >Thread-29052::INFO::2012-12-02 14:38:07,313::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29052::DEBUG::2012-12-02 14:38:07,313::task::1151::TaskManager.Task::(prepare) Task=`2f026eec-bb76-4240-96db-abf3bec40db4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29052::DEBUG::2012-12-02 14:38:07,313::task::568::TaskManager.Task::(_updateState) Task=`2f026eec-bb76-4240-96db-abf3bec40db4`::moving from state preparing -> state finished >Thread-29052::DEBUG::2012-12-02 14:38:07,313::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29052::DEBUG::2012-12-02 14:38:07,313::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29052::DEBUG::2012-12-02 14:38:07,314::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29052::DEBUG::2012-12-02 14:38:07,314::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29052::DEBUG::2012-12-02 14:38:07,314::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29052::DEBUG::2012-12-02 14:38:07,314::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29052::DEBUG::2012-12-02 14:38:07,314::task::957::TaskManager.Task::(_decref) Task=`2f026eec-bb76-4240-96db-abf3bec40db4`::ref 0 aborting False >Thread-29054::DEBUG::2012-12-02 14:38:10,202::task::568::TaskManager.Task::(_updateState) Task=`0e0f6b7a-c605-4b08-9122-3bf713d92452`::moving from state init -> state preparing >Thread-29054::INFO::2012-12-02 14:38:10,202::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29054::INFO::2012-12-02 14:38:10,202::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0074188709259', 'lastCheck': 1354451889.436789, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206589698792', 'lastCheck': 1354451885.7663341, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00182890892029', 'lastCheck': 1354451886.6524241, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00785398483276', 'lastCheck': 1354451885.804491, 'code': 0, 'valid': True}} >Thread-29054::DEBUG::2012-12-02 14:38:10,202::task::1151::TaskManager.Task::(prepare) Task=`0e0f6b7a-c605-4b08-9122-3bf713d92452`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0074188709259', 'lastCheck': 1354451889.436789, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206589698792', 'lastCheck': 1354451885.7663341, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00182890892029', 'lastCheck': 1354451886.6524241, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00785398483276', 'lastCheck': 1354451885.804491, 'code': 0, 'valid': True}} >Thread-29054::DEBUG::2012-12-02 14:38:10,203::task::568::TaskManager.Task::(_updateState) Task=`0e0f6b7a-c605-4b08-9122-3bf713d92452`::moving from state preparing -> state finished >Thread-29054::DEBUG::2012-12-02 14:38:10,203::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29054::DEBUG::2012-12-02 14:38:10,203::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29054::DEBUG::2012-12-02 14:38:10,203::task::957::TaskManager.Task::(_decref) Task=`0e0f6b7a-c605-4b08-9122-3bf713d92452`::ref 0 aborting False >Thread-29059::DEBUG::2012-12-02 14:38:17,459::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29059::DEBUG::2012-12-02 14:38:17,460::task::568::TaskManager.Task::(_updateState) Task=`6c81dd9d-9278-4387-b4b1-bdf8f3301cc6`::moving from state init -> state preparing >Thread-29059::INFO::2012-12-02 14:38:17,460::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29059::INFO::2012-12-02 14:38:17,460::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29059::DEBUG::2012-12-02 14:38:17,460::task::1151::TaskManager.Task::(prepare) Task=`6c81dd9d-9278-4387-b4b1-bdf8f3301cc6`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29059::DEBUG::2012-12-02 14:38:17,460::task::568::TaskManager.Task::(_updateState) Task=`6c81dd9d-9278-4387-b4b1-bdf8f3301cc6`::moving from state preparing -> state finished >Thread-29059::DEBUG::2012-12-02 14:38:17,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29059::DEBUG::2012-12-02 14:38:17,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29059::DEBUG::2012-12-02 14:38:17,461::task::957::TaskManager.Task::(_decref) Task=`6c81dd9d-9278-4387-b4b1-bdf8f3301cc6`::ref 0 aborting False >Thread-29060::DEBUG::2012-12-02 14:38:17,463::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29060::DEBUG::2012-12-02 14:38:17,463::task::568::TaskManager.Task::(_updateState) Task=`df79b503-741b-4668-81f2-90ecb6cd8255`::moving from state init -> state preparing >Thread-29060::INFO::2012-12-02 14:38:17,463::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29060::DEBUG::2012-12-02 14:38:17,464::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8b70b299-1bc7-44c1-ac9b-1944648a68d7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29060::DEBUG::2012-12-02 14:38:17,464::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29060::DEBUG::2012-12-02 14:38:17,464::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29060::DEBUG::2012-12-02 14:38:17,464::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8b70b299-1bc7-44c1-ac9b-1944648a68d7`::Granted request >Thread-29060::DEBUG::2012-12-02 14:38:17,464::task::794::TaskManager.Task::(resourceAcquired) Task=`df79b503-741b-4668-81f2-90ecb6cd8255`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29060::DEBUG::2012-12-02 14:38:17,465::task::957::TaskManager.Task::(_decref) Task=`df79b503-741b-4668-81f2-90ecb6cd8255`::ref 1 aborting False >Thread-29060::INFO::2012-12-02 14:38:17,469::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29060::DEBUG::2012-12-02 14:38:17,469::task::1151::TaskManager.Task::(prepare) Task=`df79b503-741b-4668-81f2-90ecb6cd8255`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29060::DEBUG::2012-12-02 14:38:17,469::task::568::TaskManager.Task::(_updateState) Task=`df79b503-741b-4668-81f2-90ecb6cd8255`::moving from state preparing -> state finished >Thread-29060::DEBUG::2012-12-02 14:38:17,469::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29060::DEBUG::2012-12-02 14:38:17,469::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29060::DEBUG::2012-12-02 14:38:17,469::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29060::DEBUG::2012-12-02 14:38:17,470::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29060::DEBUG::2012-12-02 14:38:17,470::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29060::DEBUG::2012-12-02 14:38:17,470::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29060::DEBUG::2012-12-02 14:38:17,470::task::957::TaskManager.Task::(_decref) Task=`df79b503-741b-4668-81f2-90ecb6cd8255`::ref 0 aborting False >Thread-29062::DEBUG::2012-12-02 14:38:20,298::task::568::TaskManager.Task::(_updateState) Task=`80533f7d-4daa-4c12-aff9-4e1d9f9167f1`::moving from state init -> state preparing >Thread-29062::INFO::2012-12-02 14:38:20,299::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29062::INFO::2012-12-02 14:38:20,299::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00639700889587', 'lastCheck': 1354451899.452769, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00346493721008', 'lastCheck': 1354451895.7814729, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00269794464111', 'lastCheck': 1354451896.6639271, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00555896759033', 'lastCheck': 1354451895.8190889, 'code': 0, 'valid': True}} >Thread-29062::DEBUG::2012-12-02 14:38:20,299::task::1151::TaskManager.Task::(prepare) Task=`80533f7d-4daa-4c12-aff9-4e1d9f9167f1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00639700889587', 'lastCheck': 1354451899.452769, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00346493721008', 'lastCheck': 1354451895.7814729, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00269794464111', 'lastCheck': 1354451896.6639271, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00555896759033', 'lastCheck': 1354451895.8190889, 'code': 0, 'valid': True}} >Thread-29062::DEBUG::2012-12-02 14:38:20,299::task::568::TaskManager.Task::(_updateState) Task=`80533f7d-4daa-4c12-aff9-4e1d9f9167f1`::moving from state preparing -> state finished >Thread-29062::DEBUG::2012-12-02 14:38:20,299::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29062::DEBUG::2012-12-02 14:38:20,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29062::DEBUG::2012-12-02 14:38:20,300::task::957::TaskManager.Task::(_decref) Task=`80533f7d-4daa-4c12-aff9-4e1d9f9167f1`::ref 0 aborting False >Thread-29067::DEBUG::2012-12-02 14:38:27,596::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29067::DEBUG::2012-12-02 14:38:27,597::task::568::TaskManager.Task::(_updateState) Task=`734f1475-958f-4a14-91b6-de37380935a3`::moving from state init -> state preparing >Thread-29067::INFO::2012-12-02 14:38:27,597::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29067::INFO::2012-12-02 14:38:27,597::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29067::DEBUG::2012-12-02 14:38:27,597::task::1151::TaskManager.Task::(prepare) Task=`734f1475-958f-4a14-91b6-de37380935a3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29067::DEBUG::2012-12-02 14:38:27,597::task::568::TaskManager.Task::(_updateState) Task=`734f1475-958f-4a14-91b6-de37380935a3`::moving from state preparing -> state finished >Thread-29067::DEBUG::2012-12-02 14:38:27,598::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29067::DEBUG::2012-12-02 14:38:27,598::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29067::DEBUG::2012-12-02 14:38:27,598::task::957::TaskManager.Task::(_decref) Task=`734f1475-958f-4a14-91b6-de37380935a3`::ref 0 aborting False >Thread-29068::DEBUG::2012-12-02 14:38:27,600::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29068::DEBUG::2012-12-02 14:38:27,601::task::568::TaskManager.Task::(_updateState) Task=`ad4ca7b9-7473-4a62-be67-ec3b498f3f88`::moving from state init -> state preparing >Thread-29068::INFO::2012-12-02 14:38:27,601::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29068::DEBUG::2012-12-02 14:38:27,601::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ff6d4e5d-e0b4-4964-aafc-84e09c9211f6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29068::DEBUG::2012-12-02 14:38:27,601::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29068::DEBUG::2012-12-02 14:38:27,602::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29068::DEBUG::2012-12-02 14:38:27,602::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ff6d4e5d-e0b4-4964-aafc-84e09c9211f6`::Granted request >Thread-29068::DEBUG::2012-12-02 14:38:27,602::task::794::TaskManager.Task::(resourceAcquired) Task=`ad4ca7b9-7473-4a62-be67-ec3b498f3f88`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29068::DEBUG::2012-12-02 14:38:27,602::task::957::TaskManager.Task::(_decref) Task=`ad4ca7b9-7473-4a62-be67-ec3b498f3f88`::ref 1 aborting False >Thread-29068::INFO::2012-12-02 14:38:27,607::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29068::DEBUG::2012-12-02 14:38:27,607::task::1151::TaskManager.Task::(prepare) Task=`ad4ca7b9-7473-4a62-be67-ec3b498f3f88`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29068::DEBUG::2012-12-02 14:38:27,608::task::568::TaskManager.Task::(_updateState) Task=`ad4ca7b9-7473-4a62-be67-ec3b498f3f88`::moving from state preparing -> state finished >Thread-29068::DEBUG::2012-12-02 14:38:27,608::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29068::DEBUG::2012-12-02 14:38:27,608::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29068::DEBUG::2012-12-02 14:38:27,608::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29068::DEBUG::2012-12-02 14:38:27,608::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29068::DEBUG::2012-12-02 14:38:27,608::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29068::DEBUG::2012-12-02 14:38:27,608::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29068::DEBUG::2012-12-02 14:38:27,608::task::957::TaskManager.Task::(_decref) Task=`ad4ca7b9-7473-4a62-be67-ec3b498f3f88`::ref 0 aborting False >Thread-29070::DEBUG::2012-12-02 14:38:30,387::task::568::TaskManager.Task::(_updateState) Task=`2154e07d-7867-44f2-88d7-d468304ead81`::moving from state init -> state preparing >Thread-29070::INFO::2012-12-02 14:38:30,388::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29070::INFO::2012-12-02 14:38:30,388::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00785088539124', 'lastCheck': 1354451909.4686749, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00193309783936', 'lastCheck': 1354451905.796279, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00131011009216', 'lastCheck': 1354451906.671869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00593280792236', 'lastCheck': 1354451905.8331549, 'code': 0, 'valid': True}} >Thread-29070::DEBUG::2012-12-02 14:38:30,388::task::1151::TaskManager.Task::(prepare) Task=`2154e07d-7867-44f2-88d7-d468304ead81`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00785088539124', 'lastCheck': 1354451909.4686749, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00193309783936', 'lastCheck': 1354451905.796279, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00131011009216', 'lastCheck': 1354451906.671869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00593280792236', 'lastCheck': 1354451905.8331549, 'code': 0, 'valid': True}} >Thread-29070::DEBUG::2012-12-02 14:38:30,388::task::568::TaskManager.Task::(_updateState) Task=`2154e07d-7867-44f2-88d7-d468304ead81`::moving from state preparing -> state finished >Thread-29070::DEBUG::2012-12-02 14:38:30,388::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29070::DEBUG::2012-12-02 14:38:30,388::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29070::DEBUG::2012-12-02 14:38:30,389::task::957::TaskManager.Task::(_decref) Task=`2154e07d-7867-44f2-88d7-d468304ead81`::ref 0 aborting False >Thread-29075::DEBUG::2012-12-02 14:38:37,726::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29075::DEBUG::2012-12-02 14:38:37,727::task::568::TaskManager.Task::(_updateState) Task=`751d5fa3-f827-41f4-9933-fac533fc55bd`::moving from state init -> state preparing >Thread-29075::INFO::2012-12-02 14:38:37,727::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29075::INFO::2012-12-02 14:38:37,727::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29075::DEBUG::2012-12-02 14:38:37,727::task::1151::TaskManager.Task::(prepare) Task=`751d5fa3-f827-41f4-9933-fac533fc55bd`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29075::DEBUG::2012-12-02 14:38:37,728::task::568::TaskManager.Task::(_updateState) Task=`751d5fa3-f827-41f4-9933-fac533fc55bd`::moving from state preparing -> state finished >Thread-29075::DEBUG::2012-12-02 14:38:37,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29075::DEBUG::2012-12-02 14:38:37,728::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29075::DEBUG::2012-12-02 14:38:37,728::task::957::TaskManager.Task::(_decref) Task=`751d5fa3-f827-41f4-9933-fac533fc55bd`::ref 0 aborting False >Thread-29076::DEBUG::2012-12-02 14:38:37,731::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29076::DEBUG::2012-12-02 14:38:37,731::task::568::TaskManager.Task::(_updateState) Task=`cefab80a-5ccc-403c-8168-9fae47ac14cd`::moving from state init -> state preparing >Thread-29076::INFO::2012-12-02 14:38:37,731::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29076::DEBUG::2012-12-02 14:38:37,731::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`162bb2f9-056b-41be-aa72-7c67bca8b415`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29076::DEBUG::2012-12-02 14:38:37,731::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29076::DEBUG::2012-12-02 14:38:37,732::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29076::DEBUG::2012-12-02 14:38:37,732::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`162bb2f9-056b-41be-aa72-7c67bca8b415`::Granted request >Thread-29076::DEBUG::2012-12-02 14:38:37,732::task::794::TaskManager.Task::(resourceAcquired) Task=`cefab80a-5ccc-403c-8168-9fae47ac14cd`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29076::DEBUG::2012-12-02 14:38:37,732::task::957::TaskManager.Task::(_decref) Task=`cefab80a-5ccc-403c-8168-9fae47ac14cd`::ref 1 aborting False >Thread-29076::INFO::2012-12-02 14:38:37,737::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29076::DEBUG::2012-12-02 14:38:37,738::task::1151::TaskManager.Task::(prepare) Task=`cefab80a-5ccc-403c-8168-9fae47ac14cd`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29076::DEBUG::2012-12-02 14:38:37,738::task::568::TaskManager.Task::(_updateState) Task=`cefab80a-5ccc-403c-8168-9fae47ac14cd`::moving from state preparing -> state finished >Thread-29076::DEBUG::2012-12-02 14:38:37,738::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29076::DEBUG::2012-12-02 14:38:37,738::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29076::DEBUG::2012-12-02 14:38:37,738::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29076::DEBUG::2012-12-02 14:38:37,738::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29076::DEBUG::2012-12-02 14:38:37,738::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29076::DEBUG::2012-12-02 14:38:37,738::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29076::DEBUG::2012-12-02 14:38:37,738::task::957::TaskManager.Task::(_decref) Task=`cefab80a-5ccc-403c-8168-9fae47ac14cd`::ref 0 aborting False >Thread-29078::DEBUG::2012-12-02 14:38:40,537::task::568::TaskManager.Task::(_updateState) Task=`29102684-d6ad-4e91-9d4e-93e011f40e3d`::moving from state init -> state preparing >Thread-29078::INFO::2012-12-02 14:38:40,537::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29078::INFO::2012-12-02 14:38:40,537::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00841999053955', 'lastCheck': 1354451919.4893739, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00307893753052', 'lastCheck': 1354451915.809129, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00312209129333', 'lastCheck': 1354451916.684567, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00642085075378', 'lastCheck': 1354451915.848448, 'code': 0, 'valid': True}} >Thread-29078::DEBUG::2012-12-02 14:38:40,537::task::1151::TaskManager.Task::(prepare) Task=`29102684-d6ad-4e91-9d4e-93e011f40e3d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00841999053955', 'lastCheck': 1354451919.4893739, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00307893753052', 'lastCheck': 1354451915.809129, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00312209129333', 'lastCheck': 1354451916.684567, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00642085075378', 'lastCheck': 1354451915.848448, 'code': 0, 'valid': True}} >Thread-29078::DEBUG::2012-12-02 14:38:40,538::task::568::TaskManager.Task::(_updateState) Task=`29102684-d6ad-4e91-9d4e-93e011f40e3d`::moving from state preparing -> state finished >Thread-29078::DEBUG::2012-12-02 14:38:40,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29078::DEBUG::2012-12-02 14:38:40,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29078::DEBUG::2012-12-02 14:38:40,538::task::957::TaskManager.Task::(_decref) Task=`29102684-d6ad-4e91-9d4e-93e011f40e3d`::ref 0 aborting False >Thread-29083::DEBUG::2012-12-02 14:38:47,864::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29083::DEBUG::2012-12-02 14:38:47,864::task::568::TaskManager.Task::(_updateState) Task=`8b7a8e5a-ee0c-4d33-97e9-8a7b1240fd17`::moving from state init -> state preparing >Thread-29083::INFO::2012-12-02 14:38:47,864::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29083::INFO::2012-12-02 14:38:47,865::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29083::DEBUG::2012-12-02 14:38:47,865::task::1151::TaskManager.Task::(prepare) Task=`8b7a8e5a-ee0c-4d33-97e9-8a7b1240fd17`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29083::DEBUG::2012-12-02 14:38:47,865::task::568::TaskManager.Task::(_updateState) Task=`8b7a8e5a-ee0c-4d33-97e9-8a7b1240fd17`::moving from state preparing -> state finished >Thread-29083::DEBUG::2012-12-02 14:38:47,865::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29083::DEBUG::2012-12-02 14:38:47,865::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29083::DEBUG::2012-12-02 14:38:47,865::task::957::TaskManager.Task::(_decref) Task=`8b7a8e5a-ee0c-4d33-97e9-8a7b1240fd17`::ref 0 aborting False >Thread-29084::DEBUG::2012-12-02 14:38:47,868::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29084::DEBUG::2012-12-02 14:38:47,868::task::568::TaskManager.Task::(_updateState) Task=`f30dd9bf-f383-4008-aa98-1ded96682da3`::moving from state init -> state preparing >Thread-29084::INFO::2012-12-02 14:38:47,868::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29084::DEBUG::2012-12-02 14:38:47,868::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`31ced77c-f418-41aa-b684-eedff2a134cb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29084::DEBUG::2012-12-02 14:38:47,868::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29084::DEBUG::2012-12-02 14:38:47,869::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29084::DEBUG::2012-12-02 14:38:47,869::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`31ced77c-f418-41aa-b684-eedff2a134cb`::Granted request >Thread-29084::DEBUG::2012-12-02 14:38:47,869::task::794::TaskManager.Task::(resourceAcquired) Task=`f30dd9bf-f383-4008-aa98-1ded96682da3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29084::DEBUG::2012-12-02 14:38:47,869::task::957::TaskManager.Task::(_decref) Task=`f30dd9bf-f383-4008-aa98-1ded96682da3`::ref 1 aborting False >Thread-29084::INFO::2012-12-02 14:38:47,875::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29084::DEBUG::2012-12-02 14:38:47,876::task::1151::TaskManager.Task::(prepare) Task=`f30dd9bf-f383-4008-aa98-1ded96682da3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29084::DEBUG::2012-12-02 14:38:47,876::task::568::TaskManager.Task::(_updateState) Task=`f30dd9bf-f383-4008-aa98-1ded96682da3`::moving from state preparing -> state finished >Thread-29084::DEBUG::2012-12-02 14:38:47,876::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29084::DEBUG::2012-12-02 14:38:47,876::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29084::DEBUG::2012-12-02 14:38:47,876::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29084::DEBUG::2012-12-02 14:38:47,876::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29084::DEBUG::2012-12-02 14:38:47,876::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29084::DEBUG::2012-12-02 14:38:47,876::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29084::DEBUG::2012-12-02 14:38:47,876::task::957::TaskManager.Task::(_decref) Task=`f30dd9bf-f383-4008-aa98-1ded96682da3`::ref 0 aborting False >Thread-29086::DEBUG::2012-12-02 14:38:50,637::task::568::TaskManager.Task::(_updateState) Task=`aceef005-c1d8-4703-a492-6418aa824a17`::moving from state init -> state preparing >Thread-29086::INFO::2012-12-02 14:38:50,637::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29086::INFO::2012-12-02 14:38:50,638::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00750517845154', 'lastCheck': 1354451929.5064609, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0020580291748', 'lastCheck': 1354451925.8239169, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00189185142517', 'lastCheck': 1354451926.695641, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00796008110046', 'lastCheck': 1354451925.865083, 'code': 0, 'valid': True}} >Thread-29086::DEBUG::2012-12-02 14:38:50,638::task::1151::TaskManager.Task::(prepare) Task=`aceef005-c1d8-4703-a492-6418aa824a17`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00750517845154', 'lastCheck': 1354451929.5064609, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0020580291748', 'lastCheck': 1354451925.8239169, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00189185142517', 'lastCheck': 1354451926.695641, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00796008110046', 'lastCheck': 1354451925.865083, 'code': 0, 'valid': True}} >Thread-29086::DEBUG::2012-12-02 14:38:50,638::task::568::TaskManager.Task::(_updateState) Task=`aceef005-c1d8-4703-a492-6418aa824a17`::moving from state preparing -> state finished >Thread-29086::DEBUG::2012-12-02 14:38:50,638::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29086::DEBUG::2012-12-02 14:38:50,638::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29086::DEBUG::2012-12-02 14:38:50,638::task::957::TaskManager.Task::(_decref) Task=`aceef005-c1d8-4703-a492-6418aa824a17`::ref 0 aborting False >Thread-29091::DEBUG::2012-12-02 14:38:58,018::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29091::DEBUG::2012-12-02 14:38:58,018::task::568::TaskManager.Task::(_updateState) Task=`4182af5c-e411-40aa-af6e-a1027b08f73d`::moving from state init -> state preparing >Thread-29091::INFO::2012-12-02 14:38:58,018::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29091::INFO::2012-12-02 14:38:58,019::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29091::DEBUG::2012-12-02 14:38:58,019::task::1151::TaskManager.Task::(prepare) Task=`4182af5c-e411-40aa-af6e-a1027b08f73d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29091::DEBUG::2012-12-02 14:38:58,019::task::568::TaskManager.Task::(_updateState) Task=`4182af5c-e411-40aa-af6e-a1027b08f73d`::moving from state preparing -> state finished >Thread-29091::DEBUG::2012-12-02 14:38:58,019::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29091::DEBUG::2012-12-02 14:38:58,019::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29091::DEBUG::2012-12-02 14:38:58,019::task::957::TaskManager.Task::(_decref) Task=`4182af5c-e411-40aa-af6e-a1027b08f73d`::ref 0 aborting False >Thread-29092::DEBUG::2012-12-02 14:38:58,022::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29092::DEBUG::2012-12-02 14:38:58,022::task::568::TaskManager.Task::(_updateState) Task=`fa85366e-3709-4d7a-9cca-e2799c06f54f`::moving from state init -> state preparing >Thread-29092::INFO::2012-12-02 14:38:58,022::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29092::DEBUG::2012-12-02 14:38:58,023::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`49ffd11d-b095-45df-8535-930f31996368`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29092::DEBUG::2012-12-02 14:38:58,023::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29092::DEBUG::2012-12-02 14:38:58,023::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29092::DEBUG::2012-12-02 14:38:58,023::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`49ffd11d-b095-45df-8535-930f31996368`::Granted request >Thread-29092::DEBUG::2012-12-02 14:38:58,023::task::794::TaskManager.Task::(resourceAcquired) Task=`fa85366e-3709-4d7a-9cca-e2799c06f54f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29092::DEBUG::2012-12-02 14:38:58,024::task::957::TaskManager.Task::(_decref) Task=`fa85366e-3709-4d7a-9cca-e2799c06f54f`::ref 1 aborting False >Thread-29092::INFO::2012-12-02 14:38:58,028::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29092::DEBUG::2012-12-02 14:38:58,028::task::1151::TaskManager.Task::(prepare) Task=`fa85366e-3709-4d7a-9cca-e2799c06f54f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29092::DEBUG::2012-12-02 14:38:58,029::task::568::TaskManager.Task::(_updateState) Task=`fa85366e-3709-4d7a-9cca-e2799c06f54f`::moving from state preparing -> state finished >Thread-29092::DEBUG::2012-12-02 14:38:58,029::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29092::DEBUG::2012-12-02 14:38:58,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29092::DEBUG::2012-12-02 14:38:58,029::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29092::DEBUG::2012-12-02 14:38:58,029::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29092::DEBUG::2012-12-02 14:38:58,029::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29092::DEBUG::2012-12-02 14:38:58,029::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29092::DEBUG::2012-12-02 14:38:58,029::task::957::TaskManager.Task::(_decref) Task=`fa85366e-3709-4d7a-9cca-e2799c06f54f`::ref 0 aborting False >Thread-29094::DEBUG::2012-12-02 14:39:00,725::task::568::TaskManager.Task::(_updateState) Task=`d7413e21-beb2-449c-acca-d25c6bf3cd13`::moving from state init -> state preparing >Thread-29094::INFO::2012-12-02 14:39:00,726::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29094::INFO::2012-12-02 14:39:00,726::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0100140571594', 'lastCheck': 1354451939.526113, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00187301635742', 'lastCheck': 1354451935.8340609, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0020170211792', 'lastCheck': 1354451936.7065811, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00710415840149', 'lastCheck': 1354451935.881489, 'code': 0, 'valid': True}} >Thread-29094::DEBUG::2012-12-02 14:39:00,726::task::1151::TaskManager.Task::(prepare) Task=`d7413e21-beb2-449c-acca-d25c6bf3cd13`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0100140571594', 'lastCheck': 1354451939.526113, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00187301635742', 'lastCheck': 1354451935.8340609, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0020170211792', 'lastCheck': 1354451936.7065811, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00710415840149', 'lastCheck': 1354451935.881489, 'code': 0, 'valid': True}} >Thread-29094::DEBUG::2012-12-02 14:39:00,726::task::568::TaskManager.Task::(_updateState) Task=`d7413e21-beb2-449c-acca-d25c6bf3cd13`::moving from state preparing -> state finished >Thread-29094::DEBUG::2012-12-02 14:39:00,726::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29094::DEBUG::2012-12-02 14:39:00,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29094::DEBUG::2012-12-02 14:39:00,726::task::957::TaskManager.Task::(_decref) Task=`d7413e21-beb2-449c-acca-d25c6bf3cd13`::ref 0 aborting False >Thread-29099::DEBUG::2012-12-02 14:39:08,157::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29099::DEBUG::2012-12-02 14:39:08,157::task::568::TaskManager.Task::(_updateState) Task=`20fd1a53-316e-4a1c-9b0a-a7dc79b0c04e`::moving from state init -> state preparing >Thread-29099::INFO::2012-12-02 14:39:08,157::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29099::INFO::2012-12-02 14:39:08,158::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29099::DEBUG::2012-12-02 14:39:08,158::task::1151::TaskManager.Task::(prepare) Task=`20fd1a53-316e-4a1c-9b0a-a7dc79b0c04e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29099::DEBUG::2012-12-02 14:39:08,158::task::568::TaskManager.Task::(_updateState) Task=`20fd1a53-316e-4a1c-9b0a-a7dc79b0c04e`::moving from state preparing -> state finished >Thread-29099::DEBUG::2012-12-02 14:39:08,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29099::DEBUG::2012-12-02 14:39:08,158::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29099::DEBUG::2012-12-02 14:39:08,158::task::957::TaskManager.Task::(_decref) Task=`20fd1a53-316e-4a1c-9b0a-a7dc79b0c04e`::ref 0 aborting False >Thread-29100::DEBUG::2012-12-02 14:39:08,161::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29100::DEBUG::2012-12-02 14:39:08,161::task::568::TaskManager.Task::(_updateState) Task=`4ce3eea9-5e54-42b1-8ed3-4a0e6ddfa5ab`::moving from state init -> state preparing >Thread-29100::INFO::2012-12-02 14:39:08,161::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29100::DEBUG::2012-12-02 14:39:08,162::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1185b981-050d-46fc-a74a-531e1d5968e1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29100::DEBUG::2012-12-02 14:39:08,162::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29100::DEBUG::2012-12-02 14:39:08,162::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29100::DEBUG::2012-12-02 14:39:08,162::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1185b981-050d-46fc-a74a-531e1d5968e1`::Granted request >Thread-29100::DEBUG::2012-12-02 14:39:08,162::task::794::TaskManager.Task::(resourceAcquired) Task=`4ce3eea9-5e54-42b1-8ed3-4a0e6ddfa5ab`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29100::DEBUG::2012-12-02 14:39:08,162::task::957::TaskManager.Task::(_decref) Task=`4ce3eea9-5e54-42b1-8ed3-4a0e6ddfa5ab`::ref 1 aborting False >Thread-29100::INFO::2012-12-02 14:39:08,168::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29100::DEBUG::2012-12-02 14:39:08,168::task::1151::TaskManager.Task::(prepare) Task=`4ce3eea9-5e54-42b1-8ed3-4a0e6ddfa5ab`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29100::DEBUG::2012-12-02 14:39:08,168::task::568::TaskManager.Task::(_updateState) Task=`4ce3eea9-5e54-42b1-8ed3-4a0e6ddfa5ab`::moving from state preparing -> state finished >Thread-29100::DEBUG::2012-12-02 14:39:08,168::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29100::DEBUG::2012-12-02 14:39:08,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29100::DEBUG::2012-12-02 14:39:08,168::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29100::DEBUG::2012-12-02 14:39:08,168::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29100::DEBUG::2012-12-02 14:39:08,168::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29100::DEBUG::2012-12-02 14:39:08,168::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29100::DEBUG::2012-12-02 14:39:08,169::task::957::TaskManager.Task::(_decref) Task=`4ce3eea9-5e54-42b1-8ed3-4a0e6ddfa5ab`::ref 0 aborting False >Thread-29102::DEBUG::2012-12-02 14:39:10,816::task::568::TaskManager.Task::(_updateState) Task=`8dcb1aa6-921a-47e9-84ff-f4f183e9d8bd`::moving from state init -> state preparing >Thread-29102::INFO::2012-12-02 14:39:10,816::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29102::INFO::2012-12-02 14:39:10,816::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00821185112', 'lastCheck': 1354451949.543745, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00201201438904', 'lastCheck': 1354451945.844312, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172805786133', 'lastCheck': 1354451946.716764, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00722885131836', 'lastCheck': 1354451945.897408, 'code': 0, 'valid': True}} >Thread-29102::DEBUG::2012-12-02 14:39:10,816::task::1151::TaskManager.Task::(prepare) Task=`8dcb1aa6-921a-47e9-84ff-f4f183e9d8bd`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00821185112', 'lastCheck': 1354451949.543745, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00201201438904', 'lastCheck': 1354451945.844312, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172805786133', 'lastCheck': 1354451946.716764, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00722885131836', 'lastCheck': 1354451945.897408, 'code': 0, 'valid': True}} >Thread-29102::DEBUG::2012-12-02 14:39:10,817::task::568::TaskManager.Task::(_updateState) Task=`8dcb1aa6-921a-47e9-84ff-f4f183e9d8bd`::moving from state preparing -> state finished >Thread-29102::DEBUG::2012-12-02 14:39:10,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29102::DEBUG::2012-12-02 14:39:10,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29102::DEBUG::2012-12-02 14:39:10,817::task::957::TaskManager.Task::(_decref) Task=`8dcb1aa6-921a-47e9-84ff-f4f183e9d8bd`::ref 0 aborting False >Thread-29107::DEBUG::2012-12-02 14:39:18,294::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29107::DEBUG::2012-12-02 14:39:18,294::task::568::TaskManager.Task::(_updateState) Task=`f4e7603b-a360-4fe2-8095-794ac0aa888f`::moving from state init -> state preparing >Thread-29107::INFO::2012-12-02 14:39:18,294::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29107::INFO::2012-12-02 14:39:18,295::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29107::DEBUG::2012-12-02 14:39:18,295::task::1151::TaskManager.Task::(prepare) Task=`f4e7603b-a360-4fe2-8095-794ac0aa888f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29107::DEBUG::2012-12-02 14:39:18,295::task::568::TaskManager.Task::(_updateState) Task=`f4e7603b-a360-4fe2-8095-794ac0aa888f`::moving from state preparing -> state finished >Thread-29107::DEBUG::2012-12-02 14:39:18,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29107::DEBUG::2012-12-02 14:39:18,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29107::DEBUG::2012-12-02 14:39:18,295::task::957::TaskManager.Task::(_decref) Task=`f4e7603b-a360-4fe2-8095-794ac0aa888f`::ref 0 aborting False >Thread-29108::DEBUG::2012-12-02 14:39:18,298::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29108::DEBUG::2012-12-02 14:39:18,298::task::568::TaskManager.Task::(_updateState) Task=`60fd63ec-5bd3-4132-a177-8bb33828a791`::moving from state init -> state preparing >Thread-29108::INFO::2012-12-02 14:39:18,298::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29108::DEBUG::2012-12-02 14:39:18,298::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ff9f4fc6-1ed7-41fb-8989-9de1d2cb7246`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29108::DEBUG::2012-12-02 14:39:18,298::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29108::DEBUG::2012-12-02 14:39:18,299::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29108::DEBUG::2012-12-02 14:39:18,299::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ff9f4fc6-1ed7-41fb-8989-9de1d2cb7246`::Granted request >Thread-29108::DEBUG::2012-12-02 14:39:18,300::task::794::TaskManager.Task::(resourceAcquired) Task=`60fd63ec-5bd3-4132-a177-8bb33828a791`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29108::DEBUG::2012-12-02 14:39:18,300::task::957::TaskManager.Task::(_decref) Task=`60fd63ec-5bd3-4132-a177-8bb33828a791`::ref 1 aborting False >Thread-29108::INFO::2012-12-02 14:39:18,306::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29108::DEBUG::2012-12-02 14:39:18,306::task::1151::TaskManager.Task::(prepare) Task=`60fd63ec-5bd3-4132-a177-8bb33828a791`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29108::DEBUG::2012-12-02 14:39:18,306::task::568::TaskManager.Task::(_updateState) Task=`60fd63ec-5bd3-4132-a177-8bb33828a791`::moving from state preparing -> state finished >Thread-29108::DEBUG::2012-12-02 14:39:18,306::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29108::DEBUG::2012-12-02 14:39:18,306::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29108::DEBUG::2012-12-02 14:39:18,306::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29108::DEBUG::2012-12-02 14:39:18,306::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29108::DEBUG::2012-12-02 14:39:18,307::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29108::DEBUG::2012-12-02 14:39:18,307::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29108::DEBUG::2012-12-02 14:39:18,307::task::957::TaskManager.Task::(_decref) Task=`60fd63ec-5bd3-4132-a177-8bb33828a791`::ref 0 aborting False >Thread-29110::DEBUG::2012-12-02 14:39:20,886::task::568::TaskManager.Task::(_updateState) Task=`5d857935-dbe8-4ef4-92cd-fca54c55a30e`::moving from state init -> state preparing >Thread-29110::INFO::2012-12-02 14:39:20,886::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29110::INFO::2012-12-02 14:39:20,887::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00760889053345', 'lastCheck': 1354451959.5594039, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00327396392822', 'lastCheck': 1354451955.8569241, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00260996818542', 'lastCheck': 1354451956.729398, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00654792785645', 'lastCheck': 1354451955.912571, 'code': 0, 'valid': True}} >Thread-29110::DEBUG::2012-12-02 14:39:20,887::task::1151::TaskManager.Task::(prepare) Task=`5d857935-dbe8-4ef4-92cd-fca54c55a30e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00760889053345', 'lastCheck': 1354451959.5594039, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00327396392822', 'lastCheck': 1354451955.8569241, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00260996818542', 'lastCheck': 1354451956.729398, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00654792785645', 'lastCheck': 1354451955.912571, 'code': 0, 'valid': True}} >Thread-29110::DEBUG::2012-12-02 14:39:20,887::task::568::TaskManager.Task::(_updateState) Task=`5d857935-dbe8-4ef4-92cd-fca54c55a30e`::moving from state preparing -> state finished >Thread-29110::DEBUG::2012-12-02 14:39:20,887::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29110::DEBUG::2012-12-02 14:39:20,887::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29110::DEBUG::2012-12-02 14:39:20,887::task::957::TaskManager.Task::(_decref) Task=`5d857935-dbe8-4ef4-92cd-fca54c55a30e`::ref 0 aborting False >Thread-29115::DEBUG::2012-12-02 14:39:28,448::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29115::DEBUG::2012-12-02 14:39:28,448::task::568::TaskManager.Task::(_updateState) Task=`ce876b34-3573-49ae-970b-83b70f25780a`::moving from state init -> state preparing >Thread-29115::INFO::2012-12-02 14:39:28,448::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29115::INFO::2012-12-02 14:39:28,449::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29115::DEBUG::2012-12-02 14:39:28,449::task::1151::TaskManager.Task::(prepare) Task=`ce876b34-3573-49ae-970b-83b70f25780a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29115::DEBUG::2012-12-02 14:39:28,449::task::568::TaskManager.Task::(_updateState) Task=`ce876b34-3573-49ae-970b-83b70f25780a`::moving from state preparing -> state finished >Thread-29115::DEBUG::2012-12-02 14:39:28,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29115::DEBUG::2012-12-02 14:39:28,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29115::DEBUG::2012-12-02 14:39:28,449::task::957::TaskManager.Task::(_decref) Task=`ce876b34-3573-49ae-970b-83b70f25780a`::ref 0 aborting False >Thread-29116::DEBUG::2012-12-02 14:39:28,452::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29116::DEBUG::2012-12-02 14:39:28,452::task::568::TaskManager.Task::(_updateState) Task=`2fdef6fc-b5c2-4ab6-85fa-d1a0880414ba`::moving from state init -> state preparing >Thread-29116::INFO::2012-12-02 14:39:28,452::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29116::DEBUG::2012-12-02 14:39:28,452::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`517ed673-8221-4d2f-bf52-6016b8cc9a73`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29116::DEBUG::2012-12-02 14:39:28,452::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29116::DEBUG::2012-12-02 14:39:28,453::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29116::DEBUG::2012-12-02 14:39:28,453::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`517ed673-8221-4d2f-bf52-6016b8cc9a73`::Granted request >Thread-29116::DEBUG::2012-12-02 14:39:28,453::task::794::TaskManager.Task::(resourceAcquired) Task=`2fdef6fc-b5c2-4ab6-85fa-d1a0880414ba`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29116::DEBUG::2012-12-02 14:39:28,453::task::957::TaskManager.Task::(_decref) Task=`2fdef6fc-b5c2-4ab6-85fa-d1a0880414ba`::ref 1 aborting False >Thread-29116::INFO::2012-12-02 14:39:28,460::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29116::DEBUG::2012-12-02 14:39:28,460::task::1151::TaskManager.Task::(prepare) Task=`2fdef6fc-b5c2-4ab6-85fa-d1a0880414ba`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29116::DEBUG::2012-12-02 14:39:28,460::task::568::TaskManager.Task::(_updateState) Task=`2fdef6fc-b5c2-4ab6-85fa-d1a0880414ba`::moving from state preparing -> state finished >Thread-29116::DEBUG::2012-12-02 14:39:28,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29116::DEBUG::2012-12-02 14:39:28,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29116::DEBUG::2012-12-02 14:39:28,460::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29116::DEBUG::2012-12-02 14:39:28,461::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29116::DEBUG::2012-12-02 14:39:28,461::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29116::DEBUG::2012-12-02 14:39:28,461::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29116::DEBUG::2012-12-02 14:39:28,461::task::957::TaskManager.Task::(_decref) Task=`2fdef6fc-b5c2-4ab6-85fa-d1a0880414ba`::ref 0 aborting False >Thread-29118::DEBUG::2012-12-02 14:39:30,988::task::568::TaskManager.Task::(_updateState) Task=`de043af5-c46d-45c3-9819-4bd691511cc8`::moving from state init -> state preparing >Thread-29118::INFO::2012-12-02 14:39:30,988::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29118::INFO::2012-12-02 14:39:30,988::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830793380737', 'lastCheck': 1354451969.5772519, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00228905677795', 'lastCheck': 1354451965.8710971, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163412094116', 'lastCheck': 1354451966.7397871, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00479292869568', 'lastCheck': 1354451965.926693, 'code': 0, 'valid': True}} >Thread-29118::DEBUG::2012-12-02 14:39:30,988::task::1151::TaskManager.Task::(prepare) Task=`de043af5-c46d-45c3-9819-4bd691511cc8`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830793380737', 'lastCheck': 1354451969.5772519, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00228905677795', 'lastCheck': 1354451965.8710971, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163412094116', 'lastCheck': 1354451966.7397871, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00479292869568', 'lastCheck': 1354451965.926693, 'code': 0, 'valid': True}} >Thread-29118::DEBUG::2012-12-02 14:39:30,988::task::568::TaskManager.Task::(_updateState) Task=`de043af5-c46d-45c3-9819-4bd691511cc8`::moving from state preparing -> state finished >Thread-29118::DEBUG::2012-12-02 14:39:30,988::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29118::DEBUG::2012-12-02 14:39:30,989::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29118::DEBUG::2012-12-02 14:39:30,989::task::957::TaskManager.Task::(_decref) Task=`de043af5-c46d-45c3-9819-4bd691511cc8`::ref 0 aborting False >Thread-29123::DEBUG::2012-12-02 14:39:38,636::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29123::DEBUG::2012-12-02 14:39:38,637::task::568::TaskManager.Task::(_updateState) Task=`3a350d3a-3a72-4674-9f55-3b641f6bf8d8`::moving from state init -> state preparing >Thread-29123::INFO::2012-12-02 14:39:38,637::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29123::INFO::2012-12-02 14:39:38,637::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29123::DEBUG::2012-12-02 14:39:38,637::task::1151::TaskManager.Task::(prepare) Task=`3a350d3a-3a72-4674-9f55-3b641f6bf8d8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29123::DEBUG::2012-12-02 14:39:38,637::task::568::TaskManager.Task::(_updateState) Task=`3a350d3a-3a72-4674-9f55-3b641f6bf8d8`::moving from state preparing -> state finished >Thread-29123::DEBUG::2012-12-02 14:39:38,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29123::DEBUG::2012-12-02 14:39:38,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29123::DEBUG::2012-12-02 14:39:38,638::task::957::TaskManager.Task::(_decref) Task=`3a350d3a-3a72-4674-9f55-3b641f6bf8d8`::ref 0 aborting False >Thread-29124::DEBUG::2012-12-02 14:39:38,640::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29124::DEBUG::2012-12-02 14:39:38,641::task::568::TaskManager.Task::(_updateState) Task=`c340f263-4d74-4767-b5db-29c28122aab5`::moving from state init -> state preparing >Thread-29124::INFO::2012-12-02 14:39:38,641::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29124::DEBUG::2012-12-02 14:39:38,641::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ac0cf232-3eac-4349-940f-c98914c36655`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29124::DEBUG::2012-12-02 14:39:38,641::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29124::DEBUG::2012-12-02 14:39:38,641::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29124::DEBUG::2012-12-02 14:39:38,641::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ac0cf232-3eac-4349-940f-c98914c36655`::Granted request >Thread-29124::DEBUG::2012-12-02 14:39:38,642::task::794::TaskManager.Task::(resourceAcquired) Task=`c340f263-4d74-4767-b5db-29c28122aab5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29124::DEBUG::2012-12-02 14:39:38,642::task::957::TaskManager.Task::(_decref) Task=`c340f263-4d74-4767-b5db-29c28122aab5`::ref 1 aborting False >Thread-29124::INFO::2012-12-02 14:39:38,647::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29124::DEBUG::2012-12-02 14:39:38,647::task::1151::TaskManager.Task::(prepare) Task=`c340f263-4d74-4767-b5db-29c28122aab5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29124::DEBUG::2012-12-02 14:39:38,647::task::568::TaskManager.Task::(_updateState) Task=`c340f263-4d74-4767-b5db-29c28122aab5`::moving from state preparing -> state finished >Thread-29124::DEBUG::2012-12-02 14:39:38,648::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29124::DEBUG::2012-12-02 14:39:38,648::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29124::DEBUG::2012-12-02 14:39:38,648::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29124::DEBUG::2012-12-02 14:39:38,648::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29124::DEBUG::2012-12-02 14:39:38,648::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29124::DEBUG::2012-12-02 14:39:38,648::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29124::DEBUG::2012-12-02 14:39:38,648::task::957::TaskManager.Task::(_decref) Task=`c340f263-4d74-4767-b5db-29c28122aab5`::ref 0 aborting False >Thread-29126::DEBUG::2012-12-02 14:39:41,250::task::568::TaskManager.Task::(_updateState) Task=`b498d3d9-f010-42a2-909b-bc6ae87c3102`::moving from state init -> state preparing >Thread-29126::INFO::2012-12-02 14:39:41,251::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29126::INFO::2012-12-02 14:39:41,251::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00778913497925', 'lastCheck': 1354451979.5974171, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00285196304321', 'lastCheck': 1354451975.8832409, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00268220901489', 'lastCheck': 1354451976.7519319, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0060338973999', 'lastCheck': 1354451975.941494, 'code': 0, 'valid': True}} >Thread-29126::DEBUG::2012-12-02 14:39:41,251::task::1151::TaskManager.Task::(prepare) Task=`b498d3d9-f010-42a2-909b-bc6ae87c3102`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00778913497925', 'lastCheck': 1354451979.5974171, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00285196304321', 'lastCheck': 1354451975.8832409, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00268220901489', 'lastCheck': 1354451976.7519319, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0060338973999', 'lastCheck': 1354451975.941494, 'code': 0, 'valid': True}} >Thread-29126::DEBUG::2012-12-02 14:39:41,251::task::568::TaskManager.Task::(_updateState) Task=`b498d3d9-f010-42a2-909b-bc6ae87c3102`::moving from state preparing -> state finished >Thread-29126::DEBUG::2012-12-02 14:39:41,251::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29126::DEBUG::2012-12-02 14:39:41,251::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29126::DEBUG::2012-12-02 14:39:41,252::task::957::TaskManager.Task::(_decref) Task=`b498d3d9-f010-42a2-909b-bc6ae87c3102`::ref 0 aborting False >Thread-29131::DEBUG::2012-12-02 14:39:48,756::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29131::DEBUG::2012-12-02 14:39:48,756::task::568::TaskManager.Task::(_updateState) Task=`f98c5ddf-f5e8-41e6-8c4f-9df93ec79e19`::moving from state init -> state preparing >Thread-29131::INFO::2012-12-02 14:39:48,756::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29131::INFO::2012-12-02 14:39:48,757::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29131::DEBUG::2012-12-02 14:39:48,757::task::1151::TaskManager.Task::(prepare) Task=`f98c5ddf-f5e8-41e6-8c4f-9df93ec79e19`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29131::DEBUG::2012-12-02 14:39:48,757::task::568::TaskManager.Task::(_updateState) Task=`f98c5ddf-f5e8-41e6-8c4f-9df93ec79e19`::moving from state preparing -> state finished >Thread-29131::DEBUG::2012-12-02 14:39:48,757::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29131::DEBUG::2012-12-02 14:39:48,757::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29131::DEBUG::2012-12-02 14:39:48,757::task::957::TaskManager.Task::(_decref) Task=`f98c5ddf-f5e8-41e6-8c4f-9df93ec79e19`::ref 0 aborting False >Thread-29132::DEBUG::2012-12-02 14:39:48,760::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29132::DEBUG::2012-12-02 14:39:48,760::task::568::TaskManager.Task::(_updateState) Task=`5d400585-0082-41e7-8744-9b51d9fa3654`::moving from state init -> state preparing >Thread-29132::INFO::2012-12-02 14:39:48,760::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29132::DEBUG::2012-12-02 14:39:48,760::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0c459948-66ca-420f-b5d7-93575841049c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29132::DEBUG::2012-12-02 14:39:48,761::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29132::DEBUG::2012-12-02 14:39:48,761::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29132::DEBUG::2012-12-02 14:39:48,761::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0c459948-66ca-420f-b5d7-93575841049c`::Granted request >Thread-29132::DEBUG::2012-12-02 14:39:48,761::task::794::TaskManager.Task::(resourceAcquired) Task=`5d400585-0082-41e7-8744-9b51d9fa3654`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29132::DEBUG::2012-12-02 14:39:48,761::task::957::TaskManager.Task::(_decref) Task=`5d400585-0082-41e7-8744-9b51d9fa3654`::ref 1 aborting False >Thread-29132::INFO::2012-12-02 14:39:48,767::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29132::DEBUG::2012-12-02 14:39:48,767::task::1151::TaskManager.Task::(prepare) Task=`5d400585-0082-41e7-8744-9b51d9fa3654`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29132::DEBUG::2012-12-02 14:39:48,767::task::568::TaskManager.Task::(_updateState) Task=`5d400585-0082-41e7-8744-9b51d9fa3654`::moving from state preparing -> state finished >Thread-29132::DEBUG::2012-12-02 14:39:48,767::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29132::DEBUG::2012-12-02 14:39:48,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29132::DEBUG::2012-12-02 14:39:48,768::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29132::DEBUG::2012-12-02 14:39:48,768::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29132::DEBUG::2012-12-02 14:39:48,768::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29132::DEBUG::2012-12-02 14:39:48,768::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29132::DEBUG::2012-12-02 14:39:48,768::task::957::TaskManager.Task::(_decref) Task=`5d400585-0082-41e7-8744-9b51d9fa3654`::ref 0 aborting False >Thread-29134::DEBUG::2012-12-02 14:39:51,460::task::568::TaskManager.Task::(_updateState) Task=`eb88573a-b66a-48db-81de-77565471549f`::moving from state init -> state preparing >Thread-29134::INFO::2012-12-02 14:39:51,460::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29134::INFO::2012-12-02 14:39:51,460::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00765895843506', 'lastCheck': 1354451989.615092, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180006027222', 'lastCheck': 1354451985.89588, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213003158569', 'lastCheck': 1354451986.7618959, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00673913955688', 'lastCheck': 1354451985.9582961, 'code': 0, 'valid': True}} >Thread-29134::DEBUG::2012-12-02 14:39:51,460::task::1151::TaskManager.Task::(prepare) Task=`eb88573a-b66a-48db-81de-77565471549f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00765895843506', 'lastCheck': 1354451989.615092, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180006027222', 'lastCheck': 1354451985.89588, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00213003158569', 'lastCheck': 1354451986.7618959, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00673913955688', 'lastCheck': 1354451985.9582961, 'code': 0, 'valid': True}} >Thread-29134::DEBUG::2012-12-02 14:39:51,460::task::568::TaskManager.Task::(_updateState) Task=`eb88573a-b66a-48db-81de-77565471549f`::moving from state preparing -> state finished >Thread-29134::DEBUG::2012-12-02 14:39:51,461::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29134::DEBUG::2012-12-02 14:39:51,461::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29134::DEBUG::2012-12-02 14:39:51,461::task::957::TaskManager.Task::(_decref) Task=`eb88573a-b66a-48db-81de-77565471549f`::ref 0 aborting False >Thread-29139::DEBUG::2012-12-02 14:39:58,923::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29139::DEBUG::2012-12-02 14:39:58,923::task::568::TaskManager.Task::(_updateState) Task=`84c70da9-ce47-4189-8808-5400a381d2ec`::moving from state init -> state preparing >Thread-29139::INFO::2012-12-02 14:39:58,923::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29139::INFO::2012-12-02 14:39:58,923::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29139::DEBUG::2012-12-02 14:39:58,924::task::1151::TaskManager.Task::(prepare) Task=`84c70da9-ce47-4189-8808-5400a381d2ec`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29139::DEBUG::2012-12-02 14:39:58,924::task::568::TaskManager.Task::(_updateState) Task=`84c70da9-ce47-4189-8808-5400a381d2ec`::moving from state preparing -> state finished >Thread-29139::DEBUG::2012-12-02 14:39:58,924::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29139::DEBUG::2012-12-02 14:39:58,924::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29139::DEBUG::2012-12-02 14:39:58,924::task::957::TaskManager.Task::(_decref) Task=`84c70da9-ce47-4189-8808-5400a381d2ec`::ref 0 aborting False >Thread-29140::DEBUG::2012-12-02 14:39:58,927::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29140::DEBUG::2012-12-02 14:39:58,927::task::568::TaskManager.Task::(_updateState) Task=`4d643145-92db-432f-addf-102034b73b40`::moving from state init -> state preparing >Thread-29140::INFO::2012-12-02 14:39:58,927::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29140::DEBUG::2012-12-02 14:39:58,927::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0c8a125d-e173-4f05-8617-a30cef5d7eda`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29140::DEBUG::2012-12-02 14:39:58,927::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29140::DEBUG::2012-12-02 14:39:58,927::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29140::DEBUG::2012-12-02 14:39:58,928::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0c8a125d-e173-4f05-8617-a30cef5d7eda`::Granted request >Thread-29140::DEBUG::2012-12-02 14:39:58,928::task::794::TaskManager.Task::(resourceAcquired) Task=`4d643145-92db-432f-addf-102034b73b40`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29140::DEBUG::2012-12-02 14:39:58,928::task::957::TaskManager.Task::(_decref) Task=`4d643145-92db-432f-addf-102034b73b40`::ref 1 aborting False >Thread-29140::INFO::2012-12-02 14:39:58,932::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29140::DEBUG::2012-12-02 14:39:58,932::task::1151::TaskManager.Task::(prepare) Task=`4d643145-92db-432f-addf-102034b73b40`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29140::DEBUG::2012-12-02 14:39:58,932::task::568::TaskManager.Task::(_updateState) Task=`4d643145-92db-432f-addf-102034b73b40`::moving from state preparing -> state finished >Thread-29140::DEBUG::2012-12-02 14:39:58,932::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29140::DEBUG::2012-12-02 14:39:58,933::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29140::DEBUG::2012-12-02 14:39:58,933::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29140::DEBUG::2012-12-02 14:39:58,933::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29140::DEBUG::2012-12-02 14:39:58,933::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29140::DEBUG::2012-12-02 14:39:58,933::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29140::DEBUG::2012-12-02 14:39:58,933::task::957::TaskManager.Task::(_decref) Task=`4d643145-92db-432f-addf-102034b73b40`::ref 0 aborting False >Thread-29142::DEBUG::2012-12-02 14:40:01,577::task::568::TaskManager.Task::(_updateState) Task=`9bc665c7-252a-4622-aba9-d10813be6d58`::moving from state init -> state preparing >Thread-29142::INFO::2012-12-02 14:40:01,577::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29142::INFO::2012-12-02 14:40:01,577::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00865793228149', 'lastCheck': 1354451999.6338711, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00189900398254', 'lastCheck': 1354451995.9074969, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0020809173584', 'lastCheck': 1354451996.772238, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00603699684143', 'lastCheck': 1354451995.972894, 'code': 0, 'valid': True}} >Thread-29142::DEBUG::2012-12-02 14:40:01,577::task::1151::TaskManager.Task::(prepare) Task=`9bc665c7-252a-4622-aba9-d10813be6d58`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00865793228149', 'lastCheck': 1354451999.6338711, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00189900398254', 'lastCheck': 1354451995.9074969, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0020809173584', 'lastCheck': 1354451996.772238, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00603699684143', 'lastCheck': 1354451995.972894, 'code': 0, 'valid': True}} >Thread-29142::DEBUG::2012-12-02 14:40:01,577::task::568::TaskManager.Task::(_updateState) Task=`9bc665c7-252a-4622-aba9-d10813be6d58`::moving from state preparing -> state finished >Thread-29142::DEBUG::2012-12-02 14:40:01,578::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29142::DEBUG::2012-12-02 14:40:01,578::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29142::DEBUG::2012-12-02 14:40:01,578::task::957::TaskManager.Task::(_decref) Task=`9bc665c7-252a-4622-aba9-d10813be6d58`::ref 0 aborting False >Thread-29147::DEBUG::2012-12-02 14:40:09,136::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29147::DEBUG::2012-12-02 14:40:09,137::task::568::TaskManager.Task::(_updateState) Task=`56074352-50e7-46af-bcb2-e27df5cca594`::moving from state init -> state preparing >Thread-29147::INFO::2012-12-02 14:40:09,137::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29147::INFO::2012-12-02 14:40:09,137::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29147::DEBUG::2012-12-02 14:40:09,137::task::1151::TaskManager.Task::(prepare) Task=`56074352-50e7-46af-bcb2-e27df5cca594`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29147::DEBUG::2012-12-02 14:40:09,137::task::568::TaskManager.Task::(_updateState) Task=`56074352-50e7-46af-bcb2-e27df5cca594`::moving from state preparing -> state finished >Thread-29147::DEBUG::2012-12-02 14:40:09,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29147::DEBUG::2012-12-02 14:40:09,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29147::DEBUG::2012-12-02 14:40:09,138::task::957::TaskManager.Task::(_decref) Task=`56074352-50e7-46af-bcb2-e27df5cca594`::ref 0 aborting False >Thread-29148::DEBUG::2012-12-02 14:40:09,140::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29148::DEBUG::2012-12-02 14:40:09,141::task::568::TaskManager.Task::(_updateState) Task=`dd3c4581-e863-470b-8b3e-2b9f00ff5d5b`::moving from state init -> state preparing >Thread-29148::INFO::2012-12-02 14:40:09,141::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29148::DEBUG::2012-12-02 14:40:09,141::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7e8b1889-76fa-44bc-b22e-bf9c400ed256`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29148::DEBUG::2012-12-02 14:40:09,141::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29148::DEBUG::2012-12-02 14:40:09,141::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29148::DEBUG::2012-12-02 14:40:09,141::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7e8b1889-76fa-44bc-b22e-bf9c400ed256`::Granted request >Thread-29148::DEBUG::2012-12-02 14:40:09,142::task::794::TaskManager.Task::(resourceAcquired) Task=`dd3c4581-e863-470b-8b3e-2b9f00ff5d5b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29148::DEBUG::2012-12-02 14:40:09,142::task::957::TaskManager.Task::(_decref) Task=`dd3c4581-e863-470b-8b3e-2b9f00ff5d5b`::ref 1 aborting False >Thread-29148::INFO::2012-12-02 14:40:09,148::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29148::DEBUG::2012-12-02 14:40:09,148::task::1151::TaskManager.Task::(prepare) Task=`dd3c4581-e863-470b-8b3e-2b9f00ff5d5b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29148::DEBUG::2012-12-02 14:40:09,148::task::568::TaskManager.Task::(_updateState) Task=`dd3c4581-e863-470b-8b3e-2b9f00ff5d5b`::moving from state preparing -> state finished >Thread-29148::DEBUG::2012-12-02 14:40:09,148::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29148::DEBUG::2012-12-02 14:40:09,148::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29148::DEBUG::2012-12-02 14:40:09,148::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29148::DEBUG::2012-12-02 14:40:09,148::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29148::DEBUG::2012-12-02 14:40:09,148::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29148::DEBUG::2012-12-02 14:40:09,148::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29148::DEBUG::2012-12-02 14:40:09,148::task::957::TaskManager.Task::(_decref) Task=`dd3c4581-e863-470b-8b3e-2b9f00ff5d5b`::ref 0 aborting False >Thread-29150::DEBUG::2012-12-02 14:40:11,783::task::568::TaskManager.Task::(_updateState) Task=`f2530b36-b02b-44fd-bae2-45b69d8d26d6`::moving from state init -> state preparing >Thread-29150::INFO::2012-12-02 14:40:11,783::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29150::INFO::2012-12-02 14:40:11,783::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00744199752808', 'lastCheck': 1354452009.6510749, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00195288658142', 'lastCheck': 1354452005.919271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00192213058472', 'lastCheck': 1354452006.7826779, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00695395469666', 'lastCheck': 1354452005.988461, 'code': 0, 'valid': True}} >Thread-29150::DEBUG::2012-12-02 14:40:11,783::task::1151::TaskManager.Task::(prepare) Task=`f2530b36-b02b-44fd-bae2-45b69d8d26d6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00744199752808', 'lastCheck': 1354452009.6510749, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00195288658142', 'lastCheck': 1354452005.919271, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00192213058472', 'lastCheck': 1354452006.7826779, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00695395469666', 'lastCheck': 1354452005.988461, 'code': 0, 'valid': True}} >Thread-29150::DEBUG::2012-12-02 14:40:11,783::task::568::TaskManager.Task::(_updateState) Task=`f2530b36-b02b-44fd-bae2-45b69d8d26d6`::moving from state preparing -> state finished >Thread-29150::DEBUG::2012-12-02 14:40:11,783::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29150::DEBUG::2012-12-02 14:40:11,784::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29150::DEBUG::2012-12-02 14:40:11,784::task::957::TaskManager.Task::(_decref) Task=`f2530b36-b02b-44fd-bae2-45b69d8d26d6`::ref 0 aborting False >Thread-29155::DEBUG::2012-12-02 14:40:19,274::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29155::DEBUG::2012-12-02 14:40:19,275::task::568::TaskManager.Task::(_updateState) Task=`fe94ac3b-b6c6-4de5-8b8b-4c3de3365bb7`::moving from state init -> state preparing >Thread-29155::INFO::2012-12-02 14:40:19,275::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29155::INFO::2012-12-02 14:40:19,275::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29155::DEBUG::2012-12-02 14:40:19,275::task::1151::TaskManager.Task::(prepare) Task=`fe94ac3b-b6c6-4de5-8b8b-4c3de3365bb7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29155::DEBUG::2012-12-02 14:40:19,275::task::568::TaskManager.Task::(_updateState) Task=`fe94ac3b-b6c6-4de5-8b8b-4c3de3365bb7`::moving from state preparing -> state finished >Thread-29155::DEBUG::2012-12-02 14:40:19,275::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29155::DEBUG::2012-12-02 14:40:19,276::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29155::DEBUG::2012-12-02 14:40:19,276::task::957::TaskManager.Task::(_decref) Task=`fe94ac3b-b6c6-4de5-8b8b-4c3de3365bb7`::ref 0 aborting False >Thread-29156::DEBUG::2012-12-02 14:40:19,278::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29156::DEBUG::2012-12-02 14:40:19,278::task::568::TaskManager.Task::(_updateState) Task=`6971cc92-20ea-4330-9773-45425c4d64ba`::moving from state init -> state preparing >Thread-29156::INFO::2012-12-02 14:40:19,278::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29156::DEBUG::2012-12-02 14:40:19,279::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`edcec862-4f3d-441c-81ed-99a14b8b5b53`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29156::DEBUG::2012-12-02 14:40:19,279::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29156::DEBUG::2012-12-02 14:40:19,279::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29156::DEBUG::2012-12-02 14:40:19,279::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`edcec862-4f3d-441c-81ed-99a14b8b5b53`::Granted request >Thread-29156::DEBUG::2012-12-02 14:40:19,279::task::794::TaskManager.Task::(resourceAcquired) Task=`6971cc92-20ea-4330-9773-45425c4d64ba`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29156::DEBUG::2012-12-02 14:40:19,280::task::957::TaskManager.Task::(_decref) Task=`6971cc92-20ea-4330-9773-45425c4d64ba`::ref 1 aborting False >Thread-29156::INFO::2012-12-02 14:40:19,286::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29156::DEBUG::2012-12-02 14:40:19,286::task::1151::TaskManager.Task::(prepare) Task=`6971cc92-20ea-4330-9773-45425c4d64ba`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29156::DEBUG::2012-12-02 14:40:19,286::task::568::TaskManager.Task::(_updateState) Task=`6971cc92-20ea-4330-9773-45425c4d64ba`::moving from state preparing -> state finished >Thread-29156::DEBUG::2012-12-02 14:40:19,286::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29156::DEBUG::2012-12-02 14:40:19,286::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29156::DEBUG::2012-12-02 14:40:19,286::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29156::DEBUG::2012-12-02 14:40:19,286::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29156::DEBUG::2012-12-02 14:40:19,286::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29156::DEBUG::2012-12-02 14:40:19,286::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29156::DEBUG::2012-12-02 14:40:19,287::task::957::TaskManager.Task::(_decref) Task=`6971cc92-20ea-4330-9773-45425c4d64ba`::ref 0 aborting False >Thread-29158::DEBUG::2012-12-02 14:40:21,874::task::568::TaskManager.Task::(_updateState) Task=`45adacad-c0c0-4cc4-bfe2-e72bc17cb41a`::moving from state init -> state preparing >Thread-29158::INFO::2012-12-02 14:40:21,874::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29158::INFO::2012-12-02 14:40:21,875::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00789999961853', 'lastCheck': 1354452019.668124, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00298690795898', 'lastCheck': 1354452015.9338369, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253319740295', 'lastCheck': 1354452016.7938881, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0141091346741', 'lastCheck': 1354452016.0120139, 'code': 0, 'valid': True}} >Thread-29158::DEBUG::2012-12-02 14:40:21,875::task::1151::TaskManager.Task::(prepare) Task=`45adacad-c0c0-4cc4-bfe2-e72bc17cb41a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00789999961853', 'lastCheck': 1354452019.668124, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00298690795898', 'lastCheck': 1354452015.9338369, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253319740295', 'lastCheck': 1354452016.7938881, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0141091346741', 'lastCheck': 1354452016.0120139, 'code': 0, 'valid': True}} >Thread-29158::DEBUG::2012-12-02 14:40:21,875::task::568::TaskManager.Task::(_updateState) Task=`45adacad-c0c0-4cc4-bfe2-e72bc17cb41a`::moving from state preparing -> state finished >Thread-29158::DEBUG::2012-12-02 14:40:21,875::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29158::DEBUG::2012-12-02 14:40:21,875::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29158::DEBUG::2012-12-02 14:40:21,875::task::957::TaskManager.Task::(_decref) Task=`45adacad-c0c0-4cc4-bfe2-e72bc17cb41a`::ref 0 aborting False >Thread-29163::DEBUG::2012-12-02 14:40:29,404::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29163::DEBUG::2012-12-02 14:40:29,405::task::568::TaskManager.Task::(_updateState) Task=`282598a4-d09d-442d-9ef6-f85b6ad96b88`::moving from state init -> state preparing >Thread-29163::INFO::2012-12-02 14:40:29,405::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29163::INFO::2012-12-02 14:40:29,405::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29163::DEBUG::2012-12-02 14:40:29,405::task::1151::TaskManager.Task::(prepare) Task=`282598a4-d09d-442d-9ef6-f85b6ad96b88`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29163::DEBUG::2012-12-02 14:40:29,405::task::568::TaskManager.Task::(_updateState) Task=`282598a4-d09d-442d-9ef6-f85b6ad96b88`::moving from state preparing -> state finished >Thread-29163::DEBUG::2012-12-02 14:40:29,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29163::DEBUG::2012-12-02 14:40:29,406::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29163::DEBUG::2012-12-02 14:40:29,406::task::957::TaskManager.Task::(_decref) Task=`282598a4-d09d-442d-9ef6-f85b6ad96b88`::ref 0 aborting False >Thread-29164::DEBUG::2012-12-02 14:40:29,408::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29164::DEBUG::2012-12-02 14:40:29,409::task::568::TaskManager.Task::(_updateState) Task=`66159c82-560d-4330-ac6d-579df6ef6ee8`::moving from state init -> state preparing >Thread-29164::INFO::2012-12-02 14:40:29,409::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29164::DEBUG::2012-12-02 14:40:29,409::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2643a929-ae58-4b24-9e8a-aaf0944adabd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29164::DEBUG::2012-12-02 14:40:29,409::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29164::DEBUG::2012-12-02 14:40:29,409::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29164::DEBUG::2012-12-02 14:40:29,409::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2643a929-ae58-4b24-9e8a-aaf0944adabd`::Granted request >Thread-29164::DEBUG::2012-12-02 14:40:29,410::task::794::TaskManager.Task::(resourceAcquired) Task=`66159c82-560d-4330-ac6d-579df6ef6ee8`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29164::DEBUG::2012-12-02 14:40:29,410::task::957::TaskManager.Task::(_decref) Task=`66159c82-560d-4330-ac6d-579df6ef6ee8`::ref 1 aborting False >Thread-29164::INFO::2012-12-02 14:40:29,416::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29164::DEBUG::2012-12-02 14:40:29,416::task::1151::TaskManager.Task::(prepare) Task=`66159c82-560d-4330-ac6d-579df6ef6ee8`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29164::DEBUG::2012-12-02 14:40:29,416::task::568::TaskManager.Task::(_updateState) Task=`66159c82-560d-4330-ac6d-579df6ef6ee8`::moving from state preparing -> state finished >Thread-29164::DEBUG::2012-12-02 14:40:29,416::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29164::DEBUG::2012-12-02 14:40:29,416::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29164::DEBUG::2012-12-02 14:40:29,416::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29164::DEBUG::2012-12-02 14:40:29,416::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29164::DEBUG::2012-12-02 14:40:29,417::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29164::DEBUG::2012-12-02 14:40:29,417::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29164::DEBUG::2012-12-02 14:40:29,417::task::957::TaskManager.Task::(_decref) Task=`66159c82-560d-4330-ac6d-579df6ef6ee8`::ref 0 aborting False >Thread-29166::DEBUG::2012-12-02 14:40:32,003::task::568::TaskManager.Task::(_updateState) Task=`406e2da4-8a1d-4c83-9af4-3f0e93d389d7`::moving from state init -> state preparing >Thread-29166::INFO::2012-12-02 14:40:32,003::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29166::INFO::2012-12-02 14:40:32,004::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00792789459229', 'lastCheck': 1354452029.685668, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00214505195618', 'lastCheck': 1354452025.9483521, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00162887573242', 'lastCheck': 1354452026.8028259, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0110669136047', 'lastCheck': 1354452026.0329449, 'code': 0, 'valid': True}} >Thread-29166::DEBUG::2012-12-02 14:40:32,004::task::1151::TaskManager.Task::(prepare) Task=`406e2da4-8a1d-4c83-9af4-3f0e93d389d7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00792789459229', 'lastCheck': 1354452029.685668, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00214505195618', 'lastCheck': 1354452025.9483521, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00162887573242', 'lastCheck': 1354452026.8028259, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0110669136047', 'lastCheck': 1354452026.0329449, 'code': 0, 'valid': True}} >Thread-29166::DEBUG::2012-12-02 14:40:32,004::task::568::TaskManager.Task::(_updateState) Task=`406e2da4-8a1d-4c83-9af4-3f0e93d389d7`::moving from state preparing -> state finished >Thread-29166::DEBUG::2012-12-02 14:40:32,004::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29166::DEBUG::2012-12-02 14:40:32,004::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29166::DEBUG::2012-12-02 14:40:32,004::task::957::TaskManager.Task::(_decref) Task=`406e2da4-8a1d-4c83-9af4-3f0e93d389d7`::ref 0 aborting False >Thread-29171::DEBUG::2012-12-02 14:40:39,590::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29171::DEBUG::2012-12-02 14:40:39,591::task::568::TaskManager.Task::(_updateState) Task=`897d62c3-3c19-47c4-b4da-ed0e8e9ef93e`::moving from state init -> state preparing >Thread-29171::INFO::2012-12-02 14:40:39,591::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29171::INFO::2012-12-02 14:40:39,591::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29171::DEBUG::2012-12-02 14:40:39,591::task::1151::TaskManager.Task::(prepare) Task=`897d62c3-3c19-47c4-b4da-ed0e8e9ef93e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29171::DEBUG::2012-12-02 14:40:39,591::task::568::TaskManager.Task::(_updateState) Task=`897d62c3-3c19-47c4-b4da-ed0e8e9ef93e`::moving from state preparing -> state finished >Thread-29171::DEBUG::2012-12-02 14:40:39,592::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29171::DEBUG::2012-12-02 14:40:39,592::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29171::DEBUG::2012-12-02 14:40:39,592::task::957::TaskManager.Task::(_decref) Task=`897d62c3-3c19-47c4-b4da-ed0e8e9ef93e`::ref 0 aborting False >Thread-29172::DEBUG::2012-12-02 14:40:39,594::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29172::DEBUG::2012-12-02 14:40:39,595::task::568::TaskManager.Task::(_updateState) Task=`f5eac878-0456-4181-9249-f8a99281f628`::moving from state init -> state preparing >Thread-29172::INFO::2012-12-02 14:40:39,595::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29172::DEBUG::2012-12-02 14:40:39,595::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`72ca6c97-7531-4f88-9165-68c5fee08493`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29172::DEBUG::2012-12-02 14:40:39,595::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29172::DEBUG::2012-12-02 14:40:39,595::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29172::DEBUG::2012-12-02 14:40:39,595::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`72ca6c97-7531-4f88-9165-68c5fee08493`::Granted request >Thread-29172::DEBUG::2012-12-02 14:40:39,596::task::794::TaskManager.Task::(resourceAcquired) Task=`f5eac878-0456-4181-9249-f8a99281f628`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29172::DEBUG::2012-12-02 14:40:39,596::task::957::TaskManager.Task::(_decref) Task=`f5eac878-0456-4181-9249-f8a99281f628`::ref 1 aborting False >Thread-29172::INFO::2012-12-02 14:40:39,600::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29172::DEBUG::2012-12-02 14:40:39,600::task::1151::TaskManager.Task::(prepare) Task=`f5eac878-0456-4181-9249-f8a99281f628`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29172::DEBUG::2012-12-02 14:40:39,600::task::568::TaskManager.Task::(_updateState) Task=`f5eac878-0456-4181-9249-f8a99281f628`::moving from state preparing -> state finished >Thread-29172::DEBUG::2012-12-02 14:40:39,601::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29172::DEBUG::2012-12-02 14:40:39,601::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29172::DEBUG::2012-12-02 14:40:39,601::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29172::DEBUG::2012-12-02 14:40:39,601::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29172::DEBUG::2012-12-02 14:40:39,601::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29172::DEBUG::2012-12-02 14:40:39,601::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29172::DEBUG::2012-12-02 14:40:39,601::task::957::TaskManager.Task::(_decref) Task=`f5eac878-0456-4181-9249-f8a99281f628`::ref 0 aborting False >Thread-29174::DEBUG::2012-12-02 14:40:42,172::task::568::TaskManager.Task::(_updateState) Task=`d7e608a2-ce6e-4a23-9b32-855a11c5d1c9`::moving from state init -> state preparing >Thread-29174::INFO::2012-12-02 14:40:42,173::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29174::INFO::2012-12-02 14:40:42,173::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830602645874', 'lastCheck': 1354452039.705615, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00400900840759', 'lastCheck': 1354452035.962791, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00278401374817', 'lastCheck': 1354452036.8145931, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00585293769836', 'lastCheck': 1354452036.0476739, 'code': 0, 'valid': True}} >Thread-29174::DEBUG::2012-12-02 14:40:42,173::task::1151::TaskManager.Task::(prepare) Task=`d7e608a2-ce6e-4a23-9b32-855a11c5d1c9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830602645874', 'lastCheck': 1354452039.705615, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00400900840759', 'lastCheck': 1354452035.962791, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00278401374817', 'lastCheck': 1354452036.8145931, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00585293769836', 'lastCheck': 1354452036.0476739, 'code': 0, 'valid': True}} >Thread-29174::DEBUG::2012-12-02 14:40:42,173::task::568::TaskManager.Task::(_updateState) Task=`d7e608a2-ce6e-4a23-9b32-855a11c5d1c9`::moving from state preparing -> state finished >Thread-29174::DEBUG::2012-12-02 14:40:42,173::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29174::DEBUG::2012-12-02 14:40:42,173::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29174::DEBUG::2012-12-02 14:40:42,174::task::957::TaskManager.Task::(_decref) Task=`d7e608a2-ce6e-4a23-9b32-855a11c5d1c9`::ref 0 aborting False >Thread-29179::DEBUG::2012-12-02 14:40:49,728::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29179::DEBUG::2012-12-02 14:40:49,728::task::568::TaskManager.Task::(_updateState) Task=`ef1f9a9c-cdc2-463f-aeb6-457bd6235280`::moving from state init -> state preparing >Thread-29179::INFO::2012-12-02 14:40:49,728::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29179::INFO::2012-12-02 14:40:49,729::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29179::DEBUG::2012-12-02 14:40:49,729::task::1151::TaskManager.Task::(prepare) Task=`ef1f9a9c-cdc2-463f-aeb6-457bd6235280`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29179::DEBUG::2012-12-02 14:40:49,729::task::568::TaskManager.Task::(_updateState) Task=`ef1f9a9c-cdc2-463f-aeb6-457bd6235280`::moving from state preparing -> state finished >Thread-29179::DEBUG::2012-12-02 14:40:49,729::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29179::DEBUG::2012-12-02 14:40:49,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29179::DEBUG::2012-12-02 14:40:49,729::task::957::TaskManager.Task::(_decref) Task=`ef1f9a9c-cdc2-463f-aeb6-457bd6235280`::ref 0 aborting False >Thread-29180::DEBUG::2012-12-02 14:40:49,732::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29180::DEBUG::2012-12-02 14:40:49,732::task::568::TaskManager.Task::(_updateState) Task=`9c627a19-28ea-4442-88eb-9b2ac8937e68`::moving from state init -> state preparing >Thread-29180::INFO::2012-12-02 14:40:49,732::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29180::DEBUG::2012-12-02 14:40:49,733::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a3ba420c-9103-446c-a0c2-daf440709884`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29180::DEBUG::2012-12-02 14:40:49,733::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29180::DEBUG::2012-12-02 14:40:49,733::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29180::DEBUG::2012-12-02 14:40:49,733::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a3ba420c-9103-446c-a0c2-daf440709884`::Granted request >Thread-29180::DEBUG::2012-12-02 14:40:49,733::task::794::TaskManager.Task::(resourceAcquired) Task=`9c627a19-28ea-4442-88eb-9b2ac8937e68`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29180::DEBUG::2012-12-02 14:40:49,734::task::957::TaskManager.Task::(_decref) Task=`9c627a19-28ea-4442-88eb-9b2ac8937e68`::ref 1 aborting False >Thread-29180::INFO::2012-12-02 14:40:49,738::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29180::DEBUG::2012-12-02 14:40:49,738::task::1151::TaskManager.Task::(prepare) Task=`9c627a19-28ea-4442-88eb-9b2ac8937e68`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29180::DEBUG::2012-12-02 14:40:49,739::task::568::TaskManager.Task::(_updateState) Task=`9c627a19-28ea-4442-88eb-9b2ac8937e68`::moving from state preparing -> state finished >Thread-29180::DEBUG::2012-12-02 14:40:49,739::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29180::DEBUG::2012-12-02 14:40:49,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29180::DEBUG::2012-12-02 14:40:49,739::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29180::DEBUG::2012-12-02 14:40:49,739::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29180::DEBUG::2012-12-02 14:40:49,739::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29180::DEBUG::2012-12-02 14:40:49,739::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29180::DEBUG::2012-12-02 14:40:49,739::task::957::TaskManager.Task::(_decref) Task=`9c627a19-28ea-4442-88eb-9b2ac8937e68`::ref 0 aborting False >Thread-29182::DEBUG::2012-12-02 14:40:52,244::task::568::TaskManager.Task::(_updateState) Task=`28b55625-79dc-4b80-b42e-82271726fbbd`::moving from state init -> state preparing >Thread-29182::INFO::2012-12-02 14:40:52,244::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29182::INFO::2012-12-02 14:40:52,244::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00750303268433', 'lastCheck': 1354452049.722846, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00218200683594', 'lastCheck': 1354452045.9784441, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172996520996', 'lastCheck': 1354452046.825278, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821208953857', 'lastCheck': 1354452046.0702829, 'code': 0, 'valid': True}} >Thread-29182::DEBUG::2012-12-02 14:40:52,244::task::1151::TaskManager.Task::(prepare) Task=`28b55625-79dc-4b80-b42e-82271726fbbd`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00750303268433', 'lastCheck': 1354452049.722846, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00218200683594', 'lastCheck': 1354452045.9784441, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00172996520996', 'lastCheck': 1354452046.825278, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00821208953857', 'lastCheck': 1354452046.0702829, 'code': 0, 'valid': True}} >Thread-29182::DEBUG::2012-12-02 14:40:52,244::task::568::TaskManager.Task::(_updateState) Task=`28b55625-79dc-4b80-b42e-82271726fbbd`::moving from state preparing -> state finished >Thread-29182::DEBUG::2012-12-02 14:40:52,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29182::DEBUG::2012-12-02 14:40:52,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29182::DEBUG::2012-12-02 14:40:52,245::task::957::TaskManager.Task::(_decref) Task=`28b55625-79dc-4b80-b42e-82271726fbbd`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:40:56,825::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:40:56,825::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:40:56,826::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:40:57,031::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:40:57,032::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:40:57,033::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29234::WARNING::2012-12-02 14:40:57,055::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29236::WARNING::2012-12-02 14:40:57,056::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29238::WARNING::2012-12-02 14:40:57,056::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29240::WARNING::2012-12-02 14:40:57,058::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29241::WARNING::2012-12-02 14:40:57,058::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29243::WARNING::2012-12-02 14:40:57,059::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29245::WARNING::2012-12-02 14:40:57,060::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11884::DEBUG::2012-12-02 14:40:57,095::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:40:57,095::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:40:57,097::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:40:57,099::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:40:57,100::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:40:57,100::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-11821::DEBUG::2012-12-02 14:40:59,723::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:40:59,723::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:40:59,723::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-29248::DEBUG::2012-12-02 14:40:59,891::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29248::DEBUG::2012-12-02 14:40:59,892::task::568::TaskManager.Task::(_updateState) Task=`813f07ce-5874-42ff-b30b-45da16f44b78`::moving from state init -> state preparing >Thread-29248::INFO::2012-12-02 14:40:59,892::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-11821::DEBUG::2012-12-02 14:40:59,930::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:40:59,931::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:40:59,931::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:40:59,967::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:40:59,967::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:40:59,976::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:40:59,979::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:40:59,979::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:40:59,979::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-29248::INFO::2012-12-02 14:40:59,980::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29248::DEBUG::2012-12-02 14:40:59,982::task::1151::TaskManager.Task::(prepare) Task=`813f07ce-5874-42ff-b30b-45da16f44b78`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29248::DEBUG::2012-12-02 14:40:59,982::task::568::TaskManager.Task::(_updateState) Task=`813f07ce-5874-42ff-b30b-45da16f44b78`::moving from state preparing -> state finished >Thread-29248::DEBUG::2012-12-02 14:40:59,984::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29248::DEBUG::2012-12-02 14:40:59,985::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29248::DEBUG::2012-12-02 14:40:59,986::task::957::TaskManager.Task::(_decref) Task=`813f07ce-5874-42ff-b30b-45da16f44b78`::ref 0 aborting False >Thread-29310::DEBUG::2012-12-02 14:40:59,989::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29310::DEBUG::2012-12-02 14:40:59,989::task::568::TaskManager.Task::(_updateState) Task=`93688830-6aeb-4b77-b134-90e38f01c862`::moving from state init -> state preparing >Thread-29310::INFO::2012-12-02 14:40:59,990::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29310::DEBUG::2012-12-02 14:40:59,990::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b4c48166-a8e5-4268-a7ac-6f759a6e5055`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29310::DEBUG::2012-12-02 14:40:59,990::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29310::DEBUG::2012-12-02 14:40:59,990::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29310::DEBUG::2012-12-02 14:40:59,990::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`b4c48166-a8e5-4268-a7ac-6f759a6e5055`::Granted request >Thread-29310::DEBUG::2012-12-02 14:40:59,991::task::794::TaskManager.Task::(resourceAcquired) Task=`93688830-6aeb-4b77-b134-90e38f01c862`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29310::DEBUG::2012-12-02 14:40:59,991::task::957::TaskManager.Task::(_decref) Task=`93688830-6aeb-4b77-b134-90e38f01c862`::ref 1 aborting False >Thread-29310::INFO::2012-12-02 14:40:59,998::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29310::DEBUG::2012-12-02 14:40:59,998::task::1151::TaskManager.Task::(prepare) Task=`93688830-6aeb-4b77-b134-90e38f01c862`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29310::DEBUG::2012-12-02 14:40:59,998::task::568::TaskManager.Task::(_updateState) Task=`93688830-6aeb-4b77-b134-90e38f01c862`::moving from state preparing -> state finished >Thread-29310::DEBUG::2012-12-02 14:40:59,998::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29310::DEBUG::2012-12-02 14:40:59,999::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29310::DEBUG::2012-12-02 14:40:59,999::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29310::DEBUG::2012-12-02 14:40:59,999::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29310::DEBUG::2012-12-02 14:40:59,999::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29310::DEBUG::2012-12-02 14:40:59,999::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29310::DEBUG::2012-12-02 14:40:59,999::task::957::TaskManager.Task::(_decref) Task=`93688830-6aeb-4b77-b134-90e38f01c862`::ref 0 aborting False >Thread-29312::DEBUG::2012-12-02 14:41:02,309::task::568::TaskManager.Task::(_updateState) Task=`df721592-ef30-4cb8-96e5-44a8c3ac2a9c`::moving from state init -> state preparing >Thread-29312::INFO::2012-12-02 14:41:02,309::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29312::INFO::2012-12-02 14:41:02,309::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00732398033142', 'lastCheck': 1354452059.998143, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.024425983429', 'lastCheck': 1354452056.0183899, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00129795074463', 'lastCheck': 1354452057.109587, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00895309448242', 'lastCheck': 1354452056.088037, 'code': 0, 'valid': True}} >Thread-29312::DEBUG::2012-12-02 14:41:02,309::task::1151::TaskManager.Task::(prepare) Task=`df721592-ef30-4cb8-96e5-44a8c3ac2a9c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00732398033142', 'lastCheck': 1354452059.998143, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.024425983429', 'lastCheck': 1354452056.0183899, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00129795074463', 'lastCheck': 1354452057.109587, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00895309448242', 'lastCheck': 1354452056.088037, 'code': 0, 'valid': True}} >Thread-29312::DEBUG::2012-12-02 14:41:02,309::task::568::TaskManager.Task::(_updateState) Task=`df721592-ef30-4cb8-96e5-44a8c3ac2a9c`::moving from state preparing -> state finished >Thread-29312::DEBUG::2012-12-02 14:41:02,310::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29312::DEBUG::2012-12-02 14:41:02,310::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29312::DEBUG::2012-12-02 14:41:02,310::task::957::TaskManager.Task::(_decref) Task=`df721592-ef30-4cb8-96e5-44a8c3ac2a9c`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:41:06,018::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:41:06,018::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:41:06,019::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:41:06,301::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:41:06,303::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:41:06,303::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:41:06,338::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:41:06,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:41:06,340::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:41:06,342::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:41:06,342::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:41:06,342::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:41:06,342::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:41:06,343::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:41:06,345::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:41:06,550::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:41:06,551::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:41:06,551::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:41:06,585::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:41:06,585::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:41:06,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:41:06,596::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:41:06,596::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:41:06,596::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-29439::DEBUG::2012-12-02 14:41:10,152::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29439::DEBUG::2012-12-02 14:41:10,152::task::568::TaskManager.Task::(_updateState) Task=`13c446a1-f8de-42e4-9814-8d572a61e92b`::moving from state init -> state preparing >Thread-29439::INFO::2012-12-02 14:41:10,152::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29439::INFO::2012-12-02 14:41:10,153::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29439::DEBUG::2012-12-02 14:41:10,153::task::1151::TaskManager.Task::(prepare) Task=`13c446a1-f8de-42e4-9814-8d572a61e92b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29439::DEBUG::2012-12-02 14:41:10,153::task::568::TaskManager.Task::(_updateState) Task=`13c446a1-f8de-42e4-9814-8d572a61e92b`::moving from state preparing -> state finished >Thread-29439::DEBUG::2012-12-02 14:41:10,153::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29439::DEBUG::2012-12-02 14:41:10,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29439::DEBUG::2012-12-02 14:41:10,153::task::957::TaskManager.Task::(_decref) Task=`13c446a1-f8de-42e4-9814-8d572a61e92b`::ref 0 aborting False >Thread-29440::DEBUG::2012-12-02 14:41:10,156::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29440::DEBUG::2012-12-02 14:41:10,156::task::568::TaskManager.Task::(_updateState) Task=`d9200bd9-0ddc-4c9a-91ba-ac67ed6015c2`::moving from state init -> state preparing >Thread-29440::INFO::2012-12-02 14:41:10,156::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29440::DEBUG::2012-12-02 14:41:10,157::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`12144a00-9b8a-4820-b8d8-1cae38ebb785`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29440::DEBUG::2012-12-02 14:41:10,157::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29440::DEBUG::2012-12-02 14:41:10,157::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29440::DEBUG::2012-12-02 14:41:10,157::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`12144a00-9b8a-4820-b8d8-1cae38ebb785`::Granted request >Thread-29440::DEBUG::2012-12-02 14:41:10,157::task::794::TaskManager.Task::(resourceAcquired) Task=`d9200bd9-0ddc-4c9a-91ba-ac67ed6015c2`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29440::DEBUG::2012-12-02 14:41:10,157::task::957::TaskManager.Task::(_decref) Task=`d9200bd9-0ddc-4c9a-91ba-ac67ed6015c2`::ref 1 aborting False >Thread-29440::INFO::2012-12-02 14:41:10,164::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29440::DEBUG::2012-12-02 14:41:10,164::task::1151::TaskManager.Task::(prepare) Task=`d9200bd9-0ddc-4c9a-91ba-ac67ed6015c2`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29440::DEBUG::2012-12-02 14:41:10,164::task::568::TaskManager.Task::(_updateState) Task=`d9200bd9-0ddc-4c9a-91ba-ac67ed6015c2`::moving from state preparing -> state finished >Thread-29440::DEBUG::2012-12-02 14:41:10,164::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29440::DEBUG::2012-12-02 14:41:10,164::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29440::DEBUG::2012-12-02 14:41:10,165::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29440::DEBUG::2012-12-02 14:41:10,165::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29440::DEBUG::2012-12-02 14:41:10,165::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29440::DEBUG::2012-12-02 14:41:10,165::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29440::DEBUG::2012-12-02 14:41:10,165::task::957::TaskManager.Task::(_decref) Task=`d9200bd9-0ddc-4c9a-91ba-ac67ed6015c2`::ref 0 aborting False >Thread-29442::DEBUG::2012-12-02 14:41:12,403::task::568::TaskManager.Task::(_updateState) Task=`b34057a7-50e4-46b2-8d4c-0827a81f7b5e`::moving from state init -> state preparing >Thread-29442::INFO::2012-12-02 14:41:12,403::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29442::INFO::2012-12-02 14:41:12,403::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0172710418701', 'lastCheck': 1354452070.0299771, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00225710868835', 'lastCheck': 1354452066.604908, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163698196411', 'lastCheck': 1354452067.1215911, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00586700439453', 'lastCheck': 1354452066.6128321, 'code': 0, 'valid': True}} >Thread-29442::DEBUG::2012-12-02 14:41:12,403::task::1151::TaskManager.Task::(prepare) Task=`b34057a7-50e4-46b2-8d4c-0827a81f7b5e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0172710418701', 'lastCheck': 1354452070.0299771, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00225710868835', 'lastCheck': 1354452066.604908, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163698196411', 'lastCheck': 1354452067.1215911, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00586700439453', 'lastCheck': 1354452066.6128321, 'code': 0, 'valid': True}} >Thread-29442::DEBUG::2012-12-02 14:41:12,403::task::568::TaskManager.Task::(_updateState) Task=`b34057a7-50e4-46b2-8d4c-0827a81f7b5e`::moving from state preparing -> state finished >Thread-29442::DEBUG::2012-12-02 14:41:12,404::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29442::DEBUG::2012-12-02 14:41:12,404::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29442::DEBUG::2012-12-02 14:41:12,404::task::957::TaskManager.Task::(_decref) Task=`b34057a7-50e4-46b2-8d4c-0827a81f7b5e`::ref 0 aborting False >Thread-29447::DEBUG::2012-12-02 14:41:20,321::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29447::DEBUG::2012-12-02 14:41:20,321::task::568::TaskManager.Task::(_updateState) Task=`e6aed104-cb94-49de-aead-b8ac6fc91578`::moving from state init -> state preparing >Thread-29447::INFO::2012-12-02 14:41:20,321::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29447::INFO::2012-12-02 14:41:20,322::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29447::DEBUG::2012-12-02 14:41:20,322::task::1151::TaskManager.Task::(prepare) Task=`e6aed104-cb94-49de-aead-b8ac6fc91578`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29447::DEBUG::2012-12-02 14:41:20,322::task::568::TaskManager.Task::(_updateState) Task=`e6aed104-cb94-49de-aead-b8ac6fc91578`::moving from state preparing -> state finished >Thread-29447::DEBUG::2012-12-02 14:41:20,322::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29447::DEBUG::2012-12-02 14:41:20,322::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29447::DEBUG::2012-12-02 14:41:20,322::task::957::TaskManager.Task::(_decref) Task=`e6aed104-cb94-49de-aead-b8ac6fc91578`::ref 0 aborting False >Thread-29448::DEBUG::2012-12-02 14:41:20,325::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29448::DEBUG::2012-12-02 14:41:20,325::task::568::TaskManager.Task::(_updateState) Task=`532b64c8-4a24-4e73-9c00-b73be1126422`::moving from state init -> state preparing >Thread-29448::INFO::2012-12-02 14:41:20,325::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29448::DEBUG::2012-12-02 14:41:20,326::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`682b3ab2-b0ad-4d37-b34c-ca54b98b8cb7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29448::DEBUG::2012-12-02 14:41:20,326::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29448::DEBUG::2012-12-02 14:41:20,326::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29448::DEBUG::2012-12-02 14:41:20,326::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`682b3ab2-b0ad-4d37-b34c-ca54b98b8cb7`::Granted request >Thread-29448::DEBUG::2012-12-02 14:41:20,326::task::794::TaskManager.Task::(resourceAcquired) Task=`532b64c8-4a24-4e73-9c00-b73be1126422`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29448::DEBUG::2012-12-02 14:41:20,327::task::957::TaskManager.Task::(_decref) Task=`532b64c8-4a24-4e73-9c00-b73be1126422`::ref 1 aborting False >Thread-29448::INFO::2012-12-02 14:41:20,333::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29448::DEBUG::2012-12-02 14:41:20,333::task::1151::TaskManager.Task::(prepare) Task=`532b64c8-4a24-4e73-9c00-b73be1126422`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29448::DEBUG::2012-12-02 14:41:20,333::task::568::TaskManager.Task::(_updateState) Task=`532b64c8-4a24-4e73-9c00-b73be1126422`::moving from state preparing -> state finished >Thread-29448::DEBUG::2012-12-02 14:41:20,333::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29448::DEBUG::2012-12-02 14:41:20,334::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29448::DEBUG::2012-12-02 14:41:20,334::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29448::DEBUG::2012-12-02 14:41:20,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29448::DEBUG::2012-12-02 14:41:20,334::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29448::DEBUG::2012-12-02 14:41:20,334::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29448::DEBUG::2012-12-02 14:41:20,334::task::957::TaskManager.Task::(_decref) Task=`532b64c8-4a24-4e73-9c00-b73be1126422`::ref 0 aborting False >Thread-29450::DEBUG::2012-12-02 14:41:22,500::task::568::TaskManager.Task::(_updateState) Task=`9cc871a2-eddc-4876-9dd2-6a71f92577e5`::moving from state init -> state preparing >Thread-29450::INFO::2012-12-02 14:41:22,500::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29450::INFO::2012-12-02 14:41:22,500::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00512790679932', 'lastCheck': 1354452080.045738, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0049250125885', 'lastCheck': 1354452076.621994, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00298094749451', 'lastCheck': 1354452077.134285, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072910785675', 'lastCheck': 1354452076.6301129, 'code': 0, 'valid': True}} >Thread-29450::DEBUG::2012-12-02 14:41:22,500::task::1151::TaskManager.Task::(prepare) Task=`9cc871a2-eddc-4876-9dd2-6a71f92577e5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00512790679932', 'lastCheck': 1354452080.045738, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0049250125885', 'lastCheck': 1354452076.621994, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00298094749451', 'lastCheck': 1354452077.134285, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0072910785675', 'lastCheck': 1354452076.6301129, 'code': 0, 'valid': True}} >Thread-29450::DEBUG::2012-12-02 14:41:22,500::task::568::TaskManager.Task::(_updateState) Task=`9cc871a2-eddc-4876-9dd2-6a71f92577e5`::moving from state preparing -> state finished >Thread-29450::DEBUG::2012-12-02 14:41:22,501::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29450::DEBUG::2012-12-02 14:41:22,501::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29450::DEBUG::2012-12-02 14:41:22,501::task::957::TaskManager.Task::(_decref) Task=`9cc871a2-eddc-4876-9dd2-6a71f92577e5`::ref 0 aborting False >Thread-29455::DEBUG::2012-12-02 14:41:30,533::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29455::DEBUG::2012-12-02 14:41:30,534::task::568::TaskManager.Task::(_updateState) Task=`696fa10a-4366-41c9-b686-78edf3c99473`::moving from state init -> state preparing >Thread-29455::INFO::2012-12-02 14:41:30,534::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29455::INFO::2012-12-02 14:41:30,534::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29455::DEBUG::2012-12-02 14:41:30,534::task::1151::TaskManager.Task::(prepare) Task=`696fa10a-4366-41c9-b686-78edf3c99473`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29455::DEBUG::2012-12-02 14:41:30,534::task::568::TaskManager.Task::(_updateState) Task=`696fa10a-4366-41c9-b686-78edf3c99473`::moving from state preparing -> state finished >Thread-29455::DEBUG::2012-12-02 14:41:30,534::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29455::DEBUG::2012-12-02 14:41:30,535::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29455::DEBUG::2012-12-02 14:41:30,535::task::957::TaskManager.Task::(_decref) Task=`696fa10a-4366-41c9-b686-78edf3c99473`::ref 0 aborting False >Thread-29456::DEBUG::2012-12-02 14:41:30,537::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29456::DEBUG::2012-12-02 14:41:30,537::task::568::TaskManager.Task::(_updateState) Task=`d104b483-e30f-4377-988f-c51991f057c1`::moving from state init -> state preparing >Thread-29456::INFO::2012-12-02 14:41:30,537::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29456::DEBUG::2012-12-02 14:41:30,538::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`81429b93-6ee9-473e-9ab5-e45735bd6e95`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29456::DEBUG::2012-12-02 14:41:30,538::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29456::DEBUG::2012-12-02 14:41:30,538::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29456::DEBUG::2012-12-02 14:41:30,538::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`81429b93-6ee9-473e-9ab5-e45735bd6e95`::Granted request >Thread-29456::DEBUG::2012-12-02 14:41:30,538::task::794::TaskManager.Task::(resourceAcquired) Task=`d104b483-e30f-4377-988f-c51991f057c1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29456::DEBUG::2012-12-02 14:41:30,538::task::957::TaskManager.Task::(_decref) Task=`d104b483-e30f-4377-988f-c51991f057c1`::ref 1 aborting False >Thread-29456::INFO::2012-12-02 14:41:30,542::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29456::DEBUG::2012-12-02 14:41:30,543::task::1151::TaskManager.Task::(prepare) Task=`d104b483-e30f-4377-988f-c51991f057c1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29456::DEBUG::2012-12-02 14:41:30,543::task::568::TaskManager.Task::(_updateState) Task=`d104b483-e30f-4377-988f-c51991f057c1`::moving from state preparing -> state finished >Thread-29456::DEBUG::2012-12-02 14:41:30,543::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29456::DEBUG::2012-12-02 14:41:30,543::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29456::DEBUG::2012-12-02 14:41:30,543::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29456::DEBUG::2012-12-02 14:41:30,543::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29456::DEBUG::2012-12-02 14:41:30,543::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29456::DEBUG::2012-12-02 14:41:30,543::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29456::DEBUG::2012-12-02 14:41:30,543::task::957::TaskManager.Task::(_decref) Task=`d104b483-e30f-4377-988f-c51991f057c1`::ref 0 aborting False >Thread-29458::DEBUG::2012-12-02 14:41:32,592::task::568::TaskManager.Task::(_updateState) Task=`e2991543-27a4-4234-abb2-8b3181d1dfb9`::moving from state init -> state preparing >Thread-29458::INFO::2012-12-02 14:41:32,593::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29458::INFO::2012-12-02 14:41:32,593::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00840401649475', 'lastCheck': 1354452090.0628879, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191807746887', 'lastCheck': 1354452086.638222, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00294804573059', 'lastCheck': 1354452087.145345, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00636315345764', 'lastCheck': 1354452086.6457341, 'code': 0, 'valid': True}} >Thread-29458::DEBUG::2012-12-02 14:41:32,593::task::1151::TaskManager.Task::(prepare) Task=`e2991543-27a4-4234-abb2-8b3181d1dfb9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00840401649475', 'lastCheck': 1354452090.0628879, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191807746887', 'lastCheck': 1354452086.638222, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00294804573059', 'lastCheck': 1354452087.145345, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00636315345764', 'lastCheck': 1354452086.6457341, 'code': 0, 'valid': True}} >Thread-29458::DEBUG::2012-12-02 14:41:32,593::task::568::TaskManager.Task::(_updateState) Task=`e2991543-27a4-4234-abb2-8b3181d1dfb9`::moving from state preparing -> state finished >Thread-29458::DEBUG::2012-12-02 14:41:32,593::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29458::DEBUG::2012-12-02 14:41:32,593::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29458::DEBUG::2012-12-02 14:41:32,594::task::957::TaskManager.Task::(_decref) Task=`e2991543-27a4-4234-abb2-8b3181d1dfb9`::ref 0 aborting False >Thread-29464::DEBUG::2012-12-02 14:41:40,686::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29464::DEBUG::2012-12-02 14:41:40,686::task::568::TaskManager.Task::(_updateState) Task=`f5976c93-5a59-4078-a7f6-e691924db326`::moving from state init -> state preparing >Thread-29464::INFO::2012-12-02 14:41:40,686::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29464::INFO::2012-12-02 14:41:40,687::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29464::DEBUG::2012-12-02 14:41:40,687::task::1151::TaskManager.Task::(prepare) Task=`f5976c93-5a59-4078-a7f6-e691924db326`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29464::DEBUG::2012-12-02 14:41:40,687::task::568::TaskManager.Task::(_updateState) Task=`f5976c93-5a59-4078-a7f6-e691924db326`::moving from state preparing -> state finished >Thread-29464::DEBUG::2012-12-02 14:41:40,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29464::DEBUG::2012-12-02 14:41:40,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29464::DEBUG::2012-12-02 14:41:40,687::task::957::TaskManager.Task::(_decref) Task=`f5976c93-5a59-4078-a7f6-e691924db326`::ref 0 aborting False >Thread-29465::DEBUG::2012-12-02 14:41:40,690::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29465::DEBUG::2012-12-02 14:41:40,690::task::568::TaskManager.Task::(_updateState) Task=`180b0c6b-398c-4dd2-a24a-0ac1f0a860b4`::moving from state init -> state preparing >Thread-29465::INFO::2012-12-02 14:41:40,690::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29465::DEBUG::2012-12-02 14:41:40,691::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`875ac6cc-929a-47a8-b762-6b3949d7fc83`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29465::DEBUG::2012-12-02 14:41:40,691::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29465::DEBUG::2012-12-02 14:41:40,691::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29465::DEBUG::2012-12-02 14:41:40,691::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`875ac6cc-929a-47a8-b762-6b3949d7fc83`::Granted request >Thread-29465::DEBUG::2012-12-02 14:41:40,691::task::794::TaskManager.Task::(resourceAcquired) Task=`180b0c6b-398c-4dd2-a24a-0ac1f0a860b4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29465::DEBUG::2012-12-02 14:41:40,691::task::957::TaskManager.Task::(_decref) Task=`180b0c6b-398c-4dd2-a24a-0ac1f0a860b4`::ref 1 aborting False >Thread-29465::INFO::2012-12-02 14:41:40,697::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29465::DEBUG::2012-12-02 14:41:40,697::task::1151::TaskManager.Task::(prepare) Task=`180b0c6b-398c-4dd2-a24a-0ac1f0a860b4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29465::DEBUG::2012-12-02 14:41:40,697::task::568::TaskManager.Task::(_updateState) Task=`180b0c6b-398c-4dd2-a24a-0ac1f0a860b4`::moving from state preparing -> state finished >Thread-29465::DEBUG::2012-12-02 14:41:40,697::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29465::DEBUG::2012-12-02 14:41:40,697::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29465::DEBUG::2012-12-02 14:41:40,697::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29465::DEBUG::2012-12-02 14:41:40,698::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29465::DEBUG::2012-12-02 14:41:40,698::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29465::DEBUG::2012-12-02 14:41:40,698::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29465::DEBUG::2012-12-02 14:41:40,698::task::957::TaskManager.Task::(_decref) Task=`180b0c6b-398c-4dd2-a24a-0ac1f0a860b4`::ref 0 aborting False >Thread-29466::DEBUG::2012-12-02 14:41:42,679::task::568::TaskManager.Task::(_updateState) Task=`4f113bd0-4d76-44b2-90ad-8d9aacd11eef`::moving from state init -> state preparing >Thread-29466::INFO::2012-12-02 14:41:42,679::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29466::INFO::2012-12-02 14:41:42,679::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00924205780029', 'lastCheck': 1354452100.08476, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00387692451477', 'lastCheck': 1354452096.6535389, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00349187850952', 'lastCheck': 1354452097.1607471, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00682187080383', 'lastCheck': 1354452096.6619511, 'code': 0, 'valid': True}} >Thread-29466::DEBUG::2012-12-02 14:41:42,679::task::1151::TaskManager.Task::(prepare) Task=`4f113bd0-4d76-44b2-90ad-8d9aacd11eef`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00924205780029', 'lastCheck': 1354452100.08476, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00387692451477', 'lastCheck': 1354452096.6535389, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00349187850952', 'lastCheck': 1354452097.1607471, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00682187080383', 'lastCheck': 1354452096.6619511, 'code': 0, 'valid': True}} >Thread-29466::DEBUG::2012-12-02 14:41:42,679::task::568::TaskManager.Task::(_updateState) Task=`4f113bd0-4d76-44b2-90ad-8d9aacd11eef`::moving from state preparing -> state finished >Thread-29466::DEBUG::2012-12-02 14:41:42,680::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29466::DEBUG::2012-12-02 14:41:42,680::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29466::DEBUG::2012-12-02 14:41:42,680::task::957::TaskManager.Task::(_decref) Task=`4f113bd0-4d76-44b2-90ad-8d9aacd11eef`::ref 0 aborting False >Thread-29472::DEBUG::2012-12-02 14:41:50,815::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29472::DEBUG::2012-12-02 14:41:50,815::task::568::TaskManager.Task::(_updateState) Task=`f330e7c1-9fc0-4868-9177-29afacf3591a`::moving from state init -> state preparing >Thread-29472::INFO::2012-12-02 14:41:50,815::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29472::INFO::2012-12-02 14:41:50,816::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29472::DEBUG::2012-12-02 14:41:50,816::task::1151::TaskManager.Task::(prepare) Task=`f330e7c1-9fc0-4868-9177-29afacf3591a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29472::DEBUG::2012-12-02 14:41:50,816::task::568::TaskManager.Task::(_updateState) Task=`f330e7c1-9fc0-4868-9177-29afacf3591a`::moving from state preparing -> state finished >Thread-29472::DEBUG::2012-12-02 14:41:50,816::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29472::DEBUG::2012-12-02 14:41:50,816::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29472::DEBUG::2012-12-02 14:41:50,816::task::957::TaskManager.Task::(_decref) Task=`f330e7c1-9fc0-4868-9177-29afacf3591a`::ref 0 aborting False >Thread-29473::DEBUG::2012-12-02 14:41:50,819::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29473::DEBUG::2012-12-02 14:41:50,819::task::568::TaskManager.Task::(_updateState) Task=`ac7f461f-7b1d-4933-9902-ee276010db2a`::moving from state init -> state preparing >Thread-29473::INFO::2012-12-02 14:41:50,819::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29473::DEBUG::2012-12-02 14:41:50,820::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4ddac59c-4062-48bc-bbd8-c38c9265bee1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29473::DEBUG::2012-12-02 14:41:50,820::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29473::DEBUG::2012-12-02 14:41:50,820::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29473::DEBUG::2012-12-02 14:41:50,820::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4ddac59c-4062-48bc-bbd8-c38c9265bee1`::Granted request >Thread-29473::DEBUG::2012-12-02 14:41:50,820::task::794::TaskManager.Task::(resourceAcquired) Task=`ac7f461f-7b1d-4933-9902-ee276010db2a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29473::DEBUG::2012-12-02 14:41:50,820::task::957::TaskManager.Task::(_decref) Task=`ac7f461f-7b1d-4933-9902-ee276010db2a`::ref 1 aborting False >Thread-29473::INFO::2012-12-02 14:41:50,826::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29473::DEBUG::2012-12-02 14:41:50,826::task::1151::TaskManager.Task::(prepare) Task=`ac7f461f-7b1d-4933-9902-ee276010db2a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29473::DEBUG::2012-12-02 14:41:50,827::task::568::TaskManager.Task::(_updateState) Task=`ac7f461f-7b1d-4933-9902-ee276010db2a`::moving from state preparing -> state finished >Thread-29473::DEBUG::2012-12-02 14:41:50,827::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29473::DEBUG::2012-12-02 14:41:50,827::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29473::DEBUG::2012-12-02 14:41:50,827::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29473::DEBUG::2012-12-02 14:41:50,827::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29473::DEBUG::2012-12-02 14:41:50,827::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29473::DEBUG::2012-12-02 14:41:50,827::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29473::DEBUG::2012-12-02 14:41:50,827::task::957::TaskManager.Task::(_decref) Task=`ac7f461f-7b1d-4933-9902-ee276010db2a`::ref 0 aborting False >Thread-29474::DEBUG::2012-12-02 14:41:52,777::task::568::TaskManager.Task::(_updateState) Task=`848ee262-fa08-4c0d-8fcb-0ae38dba865d`::moving from state init -> state preparing >Thread-29474::INFO::2012-12-02 14:41:52,777::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29474::INFO::2012-12-02 14:41:52,778::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794816017151', 'lastCheck': 1354452110.1042781, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00129103660583', 'lastCheck': 1354452106.705586, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171399116516', 'lastCheck': 1354452107.1717291, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00362110137939', 'lastCheck': 1354452106.7067621, 'code': 0, 'valid': True}} >Thread-29474::DEBUG::2012-12-02 14:41:52,778::task::1151::TaskManager.Task::(prepare) Task=`848ee262-fa08-4c0d-8fcb-0ae38dba865d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794816017151', 'lastCheck': 1354452110.1042781, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00129103660583', 'lastCheck': 1354452106.705586, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171399116516', 'lastCheck': 1354452107.1717291, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00362110137939', 'lastCheck': 1354452106.7067621, 'code': 0, 'valid': True}} >Thread-29474::DEBUG::2012-12-02 14:41:52,778::task::568::TaskManager.Task::(_updateState) Task=`848ee262-fa08-4c0d-8fcb-0ae38dba865d`::moving from state preparing -> state finished >Thread-29474::DEBUG::2012-12-02 14:41:52,778::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29474::DEBUG::2012-12-02 14:41:52,778::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29474::DEBUG::2012-12-02 14:41:52,778::task::957::TaskManager.Task::(_decref) Task=`848ee262-fa08-4c0d-8fcb-0ae38dba865d`::ref 0 aborting False >Thread-29480::DEBUG::2012-12-02 14:42:00,945::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29480::DEBUG::2012-12-02 14:42:00,945::task::568::TaskManager.Task::(_updateState) Task=`df43f3c7-05c1-4f01-8efe-e64ed9c2ab0c`::moving from state init -> state preparing >Thread-29480::INFO::2012-12-02 14:42:00,945::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29480::INFO::2012-12-02 14:42:00,945::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29480::DEBUG::2012-12-02 14:42:00,946::task::1151::TaskManager.Task::(prepare) Task=`df43f3c7-05c1-4f01-8efe-e64ed9c2ab0c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29480::DEBUG::2012-12-02 14:42:00,946::task::568::TaskManager.Task::(_updateState) Task=`df43f3c7-05c1-4f01-8efe-e64ed9c2ab0c`::moving from state preparing -> state finished >Thread-29480::DEBUG::2012-12-02 14:42:00,946::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29480::DEBUG::2012-12-02 14:42:00,946::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29480::DEBUG::2012-12-02 14:42:00,946::task::957::TaskManager.Task::(_decref) Task=`df43f3c7-05c1-4f01-8efe-e64ed9c2ab0c`::ref 0 aborting False >Thread-29481::DEBUG::2012-12-02 14:42:00,949::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29481::DEBUG::2012-12-02 14:42:00,949::task::568::TaskManager.Task::(_updateState) Task=`e42b6b9d-f3e9-40f4-877f-226a424212dc`::moving from state init -> state preparing >Thread-29481::INFO::2012-12-02 14:42:00,949::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29481::DEBUG::2012-12-02 14:42:00,949::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`56cb2beb-f404-451a-99a0-fd02b6e6a4b0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29481::DEBUG::2012-12-02 14:42:00,950::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29481::DEBUG::2012-12-02 14:42:00,950::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29481::DEBUG::2012-12-02 14:42:00,950::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`56cb2beb-f404-451a-99a0-fd02b6e6a4b0`::Granted request >Thread-29481::DEBUG::2012-12-02 14:42:00,950::task::794::TaskManager.Task::(resourceAcquired) Task=`e42b6b9d-f3e9-40f4-877f-226a424212dc`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29481::DEBUG::2012-12-02 14:42:00,950::task::957::TaskManager.Task::(_decref) Task=`e42b6b9d-f3e9-40f4-877f-226a424212dc`::ref 1 aborting False >Thread-29481::INFO::2012-12-02 14:42:00,956::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29481::DEBUG::2012-12-02 14:42:00,956::task::1151::TaskManager.Task::(prepare) Task=`e42b6b9d-f3e9-40f4-877f-226a424212dc`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29481::DEBUG::2012-12-02 14:42:00,956::task::568::TaskManager.Task::(_updateState) Task=`e42b6b9d-f3e9-40f4-877f-226a424212dc`::moving from state preparing -> state finished >Thread-29481::DEBUG::2012-12-02 14:42:00,956::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29481::DEBUG::2012-12-02 14:42:00,957::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29481::DEBUG::2012-12-02 14:42:00,957::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29481::DEBUG::2012-12-02 14:42:00,957::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29481::DEBUG::2012-12-02 14:42:00,957::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29481::DEBUG::2012-12-02 14:42:00,957::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29481::DEBUG::2012-12-02 14:42:00,957::task::957::TaskManager.Task::(_decref) Task=`e42b6b9d-f3e9-40f4-877f-226a424212dc`::ref 0 aborting False >Thread-29482::DEBUG::2012-12-02 14:42:02,865::task::568::TaskManager.Task::(_updateState) Task=`45c8546e-9048-4979-9a28-eea72025e1db`::moving from state init -> state preparing >Thread-29482::INFO::2012-12-02 14:42:02,865::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29482::INFO::2012-12-02 14:42:02,865::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0118601322174', 'lastCheck': 1354452120.124783, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00234699249268', 'lastCheck': 1354452116.7239261, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00351190567017', 'lastCheck': 1354452117.1839111, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00805687904358', 'lastCheck': 1354452116.729974, 'code': 0, 'valid': True}} >Thread-29482::DEBUG::2012-12-02 14:42:02,865::task::1151::TaskManager.Task::(prepare) Task=`45c8546e-9048-4979-9a28-eea72025e1db`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0118601322174', 'lastCheck': 1354452120.124783, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00234699249268', 'lastCheck': 1354452116.7239261, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00351190567017', 'lastCheck': 1354452117.1839111, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00805687904358', 'lastCheck': 1354452116.729974, 'code': 0, 'valid': True}} >Thread-29482::DEBUG::2012-12-02 14:42:02,865::task::568::TaskManager.Task::(_updateState) Task=`45c8546e-9048-4979-9a28-eea72025e1db`::moving from state preparing -> state finished >Thread-29482::DEBUG::2012-12-02 14:42:02,865::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29482::DEBUG::2012-12-02 14:42:02,865::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29482::DEBUG::2012-12-02 14:42:02,866::task::957::TaskManager.Task::(_decref) Task=`45c8546e-9048-4979-9a28-eea72025e1db`::ref 0 aborting False >Thread-29488::DEBUG::2012-12-02 14:42:11,116::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29488::DEBUG::2012-12-02 14:42:11,116::task::568::TaskManager.Task::(_updateState) Task=`51d86f40-e199-4f7b-ad73-0a4ddd6b3d71`::moving from state init -> state preparing >Thread-29488::INFO::2012-12-02 14:42:11,116::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29488::INFO::2012-12-02 14:42:11,116::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29488::DEBUG::2012-12-02 14:42:11,116::task::1151::TaskManager.Task::(prepare) Task=`51d86f40-e199-4f7b-ad73-0a4ddd6b3d71`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29488::DEBUG::2012-12-02 14:42:11,117::task::568::TaskManager.Task::(_updateState) Task=`51d86f40-e199-4f7b-ad73-0a4ddd6b3d71`::moving from state preparing -> state finished >Thread-29488::DEBUG::2012-12-02 14:42:11,117::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29488::DEBUG::2012-12-02 14:42:11,117::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29488::DEBUG::2012-12-02 14:42:11,117::task::957::TaskManager.Task::(_decref) Task=`51d86f40-e199-4f7b-ad73-0a4ddd6b3d71`::ref 0 aborting False >Thread-29489::DEBUG::2012-12-02 14:42:11,119::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29489::DEBUG::2012-12-02 14:42:11,120::task::568::TaskManager.Task::(_updateState) Task=`835541b1-d30c-4846-abd4-f342ee475e6d`::moving from state init -> state preparing >Thread-29489::INFO::2012-12-02 14:42:11,120::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29489::DEBUG::2012-12-02 14:42:11,120::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`acbb2b3c-0c3e-4188-80a6-9bab8e10a642`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29489::DEBUG::2012-12-02 14:42:11,120::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29489::DEBUG::2012-12-02 14:42:11,120::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29489::DEBUG::2012-12-02 14:42:11,121::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`acbb2b3c-0c3e-4188-80a6-9bab8e10a642`::Granted request >Thread-29489::DEBUG::2012-12-02 14:42:11,121::task::794::TaskManager.Task::(resourceAcquired) Task=`835541b1-d30c-4846-abd4-f342ee475e6d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29489::DEBUG::2012-12-02 14:42:11,121::task::957::TaskManager.Task::(_decref) Task=`835541b1-d30c-4846-abd4-f342ee475e6d`::ref 1 aborting False >Thread-29489::INFO::2012-12-02 14:42:11,127::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29489::DEBUG::2012-12-02 14:42:11,127::task::1151::TaskManager.Task::(prepare) Task=`835541b1-d30c-4846-abd4-f342ee475e6d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29489::DEBUG::2012-12-02 14:42:11,127::task::568::TaskManager.Task::(_updateState) Task=`835541b1-d30c-4846-abd4-f342ee475e6d`::moving from state preparing -> state finished >Thread-29489::DEBUG::2012-12-02 14:42:11,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29489::DEBUG::2012-12-02 14:42:11,127::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29489::DEBUG::2012-12-02 14:42:11,127::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29489::DEBUG::2012-12-02 14:42:11,128::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29489::DEBUG::2012-12-02 14:42:11,128::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29489::DEBUG::2012-12-02 14:42:11,128::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29489::DEBUG::2012-12-02 14:42:11,128::task::957::TaskManager.Task::(_decref) Task=`835541b1-d30c-4846-abd4-f342ee475e6d`::ref 0 aborting False >Thread-29490::DEBUG::2012-12-02 14:42:12,952::task::568::TaskManager.Task::(_updateState) Task=`aad6b2bd-10fb-482c-b5e4-d950ca7e9ddf`::moving from state init -> state preparing >Thread-29490::INFO::2012-12-02 14:42:12,952::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29490::INFO::2012-12-02 14:42:12,952::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00795483589172', 'lastCheck': 1354452130.1427071, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00458288192749', 'lastCheck': 1354452126.7404079, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00208592414856', 'lastCheck': 1354452127.194165, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00894498825073', 'lastCheck': 1354452126.7497399, 'code': 0, 'valid': True}} >Thread-29490::DEBUG::2012-12-02 14:42:12,952::task::1151::TaskManager.Task::(prepare) Task=`aad6b2bd-10fb-482c-b5e4-d950ca7e9ddf`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00795483589172', 'lastCheck': 1354452130.1427071, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00458288192749', 'lastCheck': 1354452126.7404079, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00208592414856', 'lastCheck': 1354452127.194165, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00894498825073', 'lastCheck': 1354452126.7497399, 'code': 0, 'valid': True}} >Thread-29490::DEBUG::2012-12-02 14:42:12,953::task::568::TaskManager.Task::(_updateState) Task=`aad6b2bd-10fb-482c-b5e4-d950ca7e9ddf`::moving from state preparing -> state finished >Thread-29490::DEBUG::2012-12-02 14:42:12,953::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29490::DEBUG::2012-12-02 14:42:12,953::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29490::DEBUG::2012-12-02 14:42:12,953::task::957::TaskManager.Task::(_decref) Task=`aad6b2bd-10fb-482c-b5e4-d950ca7e9ddf`::ref 0 aborting False >Thread-29496::DEBUG::2012-12-02 14:42:21,219::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29496::DEBUG::2012-12-02 14:42:21,219::task::568::TaskManager.Task::(_updateState) Task=`ac3bdeaf-2754-439f-879a-37a3b681ec46`::moving from state init -> state preparing >Thread-29496::INFO::2012-12-02 14:42:21,220::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29496::INFO::2012-12-02 14:42:21,220::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29496::DEBUG::2012-12-02 14:42:21,220::task::1151::TaskManager.Task::(prepare) Task=`ac3bdeaf-2754-439f-879a-37a3b681ec46`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29496::DEBUG::2012-12-02 14:42:21,220::task::568::TaskManager.Task::(_updateState) Task=`ac3bdeaf-2754-439f-879a-37a3b681ec46`::moving from state preparing -> state finished >Thread-29496::DEBUG::2012-12-02 14:42:21,220::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29496::DEBUG::2012-12-02 14:42:21,220::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29496::DEBUG::2012-12-02 14:42:21,220::task::957::TaskManager.Task::(_decref) Task=`ac3bdeaf-2754-439f-879a-37a3b681ec46`::ref 0 aborting False >Thread-29497::DEBUG::2012-12-02 14:42:21,223::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29497::DEBUG::2012-12-02 14:42:21,223::task::568::TaskManager.Task::(_updateState) Task=`5adcd010-c4c8-4c5a-bda2-48a1d0781068`::moving from state init -> state preparing >Thread-29497::INFO::2012-12-02 14:42:21,223::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29497::DEBUG::2012-12-02 14:42:21,224::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`42980df8-1f82-494d-9e24-43b0702f2de1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29497::DEBUG::2012-12-02 14:42:21,224::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29497::DEBUG::2012-12-02 14:42:21,224::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29497::DEBUG::2012-12-02 14:42:21,224::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`42980df8-1f82-494d-9e24-43b0702f2de1`::Granted request >Thread-29497::DEBUG::2012-12-02 14:42:21,225::task::794::TaskManager.Task::(resourceAcquired) Task=`5adcd010-c4c8-4c5a-bda2-48a1d0781068`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29497::DEBUG::2012-12-02 14:42:21,225::task::957::TaskManager.Task::(_decref) Task=`5adcd010-c4c8-4c5a-bda2-48a1d0781068`::ref 1 aborting False >Thread-29497::INFO::2012-12-02 14:42:21,229::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29497::DEBUG::2012-12-02 14:42:21,229::task::1151::TaskManager.Task::(prepare) Task=`5adcd010-c4c8-4c5a-bda2-48a1d0781068`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29497::DEBUG::2012-12-02 14:42:21,229::task::568::TaskManager.Task::(_updateState) Task=`5adcd010-c4c8-4c5a-bda2-48a1d0781068`::moving from state preparing -> state finished >Thread-29497::DEBUG::2012-12-02 14:42:21,229::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29497::DEBUG::2012-12-02 14:42:21,229::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29497::DEBUG::2012-12-02 14:42:21,230::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29497::DEBUG::2012-12-02 14:42:21,230::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29497::DEBUG::2012-12-02 14:42:21,230::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29497::DEBUG::2012-12-02 14:42:21,230::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29497::DEBUG::2012-12-02 14:42:21,230::task::957::TaskManager.Task::(_decref) Task=`5adcd010-c4c8-4c5a-bda2-48a1d0781068`::ref 0 aborting False >Thread-29498::DEBUG::2012-12-02 14:42:23,016::task::568::TaskManager.Task::(_updateState) Task=`7e3b6a65-3a53-4fad-bd39-4ad707a49088`::moving from state init -> state preparing >Thread-29498::INFO::2012-12-02 14:42:23,016::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29498::INFO::2012-12-02 14:42:23,017::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00918292999268', 'lastCheck': 1354452140.160562, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00458717346191', 'lastCheck': 1354452136.757889, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00301599502563', 'lastCheck': 1354452137.2062409, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00981712341309', 'lastCheck': 1354452136.7697091, 'code': 0, 'valid': True}} >Thread-29498::DEBUG::2012-12-02 14:42:23,017::task::1151::TaskManager.Task::(prepare) Task=`7e3b6a65-3a53-4fad-bd39-4ad707a49088`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00918292999268', 'lastCheck': 1354452140.160562, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00458717346191', 'lastCheck': 1354452136.757889, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00301599502563', 'lastCheck': 1354452137.2062409, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00981712341309', 'lastCheck': 1354452136.7697091, 'code': 0, 'valid': True}} >Thread-29498::DEBUG::2012-12-02 14:42:23,017::task::568::TaskManager.Task::(_updateState) Task=`7e3b6a65-3a53-4fad-bd39-4ad707a49088`::moving from state preparing -> state finished >Thread-29498::DEBUG::2012-12-02 14:42:23,017::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29498::DEBUG::2012-12-02 14:42:23,017::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29498::DEBUG::2012-12-02 14:42:23,017::task::957::TaskManager.Task::(_decref) Task=`7e3b6a65-3a53-4fad-bd39-4ad707a49088`::ref 0 aborting False >Thread-29504::DEBUG::2012-12-02 14:42:31,324::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29504::DEBUG::2012-12-02 14:42:31,325::task::568::TaskManager.Task::(_updateState) Task=`2263aade-6797-4aa8-82ec-43575fba0d01`::moving from state init -> state preparing >Thread-29504::INFO::2012-12-02 14:42:31,325::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29504::INFO::2012-12-02 14:42:31,325::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29504::DEBUG::2012-12-02 14:42:31,325::task::1151::TaskManager.Task::(prepare) Task=`2263aade-6797-4aa8-82ec-43575fba0d01`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29504::DEBUG::2012-12-02 14:42:31,326::task::568::TaskManager.Task::(_updateState) Task=`2263aade-6797-4aa8-82ec-43575fba0d01`::moving from state preparing -> state finished >Thread-29504::DEBUG::2012-12-02 14:42:31,326::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29504::DEBUG::2012-12-02 14:42:31,326::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29504::DEBUG::2012-12-02 14:42:31,326::task::957::TaskManager.Task::(_decref) Task=`2263aade-6797-4aa8-82ec-43575fba0d01`::ref 0 aborting False >Thread-29505::DEBUG::2012-12-02 14:42:31,329::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29505::DEBUG::2012-12-02 14:42:31,330::task::568::TaskManager.Task::(_updateState) Task=`8dc25166-cee2-496c-8fbc-fad9b4681df9`::moving from state init -> state preparing >Thread-29505::INFO::2012-12-02 14:42:31,330::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29505::DEBUG::2012-12-02 14:42:31,330::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`fdebe356-3713-48f6-9a65-7665ae8faaaf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29505::DEBUG::2012-12-02 14:42:31,330::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29505::DEBUG::2012-12-02 14:42:31,331::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29505::DEBUG::2012-12-02 14:42:31,331::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`fdebe356-3713-48f6-9a65-7665ae8faaaf`::Granted request >Thread-29505::DEBUG::2012-12-02 14:42:31,331::task::794::TaskManager.Task::(resourceAcquired) Task=`8dc25166-cee2-496c-8fbc-fad9b4681df9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29505::DEBUG::2012-12-02 14:42:31,331::task::957::TaskManager.Task::(_decref) Task=`8dc25166-cee2-496c-8fbc-fad9b4681df9`::ref 1 aborting False >Thread-29505::INFO::2012-12-02 14:42:31,337::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29505::DEBUG::2012-12-02 14:42:31,337::task::1151::TaskManager.Task::(prepare) Task=`8dc25166-cee2-496c-8fbc-fad9b4681df9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29505::DEBUG::2012-12-02 14:42:31,337::task::568::TaskManager.Task::(_updateState) Task=`8dc25166-cee2-496c-8fbc-fad9b4681df9`::moving from state preparing -> state finished >Thread-29505::DEBUG::2012-12-02 14:42:31,337::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29505::DEBUG::2012-12-02 14:42:31,337::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29505::DEBUG::2012-12-02 14:42:31,337::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29505::DEBUG::2012-12-02 14:42:31,337::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29505::DEBUG::2012-12-02 14:42:31,337::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29505::DEBUG::2012-12-02 14:42:31,337::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29505::DEBUG::2012-12-02 14:42:31,338::task::957::TaskManager.Task::(_decref) Task=`8dc25166-cee2-496c-8fbc-fad9b4681df9`::ref 0 aborting False >Thread-29506::DEBUG::2012-12-02 14:42:33,152::task::568::TaskManager.Task::(_updateState) Task=`f17b0a98-ea86-479c-91fc-acda71cc6461`::moving from state init -> state preparing >Thread-29506::INFO::2012-12-02 14:42:33,152::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29506::INFO::2012-12-02 14:42:33,152::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00836300849915', 'lastCheck': 1354452150.1793389, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00182390213013', 'lastCheck': 1354452146.7762711, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00206208229065', 'lastCheck': 1354452147.217345, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00700283050537', 'lastCheck': 1354452146.786303, 'code': 0, 'valid': True}} >Thread-29506::DEBUG::2012-12-02 14:42:33,152::task::1151::TaskManager.Task::(prepare) Task=`f17b0a98-ea86-479c-91fc-acda71cc6461`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00836300849915', 'lastCheck': 1354452150.1793389, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00182390213013', 'lastCheck': 1354452146.7762711, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00206208229065', 'lastCheck': 1354452147.217345, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00700283050537', 'lastCheck': 1354452146.786303, 'code': 0, 'valid': True}} >Thread-29506::DEBUG::2012-12-02 14:42:33,153::task::568::TaskManager.Task::(_updateState) Task=`f17b0a98-ea86-479c-91fc-acda71cc6461`::moving from state preparing -> state finished >Thread-29506::DEBUG::2012-12-02 14:42:33,153::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29506::DEBUG::2012-12-02 14:42:33,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29506::DEBUG::2012-12-02 14:42:33,153::task::957::TaskManager.Task::(_decref) Task=`f17b0a98-ea86-479c-91fc-acda71cc6461`::ref 0 aborting False >Thread-29512::DEBUG::2012-12-02 14:42:41,503::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29512::DEBUG::2012-12-02 14:42:41,504::task::568::TaskManager.Task::(_updateState) Task=`719269b0-052c-493a-90c8-bb4f584c6add`::moving from state init -> state preparing >Thread-29512::INFO::2012-12-02 14:42:41,504::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29512::INFO::2012-12-02 14:42:41,504::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29512::DEBUG::2012-12-02 14:42:41,504::task::1151::TaskManager.Task::(prepare) Task=`719269b0-052c-493a-90c8-bb4f584c6add`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29512::DEBUG::2012-12-02 14:42:41,504::task::568::TaskManager.Task::(_updateState) Task=`719269b0-052c-493a-90c8-bb4f584c6add`::moving from state preparing -> state finished >Thread-29512::DEBUG::2012-12-02 14:42:41,504::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29512::DEBUG::2012-12-02 14:42:41,504::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29512::DEBUG::2012-12-02 14:42:41,505::task::957::TaskManager.Task::(_decref) Task=`719269b0-052c-493a-90c8-bb4f584c6add`::ref 0 aborting False >Thread-29513::DEBUG::2012-12-02 14:42:41,508::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29513::DEBUG::2012-12-02 14:42:41,509::task::568::TaskManager.Task::(_updateState) Task=`b77eea2e-3b2f-4b70-b1d2-005d143ba72d`::moving from state init -> state preparing >Thread-29513::INFO::2012-12-02 14:42:41,509::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29513::DEBUG::2012-12-02 14:42:41,509::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5fbcb5e5-dc85-4789-a859-1161e079eb4d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29513::DEBUG::2012-12-02 14:42:41,509::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29513::DEBUG::2012-12-02 14:42:41,509::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29513::DEBUG::2012-12-02 14:42:41,509::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5fbcb5e5-dc85-4789-a859-1161e079eb4d`::Granted request >Thread-29513::DEBUG::2012-12-02 14:42:41,510::task::794::TaskManager.Task::(resourceAcquired) Task=`b77eea2e-3b2f-4b70-b1d2-005d143ba72d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29513::DEBUG::2012-12-02 14:42:41,510::task::957::TaskManager.Task::(_decref) Task=`b77eea2e-3b2f-4b70-b1d2-005d143ba72d`::ref 1 aborting False >Thread-29513::INFO::2012-12-02 14:42:41,516::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29513::DEBUG::2012-12-02 14:42:41,516::task::1151::TaskManager.Task::(prepare) Task=`b77eea2e-3b2f-4b70-b1d2-005d143ba72d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29513::DEBUG::2012-12-02 14:42:41,516::task::568::TaskManager.Task::(_updateState) Task=`b77eea2e-3b2f-4b70-b1d2-005d143ba72d`::moving from state preparing -> state finished >Thread-29513::DEBUG::2012-12-02 14:42:41,516::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29513::DEBUG::2012-12-02 14:42:41,516::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29513::DEBUG::2012-12-02 14:42:41,516::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29513::DEBUG::2012-12-02 14:42:41,517::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29513::DEBUG::2012-12-02 14:42:41,517::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29513::DEBUG::2012-12-02 14:42:41,517::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29513::DEBUG::2012-12-02 14:42:41,517::task::957::TaskManager.Task::(_decref) Task=`b77eea2e-3b2f-4b70-b1d2-005d143ba72d`::ref 0 aborting False >Thread-29514::DEBUG::2012-12-02 14:42:43,291::task::568::TaskManager.Task::(_updateState) Task=`01741730-2bf4-46f0-bb31-b1ad36417d94`::moving from state init -> state preparing >Thread-29514::INFO::2012-12-02 14:42:43,291::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29514::INFO::2012-12-02 14:42:43,291::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00892901420593', 'lastCheck': 1354452160.2023649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00504803657532', 'lastCheck': 1354452156.7943821, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00312209129333', 'lastCheck': 1354452157.230428, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0071918964386', 'lastCheck': 1354452156.8037341, 'code': 0, 'valid': True}} >Thread-29514::DEBUG::2012-12-02 14:42:43,291::task::1151::TaskManager.Task::(prepare) Task=`01741730-2bf4-46f0-bb31-b1ad36417d94`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00892901420593', 'lastCheck': 1354452160.2023649, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00504803657532', 'lastCheck': 1354452156.7943821, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00312209129333', 'lastCheck': 1354452157.230428, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0071918964386', 'lastCheck': 1354452156.8037341, 'code': 0, 'valid': True}} >Thread-29514::DEBUG::2012-12-02 14:42:43,291::task::568::TaskManager.Task::(_updateState) Task=`01741730-2bf4-46f0-bb31-b1ad36417d94`::moving from state preparing -> state finished >Thread-29514::DEBUG::2012-12-02 14:42:43,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29514::DEBUG::2012-12-02 14:42:43,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29514::DEBUG::2012-12-02 14:42:43,292::task::957::TaskManager.Task::(_decref) Task=`01741730-2bf4-46f0-bb31-b1ad36417d94`::ref 0 aborting False >Thread-29520::DEBUG::2012-12-02 14:42:51,690::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29520::DEBUG::2012-12-02 14:42:51,691::task::568::TaskManager.Task::(_updateState) Task=`c8e8ce65-ffcb-44f6-b9b3-56a3a7b23cac`::moving from state init -> state preparing >Thread-29520::INFO::2012-12-02 14:42:51,691::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29520::INFO::2012-12-02 14:42:51,691::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29520::DEBUG::2012-12-02 14:42:51,691::task::1151::TaskManager.Task::(prepare) Task=`c8e8ce65-ffcb-44f6-b9b3-56a3a7b23cac`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29520::DEBUG::2012-12-02 14:42:51,691::task::568::TaskManager.Task::(_updateState) Task=`c8e8ce65-ffcb-44f6-b9b3-56a3a7b23cac`::moving from state preparing -> state finished >Thread-29520::DEBUG::2012-12-02 14:42:51,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29520::DEBUG::2012-12-02 14:42:51,692::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29520::DEBUG::2012-12-02 14:42:51,692::task::957::TaskManager.Task::(_decref) Task=`c8e8ce65-ffcb-44f6-b9b3-56a3a7b23cac`::ref 0 aborting False >Thread-29521::DEBUG::2012-12-02 14:42:51,695::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29521::DEBUG::2012-12-02 14:42:51,696::task::568::TaskManager.Task::(_updateState) Task=`d3b1d922-fe11-42e6-97b4-c2212a955aa4`::moving from state init -> state preparing >Thread-29521::INFO::2012-12-02 14:42:51,696::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29521::DEBUG::2012-12-02 14:42:51,696::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`32161742-7d9d-4a67-8c6f-322b2bd2d5e0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29521::DEBUG::2012-12-02 14:42:51,696::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29521::DEBUG::2012-12-02 14:42:51,697::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29521::DEBUG::2012-12-02 14:42:51,697::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`32161742-7d9d-4a67-8c6f-322b2bd2d5e0`::Granted request >Thread-29521::DEBUG::2012-12-02 14:42:51,697::task::794::TaskManager.Task::(resourceAcquired) Task=`d3b1d922-fe11-42e6-97b4-c2212a955aa4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29521::DEBUG::2012-12-02 14:42:51,697::task::957::TaskManager.Task::(_decref) Task=`d3b1d922-fe11-42e6-97b4-c2212a955aa4`::ref 1 aborting False >Thread-29521::INFO::2012-12-02 14:42:51,703::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29521::DEBUG::2012-12-02 14:42:51,704::task::1151::TaskManager.Task::(prepare) Task=`d3b1d922-fe11-42e6-97b4-c2212a955aa4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29521::DEBUG::2012-12-02 14:42:51,704::task::568::TaskManager.Task::(_updateState) Task=`d3b1d922-fe11-42e6-97b4-c2212a955aa4`::moving from state preparing -> state finished >Thread-29521::DEBUG::2012-12-02 14:42:51,704::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29521::DEBUG::2012-12-02 14:42:51,704::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29521::DEBUG::2012-12-02 14:42:51,704::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29521::DEBUG::2012-12-02 14:42:51,704::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29521::DEBUG::2012-12-02 14:42:51,705::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29521::DEBUG::2012-12-02 14:42:51,705::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29521::DEBUG::2012-12-02 14:42:51,705::task::957::TaskManager.Task::(_decref) Task=`d3b1d922-fe11-42e6-97b4-c2212a955aa4`::ref 0 aborting False >Thread-29522::DEBUG::2012-12-02 14:42:53,412::task::568::TaskManager.Task::(_updateState) Task=`7b2f9176-6b0b-44ee-b27a-bc4360878745`::moving from state init -> state preparing >Thread-29522::INFO::2012-12-02 14:42:53,413::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29522::INFO::2012-12-02 14:42:53,413::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794911384583', 'lastCheck': 1354452170.2206621, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0026159286499', 'lastCheck': 1354452166.8132091, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00170493125916', 'lastCheck': 1354452167.2407629, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00799798965454', 'lastCheck': 1354452166.8245051, 'code': 0, 'valid': True}} >Thread-29522::DEBUG::2012-12-02 14:42:53,413::task::1151::TaskManager.Task::(prepare) Task=`7b2f9176-6b0b-44ee-b27a-bc4360878745`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794911384583', 'lastCheck': 1354452170.2206621, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0026159286499', 'lastCheck': 1354452166.8132091, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00170493125916', 'lastCheck': 1354452167.2407629, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00799798965454', 'lastCheck': 1354452166.8245051, 'code': 0, 'valid': True}} >Thread-29522::DEBUG::2012-12-02 14:42:53,413::task::568::TaskManager.Task::(_updateState) Task=`7b2f9176-6b0b-44ee-b27a-bc4360878745`::moving from state preparing -> state finished >Thread-29522::DEBUG::2012-12-02 14:42:53,413::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29522::DEBUG::2012-12-02 14:42:53,414::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29522::DEBUG::2012-12-02 14:42:53,414::task::957::TaskManager.Task::(_decref) Task=`7b2f9176-6b0b-44ee-b27a-bc4360878745`::ref 0 aborting False >Thread-29528::DEBUG::2012-12-02 14:43:01,869::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29528::DEBUG::2012-12-02 14:43:01,870::task::568::TaskManager.Task::(_updateState) Task=`39d32276-1241-43dd-97a7-b945d278aaf3`::moving from state init -> state preparing >Thread-29528::INFO::2012-12-02 14:43:01,870::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29528::INFO::2012-12-02 14:43:01,870::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29528::DEBUG::2012-12-02 14:43:01,870::task::1151::TaskManager.Task::(prepare) Task=`39d32276-1241-43dd-97a7-b945d278aaf3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29528::DEBUG::2012-12-02 14:43:01,870::task::568::TaskManager.Task::(_updateState) Task=`39d32276-1241-43dd-97a7-b945d278aaf3`::moving from state preparing -> state finished >Thread-29528::DEBUG::2012-12-02 14:43:01,870::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29528::DEBUG::2012-12-02 14:43:01,871::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29528::DEBUG::2012-12-02 14:43:01,871::task::957::TaskManager.Task::(_decref) Task=`39d32276-1241-43dd-97a7-b945d278aaf3`::ref 0 aborting False >Thread-29529::DEBUG::2012-12-02 14:43:01,874::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29529::DEBUG::2012-12-02 14:43:01,875::task::568::TaskManager.Task::(_updateState) Task=`be8e4031-1062-4a58-98a6-0e6f18f98bea`::moving from state init -> state preparing >Thread-29529::INFO::2012-12-02 14:43:01,875::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29529::DEBUG::2012-12-02 14:43:01,875::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1f0f4a48-04b2-4056-b8f3-f52ebecf6659`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29529::DEBUG::2012-12-02 14:43:01,875::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29529::DEBUG::2012-12-02 14:43:01,875::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29529::DEBUG::2012-12-02 14:43:01,876::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1f0f4a48-04b2-4056-b8f3-f52ebecf6659`::Granted request >Thread-29529::DEBUG::2012-12-02 14:43:01,876::task::794::TaskManager.Task::(resourceAcquired) Task=`be8e4031-1062-4a58-98a6-0e6f18f98bea`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29529::DEBUG::2012-12-02 14:43:01,876::task::957::TaskManager.Task::(_decref) Task=`be8e4031-1062-4a58-98a6-0e6f18f98bea`::ref 1 aborting False >Thread-29529::INFO::2012-12-02 14:43:01,880::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29529::DEBUG::2012-12-02 14:43:01,880::task::1151::TaskManager.Task::(prepare) Task=`be8e4031-1062-4a58-98a6-0e6f18f98bea`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29529::DEBUG::2012-12-02 14:43:01,881::task::568::TaskManager.Task::(_updateState) Task=`be8e4031-1062-4a58-98a6-0e6f18f98bea`::moving from state preparing -> state finished >Thread-29529::DEBUG::2012-12-02 14:43:01,881::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29529::DEBUG::2012-12-02 14:43:01,881::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29529::DEBUG::2012-12-02 14:43:01,881::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29529::DEBUG::2012-12-02 14:43:01,881::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29529::DEBUG::2012-12-02 14:43:01,881::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29529::DEBUG::2012-12-02 14:43:01,881::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29529::DEBUG::2012-12-02 14:43:01,881::task::957::TaskManager.Task::(_decref) Task=`be8e4031-1062-4a58-98a6-0e6f18f98bea`::ref 0 aborting False >Thread-29530::DEBUG::2012-12-02 14:43:03,512::task::568::TaskManager.Task::(_updateState) Task=`ce8fc3e9-de66-4e38-b953-cbe635d4a548`::moving from state init -> state preparing >Thread-29530::INFO::2012-12-02 14:43:03,513::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29530::INFO::2012-12-02 14:43:03,513::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0101230144501', 'lastCheck': 1354452180.2396071, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00177907943726', 'lastCheck': 1354452176.823729, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163197517395', 'lastCheck': 1354452177.2505269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00663995742798', 'lastCheck': 1354452176.840286, 'code': 0, 'valid': True}} >Thread-29530::DEBUG::2012-12-02 14:43:03,513::task::1151::TaskManager.Task::(prepare) Task=`ce8fc3e9-de66-4e38-b953-cbe635d4a548`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0101230144501', 'lastCheck': 1354452180.2396071, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00177907943726', 'lastCheck': 1354452176.823729, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163197517395', 'lastCheck': 1354452177.2505269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00663995742798', 'lastCheck': 1354452176.840286, 'code': 0, 'valid': True}} >Thread-29530::DEBUG::2012-12-02 14:43:03,513::task::568::TaskManager.Task::(_updateState) Task=`ce8fc3e9-de66-4e38-b953-cbe635d4a548`::moving from state preparing -> state finished >Thread-29530::DEBUG::2012-12-02 14:43:03,513::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29530::DEBUG::2012-12-02 14:43:03,513::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29530::DEBUG::2012-12-02 14:43:03,514::task::957::TaskManager.Task::(_decref) Task=`ce8fc3e9-de66-4e38-b953-cbe635d4a548`::ref 0 aborting False >Thread-29536::DEBUG::2012-12-02 14:43:12,048::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29536::DEBUG::2012-12-02 14:43:12,048::task::568::TaskManager.Task::(_updateState) Task=`101c2f24-c121-4164-97b2-3008387d6b0a`::moving from state init -> state preparing >Thread-29536::INFO::2012-12-02 14:43:12,048::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29536::INFO::2012-12-02 14:43:12,049::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29536::DEBUG::2012-12-02 14:43:12,049::task::1151::TaskManager.Task::(prepare) Task=`101c2f24-c121-4164-97b2-3008387d6b0a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29536::DEBUG::2012-12-02 14:43:12,049::task::568::TaskManager.Task::(_updateState) Task=`101c2f24-c121-4164-97b2-3008387d6b0a`::moving from state preparing -> state finished >Thread-29536::DEBUG::2012-12-02 14:43:12,049::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29536::DEBUG::2012-12-02 14:43:12,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29536::DEBUG::2012-12-02 14:43:12,049::task::957::TaskManager.Task::(_decref) Task=`101c2f24-c121-4164-97b2-3008387d6b0a`::ref 0 aborting False >Thread-29537::DEBUG::2012-12-02 14:43:12,052::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29537::DEBUG::2012-12-02 14:43:12,052::task::568::TaskManager.Task::(_updateState) Task=`ff185964-3734-4147-b8e2-8a8cb5162418`::moving from state init -> state preparing >Thread-29537::INFO::2012-12-02 14:43:12,052::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29537::DEBUG::2012-12-02 14:43:12,053::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`19fb8098-b56d-49de-b9cd-a132c3aaa3a7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29537::DEBUG::2012-12-02 14:43:12,053::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29537::DEBUG::2012-12-02 14:43:12,053::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29537::DEBUG::2012-12-02 14:43:12,053::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`19fb8098-b56d-49de-b9cd-a132c3aaa3a7`::Granted request >Thread-29537::DEBUG::2012-12-02 14:43:12,053::task::794::TaskManager.Task::(resourceAcquired) Task=`ff185964-3734-4147-b8e2-8a8cb5162418`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29537::DEBUG::2012-12-02 14:43:12,053::task::957::TaskManager.Task::(_decref) Task=`ff185964-3734-4147-b8e2-8a8cb5162418`::ref 1 aborting False >Thread-29537::INFO::2012-12-02 14:43:12,058::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29537::DEBUG::2012-12-02 14:43:12,058::task::1151::TaskManager.Task::(prepare) Task=`ff185964-3734-4147-b8e2-8a8cb5162418`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29537::DEBUG::2012-12-02 14:43:12,058::task::568::TaskManager.Task::(_updateState) Task=`ff185964-3734-4147-b8e2-8a8cb5162418`::moving from state preparing -> state finished >Thread-29537::DEBUG::2012-12-02 14:43:12,059::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29537::DEBUG::2012-12-02 14:43:12,059::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29537::DEBUG::2012-12-02 14:43:12,059::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29537::DEBUG::2012-12-02 14:43:12,059::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29537::DEBUG::2012-12-02 14:43:12,059::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29537::DEBUG::2012-12-02 14:43:12,059::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29537::DEBUG::2012-12-02 14:43:12,059::task::957::TaskManager.Task::(_decref) Task=`ff185964-3734-4147-b8e2-8a8cb5162418`::ref 0 aborting False >Thread-29538::DEBUG::2012-12-02 14:43:13,665::task::568::TaskManager.Task::(_updateState) Task=`bbeb48c7-7af0-4292-9dda-13bdd3141605`::moving from state init -> state preparing >Thread-29538::INFO::2012-12-02 14:43:13,665::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29538::INFO::2012-12-02 14:43:13,666::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833487510681', 'lastCheck': 1354452190.2575259, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00169706344604', 'lastCheck': 1354452186.8335669, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00224113464355', 'lastCheck': 1354452187.2615819, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00953698158264', 'lastCheck': 1354452186.858494, 'code': 0, 'valid': True}} >Thread-29538::DEBUG::2012-12-02 14:43:13,666::task::1151::TaskManager.Task::(prepare) Task=`bbeb48c7-7af0-4292-9dda-13bdd3141605`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00833487510681', 'lastCheck': 1354452190.2575259, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00169706344604', 'lastCheck': 1354452186.8335669, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00224113464355', 'lastCheck': 1354452187.2615819, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00953698158264', 'lastCheck': 1354452186.858494, 'code': 0, 'valid': True}} >Thread-29538::DEBUG::2012-12-02 14:43:13,666::task::568::TaskManager.Task::(_updateState) Task=`bbeb48c7-7af0-4292-9dda-13bdd3141605`::moving from state preparing -> state finished >Thread-29538::DEBUG::2012-12-02 14:43:13,666::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29538::DEBUG::2012-12-02 14:43:13,666::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29538::DEBUG::2012-12-02 14:43:13,666::task::957::TaskManager.Task::(_decref) Task=`bbeb48c7-7af0-4292-9dda-13bdd3141605`::ref 0 aborting False >Thread-29544::DEBUG::2012-12-02 14:43:22,212::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29544::DEBUG::2012-12-02 14:43:22,212::task::568::TaskManager.Task::(_updateState) Task=`b412363f-a188-4ea7-908a-92b5be4ac32a`::moving from state init -> state preparing >Thread-29544::INFO::2012-12-02 14:43:22,212::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29544::INFO::2012-12-02 14:43:22,212::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29544::DEBUG::2012-12-02 14:43:22,213::task::1151::TaskManager.Task::(prepare) Task=`b412363f-a188-4ea7-908a-92b5be4ac32a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29544::DEBUG::2012-12-02 14:43:22,213::task::568::TaskManager.Task::(_updateState) Task=`b412363f-a188-4ea7-908a-92b5be4ac32a`::moving from state preparing -> state finished >Thread-29544::DEBUG::2012-12-02 14:43:22,213::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29544::DEBUG::2012-12-02 14:43:22,213::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29544::DEBUG::2012-12-02 14:43:22,213::task::957::TaskManager.Task::(_decref) Task=`b412363f-a188-4ea7-908a-92b5be4ac32a`::ref 0 aborting False >Thread-29545::DEBUG::2012-12-02 14:43:22,217::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29545::DEBUG::2012-12-02 14:43:22,217::task::568::TaskManager.Task::(_updateState) Task=`27c490bb-bc29-47cc-9168-338be628a48b`::moving from state init -> state preparing >Thread-29545::INFO::2012-12-02 14:43:22,217::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29545::DEBUG::2012-12-02 14:43:22,217::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`01bb53fe-9194-46c0-9dea-f637ab58acad`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29545::DEBUG::2012-12-02 14:43:22,218::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29545::DEBUG::2012-12-02 14:43:22,218::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29545::DEBUG::2012-12-02 14:43:22,218::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`01bb53fe-9194-46c0-9dea-f637ab58acad`::Granted request >Thread-29545::DEBUG::2012-12-02 14:43:22,218::task::794::TaskManager.Task::(resourceAcquired) Task=`27c490bb-bc29-47cc-9168-338be628a48b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29545::DEBUG::2012-12-02 14:43:22,218::task::957::TaskManager.Task::(_decref) Task=`27c490bb-bc29-47cc-9168-338be628a48b`::ref 1 aborting False >Thread-29545::INFO::2012-12-02 14:43:22,225::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29545::DEBUG::2012-12-02 14:43:22,225::task::1151::TaskManager.Task::(prepare) Task=`27c490bb-bc29-47cc-9168-338be628a48b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29545::DEBUG::2012-12-02 14:43:22,225::task::568::TaskManager.Task::(_updateState) Task=`27c490bb-bc29-47cc-9168-338be628a48b`::moving from state preparing -> state finished >Thread-29545::DEBUG::2012-12-02 14:43:22,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29545::DEBUG::2012-12-02 14:43:22,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29545::DEBUG::2012-12-02 14:43:22,225::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29545::DEBUG::2012-12-02 14:43:22,225::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29545::DEBUG::2012-12-02 14:43:22,226::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29545::DEBUG::2012-12-02 14:43:22,226::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29545::DEBUG::2012-12-02 14:43:22,226::task::957::TaskManager.Task::(_decref) Task=`27c490bb-bc29-47cc-9168-338be628a48b`::ref 0 aborting False >Thread-29546::DEBUG::2012-12-02 14:43:23,821::task::568::TaskManager.Task::(_updateState) Task=`4f8eef04-a4e3-4037-985b-74e4bdba613c`::moving from state init -> state preparing >Thread-29546::INFO::2012-12-02 14:43:23,821::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29546::INFO::2012-12-02 14:43:23,821::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00776290893555', 'lastCheck': 1354452200.2765801, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00305700302124', 'lastCheck': 1354452196.8457839, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00341486930847', 'lastCheck': 1354452197.274364, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00933909416199', 'lastCheck': 1354452196.8774979, 'code': 0, 'valid': True}} >Thread-29546::DEBUG::2012-12-02 14:43:23,822::task::1151::TaskManager.Task::(prepare) Task=`4f8eef04-a4e3-4037-985b-74e4bdba613c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00776290893555', 'lastCheck': 1354452200.2765801, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00305700302124', 'lastCheck': 1354452196.8457839, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00341486930847', 'lastCheck': 1354452197.274364, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00933909416199', 'lastCheck': 1354452196.8774979, 'code': 0, 'valid': True}} >Thread-29546::DEBUG::2012-12-02 14:43:23,822::task::568::TaskManager.Task::(_updateState) Task=`4f8eef04-a4e3-4037-985b-74e4bdba613c`::moving from state preparing -> state finished >Thread-29546::DEBUG::2012-12-02 14:43:23,822::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29546::DEBUG::2012-12-02 14:43:23,822::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29546::DEBUG::2012-12-02 14:43:23,822::task::957::TaskManager.Task::(_decref) Task=`4f8eef04-a4e3-4037-985b-74e4bdba613c`::ref 0 aborting False >Thread-29552::DEBUG::2012-12-02 14:43:32,400::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29552::DEBUG::2012-12-02 14:43:32,400::task::568::TaskManager.Task::(_updateState) Task=`2b12ccc1-f2cb-4abb-94fa-6dba3ed1feb0`::moving from state init -> state preparing >Thread-29552::INFO::2012-12-02 14:43:32,401::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29552::INFO::2012-12-02 14:43:32,401::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29552::DEBUG::2012-12-02 14:43:32,401::task::1151::TaskManager.Task::(prepare) Task=`2b12ccc1-f2cb-4abb-94fa-6dba3ed1feb0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29552::DEBUG::2012-12-02 14:43:32,401::task::568::TaskManager.Task::(_updateState) Task=`2b12ccc1-f2cb-4abb-94fa-6dba3ed1feb0`::moving from state preparing -> state finished >Thread-29552::DEBUG::2012-12-02 14:43:32,401::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29552::DEBUG::2012-12-02 14:43:32,401::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29552::DEBUG::2012-12-02 14:43:32,401::task::957::TaskManager.Task::(_decref) Task=`2b12ccc1-f2cb-4abb-94fa-6dba3ed1feb0`::ref 0 aborting False >Thread-29553::DEBUG::2012-12-02 14:43:32,405::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29553::DEBUG::2012-12-02 14:43:32,406::task::568::TaskManager.Task::(_updateState) Task=`e00d3d45-b9dc-4c92-b1e5-4505b3c55258`::moving from state init -> state preparing >Thread-29553::INFO::2012-12-02 14:43:32,406::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29553::DEBUG::2012-12-02 14:43:32,406::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e6541c71-630d-4b30-b43f-a2e5abeda612`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29553::DEBUG::2012-12-02 14:43:32,406::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29553::DEBUG::2012-12-02 14:43:32,406::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29553::DEBUG::2012-12-02 14:43:32,407::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e6541c71-630d-4b30-b43f-a2e5abeda612`::Granted request >Thread-29553::DEBUG::2012-12-02 14:43:32,407::task::794::TaskManager.Task::(resourceAcquired) Task=`e00d3d45-b9dc-4c92-b1e5-4505b3c55258`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29553::DEBUG::2012-12-02 14:43:32,407::task::957::TaskManager.Task::(_decref) Task=`e00d3d45-b9dc-4c92-b1e5-4505b3c55258`::ref 1 aborting False >Thread-29553::INFO::2012-12-02 14:43:32,413::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29553::DEBUG::2012-12-02 14:43:32,414::task::1151::TaskManager.Task::(prepare) Task=`e00d3d45-b9dc-4c92-b1e5-4505b3c55258`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29553::DEBUG::2012-12-02 14:43:32,414::task::568::TaskManager.Task::(_updateState) Task=`e00d3d45-b9dc-4c92-b1e5-4505b3c55258`::moving from state preparing -> state finished >Thread-29553::DEBUG::2012-12-02 14:43:32,414::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29553::DEBUG::2012-12-02 14:43:32,414::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29553::DEBUG::2012-12-02 14:43:32,414::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29553::DEBUG::2012-12-02 14:43:32,414::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29553::DEBUG::2012-12-02 14:43:32,414::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29553::DEBUG::2012-12-02 14:43:32,415::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29553::DEBUG::2012-12-02 14:43:32,415::task::957::TaskManager.Task::(_decref) Task=`e00d3d45-b9dc-4c92-b1e5-4505b3c55258`::ref 0 aborting False >Thread-29554::DEBUG::2012-12-02 14:43:34,033::task::568::TaskManager.Task::(_updateState) Task=`2399bc76-9532-45d5-83d4-3e0c2454f1d3`::moving from state init -> state preparing >Thread-29554::INFO::2012-12-02 14:43:34,033::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29554::INFO::2012-12-02 14:43:34,033::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00788497924805', 'lastCheck': 1354452210.2946119, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191211700439', 'lastCheck': 1354452206.859901, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00216603279114', 'lastCheck': 1354452207.285341, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828194618225', 'lastCheck': 1354452206.8945811, 'code': 0, 'valid': True}} >Thread-29554::DEBUG::2012-12-02 14:43:34,033::task::1151::TaskManager.Task::(prepare) Task=`2399bc76-9532-45d5-83d4-3e0c2454f1d3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00788497924805', 'lastCheck': 1354452210.2946119, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191211700439', 'lastCheck': 1354452206.859901, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00216603279114', 'lastCheck': 1354452207.285341, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828194618225', 'lastCheck': 1354452206.8945811, 'code': 0, 'valid': True}} >Thread-29554::DEBUG::2012-12-02 14:43:34,033::task::568::TaskManager.Task::(_updateState) Task=`2399bc76-9532-45d5-83d4-3e0c2454f1d3`::moving from state preparing -> state finished >Thread-29554::DEBUG::2012-12-02 14:43:34,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29554::DEBUG::2012-12-02 14:43:34,034::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29554::DEBUG::2012-12-02 14:43:34,034::task::957::TaskManager.Task::(_decref) Task=`2399bc76-9532-45d5-83d4-3e0c2454f1d3`::ref 0 aborting False >Thread-29560::DEBUG::2012-12-02 14:43:42,547::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29560::DEBUG::2012-12-02 14:43:42,547::task::568::TaskManager.Task::(_updateState) Task=`2b07420c-8797-486f-afbb-78f0a170aece`::moving from state init -> state preparing >Thread-29560::INFO::2012-12-02 14:43:42,548::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29560::INFO::2012-12-02 14:43:42,548::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29560::DEBUG::2012-12-02 14:43:42,548::task::1151::TaskManager.Task::(prepare) Task=`2b07420c-8797-486f-afbb-78f0a170aece`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29560::DEBUG::2012-12-02 14:43:42,548::task::568::TaskManager.Task::(_updateState) Task=`2b07420c-8797-486f-afbb-78f0a170aece`::moving from state preparing -> state finished >Thread-29560::DEBUG::2012-12-02 14:43:42,548::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29560::DEBUG::2012-12-02 14:43:42,548::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29560::DEBUG::2012-12-02 14:43:42,548::task::957::TaskManager.Task::(_decref) Task=`2b07420c-8797-486f-afbb-78f0a170aece`::ref 0 aborting False >Thread-29561::DEBUG::2012-12-02 14:43:42,552::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29561::DEBUG::2012-12-02 14:43:42,552::task::568::TaskManager.Task::(_updateState) Task=`3a070470-b3af-41b3-8339-dacb1cf8b5cd`::moving from state init -> state preparing >Thread-29561::INFO::2012-12-02 14:43:42,552::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29561::DEBUG::2012-12-02 14:43:42,553::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e96ac867-46ae-4f07-81d2-85b6c43ac6b9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29561::DEBUG::2012-12-02 14:43:42,553::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29561::DEBUG::2012-12-02 14:43:42,553::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29561::DEBUG::2012-12-02 14:43:42,553::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e96ac867-46ae-4f07-81d2-85b6c43ac6b9`::Granted request >Thread-29561::DEBUG::2012-12-02 14:43:42,553::task::794::TaskManager.Task::(resourceAcquired) Task=`3a070470-b3af-41b3-8339-dacb1cf8b5cd`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29561::DEBUG::2012-12-02 14:43:42,554::task::957::TaskManager.Task::(_decref) Task=`3a070470-b3af-41b3-8339-dacb1cf8b5cd`::ref 1 aborting False >Thread-29561::INFO::2012-12-02 14:43:42,558::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29561::DEBUG::2012-12-02 14:43:42,559::task::1151::TaskManager.Task::(prepare) Task=`3a070470-b3af-41b3-8339-dacb1cf8b5cd`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29561::DEBUG::2012-12-02 14:43:42,559::task::568::TaskManager.Task::(_updateState) Task=`3a070470-b3af-41b3-8339-dacb1cf8b5cd`::moving from state preparing -> state finished >Thread-29561::DEBUG::2012-12-02 14:43:42,559::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29561::DEBUG::2012-12-02 14:43:42,559::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29561::DEBUG::2012-12-02 14:43:42,559::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29561::DEBUG::2012-12-02 14:43:42,559::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29561::DEBUG::2012-12-02 14:43:42,559::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29561::DEBUG::2012-12-02 14:43:42,559::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29561::DEBUG::2012-12-02 14:43:42,559::task::957::TaskManager.Task::(_decref) Task=`3a070470-b3af-41b3-8339-dacb1cf8b5cd`::ref 0 aborting False >Thread-29562::DEBUG::2012-12-02 14:43:44,239::task::568::TaskManager.Task::(_updateState) Task=`b34f6038-c78a-4945-b185-9006895c0259`::moving from state init -> state preparing >Thread-29562::INFO::2012-12-02 14:43:44,239::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29562::INFO::2012-12-02 14:43:44,239::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00762391090393', 'lastCheck': 1354452220.315295, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00285291671753', 'lastCheck': 1354452216.872498, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00446510314941', 'lastCheck': 1354452217.3000979, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00704312324524', 'lastCheck': 1354452216.910481, 'code': 0, 'valid': True}} >Thread-29562::DEBUG::2012-12-02 14:43:44,240::task::1151::TaskManager.Task::(prepare) Task=`b34f6038-c78a-4945-b185-9006895c0259`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00762391090393', 'lastCheck': 1354452220.315295, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00285291671753', 'lastCheck': 1354452216.872498, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00446510314941', 'lastCheck': 1354452217.3000979, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00704312324524', 'lastCheck': 1354452216.910481, 'code': 0, 'valid': True}} >Thread-29562::DEBUG::2012-12-02 14:43:44,240::task::568::TaskManager.Task::(_updateState) Task=`b34f6038-c78a-4945-b185-9006895c0259`::moving from state preparing -> state finished >Thread-29562::DEBUG::2012-12-02 14:43:44,240::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29562::DEBUG::2012-12-02 14:43:44,240::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29562::DEBUG::2012-12-02 14:43:44,240::task::957::TaskManager.Task::(_decref) Task=`b34f6038-c78a-4945-b185-9006895c0259`::ref 0 aborting False >Thread-29568::DEBUG::2012-12-02 14:43:52,718::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29568::DEBUG::2012-12-02 14:43:52,718::task::568::TaskManager.Task::(_updateState) Task=`c1c779a9-f24e-40ca-9a25-3ccd3c442990`::moving from state init -> state preparing >Thread-29568::INFO::2012-12-02 14:43:52,718::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29568::INFO::2012-12-02 14:43:52,718::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29568::DEBUG::2012-12-02 14:43:52,719::task::1151::TaskManager.Task::(prepare) Task=`c1c779a9-f24e-40ca-9a25-3ccd3c442990`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29568::DEBUG::2012-12-02 14:43:52,719::task::568::TaskManager.Task::(_updateState) Task=`c1c779a9-f24e-40ca-9a25-3ccd3c442990`::moving from state preparing -> state finished >Thread-29568::DEBUG::2012-12-02 14:43:52,719::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29568::DEBUG::2012-12-02 14:43:52,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29568::DEBUG::2012-12-02 14:43:52,719::task::957::TaskManager.Task::(_decref) Task=`c1c779a9-f24e-40ca-9a25-3ccd3c442990`::ref 0 aborting False >Thread-29569::DEBUG::2012-12-02 14:43:52,722::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29569::DEBUG::2012-12-02 14:43:52,722::task::568::TaskManager.Task::(_updateState) Task=`fbec636e-0e25-4888-9858-7304ade39385`::moving from state init -> state preparing >Thread-29569::INFO::2012-12-02 14:43:52,722::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29569::DEBUG::2012-12-02 14:43:52,722::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3ec9b872-a3b4-4c7c-af6b-0a8ac94e40e6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29569::DEBUG::2012-12-02 14:43:52,723::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29569::DEBUG::2012-12-02 14:43:52,723::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29569::DEBUG::2012-12-02 14:43:52,723::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3ec9b872-a3b4-4c7c-af6b-0a8ac94e40e6`::Granted request >Thread-29569::DEBUG::2012-12-02 14:43:52,724::task::794::TaskManager.Task::(resourceAcquired) Task=`fbec636e-0e25-4888-9858-7304ade39385`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29569::DEBUG::2012-12-02 14:43:52,724::task::957::TaskManager.Task::(_decref) Task=`fbec636e-0e25-4888-9858-7304ade39385`::ref 1 aborting False >Thread-29569::INFO::2012-12-02 14:43:52,729::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29569::DEBUG::2012-12-02 14:43:52,729::task::1151::TaskManager.Task::(prepare) Task=`fbec636e-0e25-4888-9858-7304ade39385`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29569::DEBUG::2012-12-02 14:43:52,729::task::568::TaskManager.Task::(_updateState) Task=`fbec636e-0e25-4888-9858-7304ade39385`::moving from state preparing -> state finished >Thread-29569::DEBUG::2012-12-02 14:43:52,729::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29569::DEBUG::2012-12-02 14:43:52,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29569::DEBUG::2012-12-02 14:43:52,729::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29569::DEBUG::2012-12-02 14:43:52,729::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29569::DEBUG::2012-12-02 14:43:52,729::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29569::DEBUG::2012-12-02 14:43:52,729::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29569::DEBUG::2012-12-02 14:43:52,730::task::957::TaskManager.Task::(_decref) Task=`fbec636e-0e25-4888-9858-7304ade39385`::ref 0 aborting False >Thread-29570::DEBUG::2012-12-02 14:43:54,326::task::568::TaskManager.Task::(_updateState) Task=`0076d10d-a6e4-4474-9d97-7b3b029c3396`::moving from state init -> state preparing >Thread-29570::INFO::2012-12-02 14:43:54,326::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29570::INFO::2012-12-02 14:43:54,326::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0108380317688', 'lastCheck': 1354452230.421953, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198101997375', 'lastCheck': 1354452226.885241, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00232696533203', 'lastCheck': 1354452227.3107779, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0079619884491', 'lastCheck': 1354452226.9284849, 'code': 0, 'valid': True}} >Thread-29570::DEBUG::2012-12-02 14:43:54,326::task::1151::TaskManager.Task::(prepare) Task=`0076d10d-a6e4-4474-9d97-7b3b029c3396`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0108380317688', 'lastCheck': 1354452230.421953, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198101997375', 'lastCheck': 1354452226.885241, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00232696533203', 'lastCheck': 1354452227.3107779, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0079619884491', 'lastCheck': 1354452226.9284849, 'code': 0, 'valid': True}} >Thread-29570::DEBUG::2012-12-02 14:43:54,327::task::568::TaskManager.Task::(_updateState) Task=`0076d10d-a6e4-4474-9d97-7b3b029c3396`::moving from state preparing -> state finished >Thread-29570::DEBUG::2012-12-02 14:43:54,327::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29570::DEBUG::2012-12-02 14:43:54,327::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29570::DEBUG::2012-12-02 14:43:54,327::task::957::TaskManager.Task::(_decref) Task=`0076d10d-a6e4-4474-9d97-7b3b029c3396`::ref 0 aborting False >Thread-29576::DEBUG::2012-12-02 14:44:02,871::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29576::DEBUG::2012-12-02 14:44:02,871::task::568::TaskManager.Task::(_updateState) Task=`c6840dfc-19d1-4f6e-976f-3d31629faaac`::moving from state init -> state preparing >Thread-29576::INFO::2012-12-02 14:44:02,872::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29576::INFO::2012-12-02 14:44:02,872::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29576::DEBUG::2012-12-02 14:44:02,872::task::1151::TaskManager.Task::(prepare) Task=`c6840dfc-19d1-4f6e-976f-3d31629faaac`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29576::DEBUG::2012-12-02 14:44:02,872::task::568::TaskManager.Task::(_updateState) Task=`c6840dfc-19d1-4f6e-976f-3d31629faaac`::moving from state preparing -> state finished >Thread-29576::DEBUG::2012-12-02 14:44:02,872::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29576::DEBUG::2012-12-02 14:44:02,872::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29576::DEBUG::2012-12-02 14:44:02,873::task::957::TaskManager.Task::(_decref) Task=`c6840dfc-19d1-4f6e-976f-3d31629faaac`::ref 0 aborting False >Thread-29577::DEBUG::2012-12-02 14:44:02,875::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29577::DEBUG::2012-12-02 14:44:02,875::task::568::TaskManager.Task::(_updateState) Task=`c3dcceac-94e4-4a01-8eb3-e5eb0e9aa278`::moving from state init -> state preparing >Thread-29577::INFO::2012-12-02 14:44:02,876::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29577::DEBUG::2012-12-02 14:44:02,876::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4ae0eaa7-ddad-4a0c-aa53-dc4604e37b66`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29577::DEBUG::2012-12-02 14:44:02,876::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29577::DEBUG::2012-12-02 14:44:02,876::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29577::DEBUG::2012-12-02 14:44:02,876::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4ae0eaa7-ddad-4a0c-aa53-dc4604e37b66`::Granted request >Thread-29577::DEBUG::2012-12-02 14:44:02,877::task::794::TaskManager.Task::(resourceAcquired) Task=`c3dcceac-94e4-4a01-8eb3-e5eb0e9aa278`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29577::DEBUG::2012-12-02 14:44:02,877::task::957::TaskManager.Task::(_decref) Task=`c3dcceac-94e4-4a01-8eb3-e5eb0e9aa278`::ref 1 aborting False >Thread-29577::INFO::2012-12-02 14:44:02,883::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29577::DEBUG::2012-12-02 14:44:02,883::task::1151::TaskManager.Task::(prepare) Task=`c3dcceac-94e4-4a01-8eb3-e5eb0e9aa278`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29577::DEBUG::2012-12-02 14:44:02,883::task::568::TaskManager.Task::(_updateState) Task=`c3dcceac-94e4-4a01-8eb3-e5eb0e9aa278`::moving from state preparing -> state finished >Thread-29577::DEBUG::2012-12-02 14:44:02,883::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29577::DEBUG::2012-12-02 14:44:02,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29577::DEBUG::2012-12-02 14:44:02,884::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29577::DEBUG::2012-12-02 14:44:02,884::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29577::DEBUG::2012-12-02 14:44:02,884::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29577::DEBUG::2012-12-02 14:44:02,884::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29577::DEBUG::2012-12-02 14:44:02,884::task::957::TaskManager.Task::(_decref) Task=`c3dcceac-94e4-4a01-8eb3-e5eb0e9aa278`::ref 0 aborting False >Thread-29578::DEBUG::2012-12-02 14:44:04,456::task::568::TaskManager.Task::(_updateState) Task=`680ca638-0fde-4498-80e3-7dda570378a6`::moving from state init -> state preparing >Thread-29578::INFO::2012-12-02 14:44:04,456::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29578::INFO::2012-12-02 14:44:04,456::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00901412963867', 'lastCheck': 1354452240.4425559, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00195693969727', 'lastCheck': 1354452236.8972509, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238704681396', 'lastCheck': 1354452237.3225851, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00630807876587', 'lastCheck': 1354452236.943311, 'code': 0, 'valid': True}} >Thread-29578::DEBUG::2012-12-02 14:44:04,457::task::1151::TaskManager.Task::(prepare) Task=`680ca638-0fde-4498-80e3-7dda570378a6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00901412963867', 'lastCheck': 1354452240.4425559, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00195693969727', 'lastCheck': 1354452236.8972509, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238704681396', 'lastCheck': 1354452237.3225851, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00630807876587', 'lastCheck': 1354452236.943311, 'code': 0, 'valid': True}} >Thread-29578::DEBUG::2012-12-02 14:44:04,457::task::568::TaskManager.Task::(_updateState) Task=`680ca638-0fde-4498-80e3-7dda570378a6`::moving from state preparing -> state finished >Thread-29578::DEBUG::2012-12-02 14:44:04,457::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29578::DEBUG::2012-12-02 14:44:04,457::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29578::DEBUG::2012-12-02 14:44:04,457::task::957::TaskManager.Task::(_decref) Task=`680ca638-0fde-4498-80e3-7dda570378a6`::ref 0 aborting False >Thread-29584::DEBUG::2012-12-02 14:44:13,010::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29584::DEBUG::2012-12-02 14:44:13,011::task::568::TaskManager.Task::(_updateState) Task=`8e2a9d46-c0a7-404e-ae20-4e3003456862`::moving from state init -> state preparing >Thread-29584::INFO::2012-12-02 14:44:13,011::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29584::INFO::2012-12-02 14:44:13,011::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29584::DEBUG::2012-12-02 14:44:13,011::task::1151::TaskManager.Task::(prepare) Task=`8e2a9d46-c0a7-404e-ae20-4e3003456862`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29584::DEBUG::2012-12-02 14:44:13,011::task::568::TaskManager.Task::(_updateState) Task=`8e2a9d46-c0a7-404e-ae20-4e3003456862`::moving from state preparing -> state finished >Thread-29584::DEBUG::2012-12-02 14:44:13,011::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29584::DEBUG::2012-12-02 14:44:13,011::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29584::DEBUG::2012-12-02 14:44:13,012::task::957::TaskManager.Task::(_decref) Task=`8e2a9d46-c0a7-404e-ae20-4e3003456862`::ref 0 aborting False >Thread-29585::DEBUG::2012-12-02 14:44:13,015::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29585::DEBUG::2012-12-02 14:44:13,015::task::568::TaskManager.Task::(_updateState) Task=`3e75b5f9-97bb-45aa-8d0a-743f6c71c30f`::moving from state init -> state preparing >Thread-29585::INFO::2012-12-02 14:44:13,015::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29585::DEBUG::2012-12-02 14:44:13,016::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a3c633fd-008d-4855-b270-23ffbd43e780`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29585::DEBUG::2012-12-02 14:44:13,016::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29585::DEBUG::2012-12-02 14:44:13,016::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29585::DEBUG::2012-12-02 14:44:13,016::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a3c633fd-008d-4855-b270-23ffbd43e780`::Granted request >Thread-29585::DEBUG::2012-12-02 14:44:13,016::task::794::TaskManager.Task::(resourceAcquired) Task=`3e75b5f9-97bb-45aa-8d0a-743f6c71c30f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29585::DEBUG::2012-12-02 14:44:13,017::task::957::TaskManager.Task::(_decref) Task=`3e75b5f9-97bb-45aa-8d0a-743f6c71c30f`::ref 1 aborting False >Thread-29585::INFO::2012-12-02 14:44:13,023::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29585::DEBUG::2012-12-02 14:44:13,023::task::1151::TaskManager.Task::(prepare) Task=`3e75b5f9-97bb-45aa-8d0a-743f6c71c30f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29585::DEBUG::2012-12-02 14:44:13,023::task::568::TaskManager.Task::(_updateState) Task=`3e75b5f9-97bb-45aa-8d0a-743f6c71c30f`::moving from state preparing -> state finished >Thread-29585::DEBUG::2012-12-02 14:44:13,023::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29585::DEBUG::2012-12-02 14:44:13,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29585::DEBUG::2012-12-02 14:44:13,024::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29585::DEBUG::2012-12-02 14:44:13,024::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29585::DEBUG::2012-12-02 14:44:13,024::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29585::DEBUG::2012-12-02 14:44:13,024::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29585::DEBUG::2012-12-02 14:44:13,024::task::957::TaskManager.Task::(_decref) Task=`3e75b5f9-97bb-45aa-8d0a-743f6c71c30f`::ref 0 aborting False >Thread-29586::DEBUG::2012-12-02 14:44:14,524::task::568::TaskManager.Task::(_updateState) Task=`122a8bb7-0fe2-4055-ad99-dd04ecdd9142`::moving from state init -> state preparing >Thread-29586::INFO::2012-12-02 14:44:14,524::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29586::INFO::2012-12-02 14:44:14,524::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0034031867981', 'lastCheck': 1354452250.454428, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00175404548645', 'lastCheck': 1354452246.9083829, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00195598602295', 'lastCheck': 1354452247.3325579, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00775408744812', 'lastCheck': 1354452246.959898, 'code': 0, 'valid': True}} >Thread-29586::DEBUG::2012-12-02 14:44:14,524::task::1151::TaskManager.Task::(prepare) Task=`122a8bb7-0fe2-4055-ad99-dd04ecdd9142`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0034031867981', 'lastCheck': 1354452250.454428, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00175404548645', 'lastCheck': 1354452246.9083829, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00195598602295', 'lastCheck': 1354452247.3325579, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00775408744812', 'lastCheck': 1354452246.959898, 'code': 0, 'valid': True}} >Thread-29586::DEBUG::2012-12-02 14:44:14,525::task::568::TaskManager.Task::(_updateState) Task=`122a8bb7-0fe2-4055-ad99-dd04ecdd9142`::moving from state preparing -> state finished >Thread-29586::DEBUG::2012-12-02 14:44:14,525::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29586::DEBUG::2012-12-02 14:44:14,525::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29586::DEBUG::2012-12-02 14:44:14,525::task::957::TaskManager.Task::(_decref) Task=`122a8bb7-0fe2-4055-ad99-dd04ecdd9142`::ref 0 aborting False >Thread-29592::DEBUG::2012-12-02 14:44:23,150::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29592::DEBUG::2012-12-02 14:44:23,150::task::568::TaskManager.Task::(_updateState) Task=`adccb873-5afd-4a29-b23e-0a8d2c8801d2`::moving from state init -> state preparing >Thread-29592::INFO::2012-12-02 14:44:23,150::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29592::INFO::2012-12-02 14:44:23,151::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29592::DEBUG::2012-12-02 14:44:23,151::task::1151::TaskManager.Task::(prepare) Task=`adccb873-5afd-4a29-b23e-0a8d2c8801d2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29592::DEBUG::2012-12-02 14:44:23,151::task::568::TaskManager.Task::(_updateState) Task=`adccb873-5afd-4a29-b23e-0a8d2c8801d2`::moving from state preparing -> state finished >Thread-29592::DEBUG::2012-12-02 14:44:23,151::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29592::DEBUG::2012-12-02 14:44:23,151::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29592::DEBUG::2012-12-02 14:44:23,151::task::957::TaskManager.Task::(_decref) Task=`adccb873-5afd-4a29-b23e-0a8d2c8801d2`::ref 0 aborting False >Thread-29593::DEBUG::2012-12-02 14:44:23,155::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29593::DEBUG::2012-12-02 14:44:23,155::task::568::TaskManager.Task::(_updateState) Task=`caaccad4-fe95-42c5-8a39-6f4b5ab09d9d`::moving from state init -> state preparing >Thread-29593::INFO::2012-12-02 14:44:23,155::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29593::DEBUG::2012-12-02 14:44:23,155::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`79c438f0-c319-4a5d-ad85-13c81b196f4f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29593::DEBUG::2012-12-02 14:44:23,156::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29593::DEBUG::2012-12-02 14:44:23,156::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29593::DEBUG::2012-12-02 14:44:23,156::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`79c438f0-c319-4a5d-ad85-13c81b196f4f`::Granted request >Thread-29593::DEBUG::2012-12-02 14:44:23,156::task::794::TaskManager.Task::(resourceAcquired) Task=`caaccad4-fe95-42c5-8a39-6f4b5ab09d9d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29593::DEBUG::2012-12-02 14:44:23,156::task::957::TaskManager.Task::(_decref) Task=`caaccad4-fe95-42c5-8a39-6f4b5ab09d9d`::ref 1 aborting False >Thread-29593::INFO::2012-12-02 14:44:23,163::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29593::DEBUG::2012-12-02 14:44:23,163::task::1151::TaskManager.Task::(prepare) Task=`caaccad4-fe95-42c5-8a39-6f4b5ab09d9d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29593::DEBUG::2012-12-02 14:44:23,163::task::568::TaskManager.Task::(_updateState) Task=`caaccad4-fe95-42c5-8a39-6f4b5ab09d9d`::moving from state preparing -> state finished >Thread-29593::DEBUG::2012-12-02 14:44:23,163::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29593::DEBUG::2012-12-02 14:44:23,163::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29593::DEBUG::2012-12-02 14:44:23,163::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29593::DEBUG::2012-12-02 14:44:23,164::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29593::DEBUG::2012-12-02 14:44:23,164::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29593::DEBUG::2012-12-02 14:44:23,164::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29593::DEBUG::2012-12-02 14:44:23,164::task::957::TaskManager.Task::(_decref) Task=`caaccad4-fe95-42c5-8a39-6f4b5ab09d9d`::ref 0 aborting False >Thread-29594::DEBUG::2012-12-02 14:44:24,750::task::568::TaskManager.Task::(_updateState) Task=`14f58d4d-b89a-4c84-b2cc-2bddfe685a0a`::moving from state init -> state preparing >Thread-29594::INFO::2012-12-02 14:44:24,751::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29594::INFO::2012-12-02 14:44:24,751::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0079619884491', 'lastCheck': 1354452260.471904, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00312089920044', 'lastCheck': 1354452256.9220819, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00200891494751', 'lastCheck': 1354452257.342253, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0069739818573', 'lastCheck': 1354452256.9758799, 'code': 0, 'valid': True}} >Thread-29594::DEBUG::2012-12-02 14:44:24,751::task::1151::TaskManager.Task::(prepare) Task=`14f58d4d-b89a-4c84-b2cc-2bddfe685a0a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0079619884491', 'lastCheck': 1354452260.471904, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00312089920044', 'lastCheck': 1354452256.9220819, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00200891494751', 'lastCheck': 1354452257.342253, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0069739818573', 'lastCheck': 1354452256.9758799, 'code': 0, 'valid': True}} >Thread-29594::DEBUG::2012-12-02 14:44:24,751::task::568::TaskManager.Task::(_updateState) Task=`14f58d4d-b89a-4c84-b2cc-2bddfe685a0a`::moving from state preparing -> state finished >Thread-29594::DEBUG::2012-12-02 14:44:24,751::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29594::DEBUG::2012-12-02 14:44:24,751::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29594::DEBUG::2012-12-02 14:44:24,752::task::957::TaskManager.Task::(_decref) Task=`14f58d4d-b89a-4c84-b2cc-2bddfe685a0a`::ref 0 aborting False >Thread-29600::DEBUG::2012-12-02 14:44:33,362::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29600::DEBUG::2012-12-02 14:44:33,362::task::568::TaskManager.Task::(_updateState) Task=`b3ededee-ec0a-4386-8677-674e1b850eb0`::moving from state init -> state preparing >Thread-29600::INFO::2012-12-02 14:44:33,362::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29600::INFO::2012-12-02 14:44:33,363::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29600::DEBUG::2012-12-02 14:44:33,363::task::1151::TaskManager.Task::(prepare) Task=`b3ededee-ec0a-4386-8677-674e1b850eb0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29600::DEBUG::2012-12-02 14:44:33,363::task::568::TaskManager.Task::(_updateState) Task=`b3ededee-ec0a-4386-8677-674e1b850eb0`::moving from state preparing -> state finished >Thread-29600::DEBUG::2012-12-02 14:44:33,363::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29600::DEBUG::2012-12-02 14:44:33,363::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29600::DEBUG::2012-12-02 14:44:33,363::task::957::TaskManager.Task::(_decref) Task=`b3ededee-ec0a-4386-8677-674e1b850eb0`::ref 0 aborting False >Thread-29601::DEBUG::2012-12-02 14:44:33,366::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29601::DEBUG::2012-12-02 14:44:33,367::task::568::TaskManager.Task::(_updateState) Task=`96f6856d-f882-4062-a6d1-b71ded66e46a`::moving from state init -> state preparing >Thread-29601::INFO::2012-12-02 14:44:33,367::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29601::DEBUG::2012-12-02 14:44:33,367::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cb0bd971-0d23-4b6f-addc-f1840856ff25`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29601::DEBUG::2012-12-02 14:44:33,367::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29601::DEBUG::2012-12-02 14:44:33,368::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29601::DEBUG::2012-12-02 14:44:33,368::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cb0bd971-0d23-4b6f-addc-f1840856ff25`::Granted request >Thread-29601::DEBUG::2012-12-02 14:44:33,368::task::794::TaskManager.Task::(resourceAcquired) Task=`96f6856d-f882-4062-a6d1-b71ded66e46a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29601::DEBUG::2012-12-02 14:44:33,368::task::957::TaskManager.Task::(_decref) Task=`96f6856d-f882-4062-a6d1-b71ded66e46a`::ref 1 aborting False >Thread-29601::INFO::2012-12-02 14:44:33,375::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29601::DEBUG::2012-12-02 14:44:33,375::task::1151::TaskManager.Task::(prepare) Task=`96f6856d-f882-4062-a6d1-b71ded66e46a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29601::DEBUG::2012-12-02 14:44:33,375::task::568::TaskManager.Task::(_updateState) Task=`96f6856d-f882-4062-a6d1-b71ded66e46a`::moving from state preparing -> state finished >Thread-29601::DEBUG::2012-12-02 14:44:33,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29601::DEBUG::2012-12-02 14:44:33,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29601::DEBUG::2012-12-02 14:44:33,375::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29601::DEBUG::2012-12-02 14:44:33,375::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29601::DEBUG::2012-12-02 14:44:33,375::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29601::DEBUG::2012-12-02 14:44:33,376::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29601::DEBUG::2012-12-02 14:44:33,376::task::957::TaskManager.Task::(_decref) Task=`96f6856d-f882-4062-a6d1-b71ded66e46a`::ref 0 aborting False >Thread-29602::DEBUG::2012-12-02 14:44:34,887::task::568::TaskManager.Task::(_updateState) Task=`4ed5e507-c4a9-4bc0-948f-35bfe6f3474c`::moving from state init -> state preparing >Thread-29602::INFO::2012-12-02 14:44:34,887::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29602::INFO::2012-12-02 14:44:34,887::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00813698768616', 'lastCheck': 1354452270.489547, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198793411255', 'lastCheck': 1354452266.9355071, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00237202644348', 'lastCheck': 1354452267.3525259, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00676202774048', 'lastCheck': 1354452266.9914119, 'code': 0, 'valid': True}} >Thread-29602::DEBUG::2012-12-02 14:44:34,888::task::1151::TaskManager.Task::(prepare) Task=`4ed5e507-c4a9-4bc0-948f-35bfe6f3474c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00813698768616', 'lastCheck': 1354452270.489547, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00198793411255', 'lastCheck': 1354452266.9355071, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00237202644348', 'lastCheck': 1354452267.3525259, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00676202774048', 'lastCheck': 1354452266.9914119, 'code': 0, 'valid': True}} >Thread-29602::DEBUG::2012-12-02 14:44:34,888::task::568::TaskManager.Task::(_updateState) Task=`4ed5e507-c4a9-4bc0-948f-35bfe6f3474c`::moving from state preparing -> state finished >Thread-29602::DEBUG::2012-12-02 14:44:34,888::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29602::DEBUG::2012-12-02 14:44:34,888::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29602::DEBUG::2012-12-02 14:44:34,888::task::957::TaskManager.Task::(_decref) Task=`4ed5e507-c4a9-4bc0-948f-35bfe6f3474c`::ref 0 aborting False >Thread-29608::DEBUG::2012-12-02 14:44:43,550::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29608::DEBUG::2012-12-02 14:44:43,550::task::568::TaskManager.Task::(_updateState) Task=`42bc1973-cbf4-4a31-bf38-c6546adb455c`::moving from state init -> state preparing >Thread-29608::INFO::2012-12-02 14:44:43,550::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29608::INFO::2012-12-02 14:44:43,551::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29608::DEBUG::2012-12-02 14:44:43,551::task::1151::TaskManager.Task::(prepare) Task=`42bc1973-cbf4-4a31-bf38-c6546adb455c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29608::DEBUG::2012-12-02 14:44:43,551::task::568::TaskManager.Task::(_updateState) Task=`42bc1973-cbf4-4a31-bf38-c6546adb455c`::moving from state preparing -> state finished >Thread-29608::DEBUG::2012-12-02 14:44:43,551::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29608::DEBUG::2012-12-02 14:44:43,551::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29608::DEBUG::2012-12-02 14:44:43,551::task::957::TaskManager.Task::(_decref) Task=`42bc1973-cbf4-4a31-bf38-c6546adb455c`::ref 0 aborting False >Thread-29609::DEBUG::2012-12-02 14:44:43,555::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29609::DEBUG::2012-12-02 14:44:43,555::task::568::TaskManager.Task::(_updateState) Task=`3255df96-9551-4ee2-a063-87dc1b56ab09`::moving from state init -> state preparing >Thread-29609::INFO::2012-12-02 14:44:43,555::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29609::DEBUG::2012-12-02 14:44:43,556::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7939abab-6d09-4122-ac3e-f6057e35e76f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29609::DEBUG::2012-12-02 14:44:43,556::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29609::DEBUG::2012-12-02 14:44:43,556::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29609::DEBUG::2012-12-02 14:44:43,556::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7939abab-6d09-4122-ac3e-f6057e35e76f`::Granted request >Thread-29609::DEBUG::2012-12-02 14:44:43,556::task::794::TaskManager.Task::(resourceAcquired) Task=`3255df96-9551-4ee2-a063-87dc1b56ab09`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29609::DEBUG::2012-12-02 14:44:43,556::task::957::TaskManager.Task::(_decref) Task=`3255df96-9551-4ee2-a063-87dc1b56ab09`::ref 1 aborting False >Thread-29609::INFO::2012-12-02 14:44:43,563::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29609::DEBUG::2012-12-02 14:44:43,563::task::1151::TaskManager.Task::(prepare) Task=`3255df96-9551-4ee2-a063-87dc1b56ab09`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29609::DEBUG::2012-12-02 14:44:43,563::task::568::TaskManager.Task::(_updateState) Task=`3255df96-9551-4ee2-a063-87dc1b56ab09`::moving from state preparing -> state finished >Thread-29609::DEBUG::2012-12-02 14:44:43,563::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29609::DEBUG::2012-12-02 14:44:43,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29609::DEBUG::2012-12-02 14:44:43,564::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29609::DEBUG::2012-12-02 14:44:43,564::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29609::DEBUG::2012-12-02 14:44:43,564::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29609::DEBUG::2012-12-02 14:44:43,564::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29609::DEBUG::2012-12-02 14:44:43,564::task::957::TaskManager.Task::(_decref) Task=`3255df96-9551-4ee2-a063-87dc1b56ab09`::ref 0 aborting False >Thread-29610::DEBUG::2012-12-02 14:44:45,061::task::568::TaskManager.Task::(_updateState) Task=`667a5077-e74c-4bdb-814b-cd0e56ba64b1`::moving from state init -> state preparing >Thread-29610::INFO::2012-12-02 14:44:45,061::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29610::INFO::2012-12-02 14:44:45,062::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00414514541626', 'lastCheck': 1354452280.5033851, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00259900093079', 'lastCheck': 1354452276.9475319, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00295805931091', 'lastCheck': 1354452277.363694, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0102849006653', 'lastCheck': 1354452277.0108349, 'code': 0, 'valid': True}} >Thread-29610::DEBUG::2012-12-02 14:44:45,062::task::1151::TaskManager.Task::(prepare) Task=`667a5077-e74c-4bdb-814b-cd0e56ba64b1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00414514541626', 'lastCheck': 1354452280.5033851, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00259900093079', 'lastCheck': 1354452276.9475319, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00295805931091', 'lastCheck': 1354452277.363694, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0102849006653', 'lastCheck': 1354452277.0108349, 'code': 0, 'valid': True}} >Thread-29610::DEBUG::2012-12-02 14:44:45,062::task::568::TaskManager.Task::(_updateState) Task=`667a5077-e74c-4bdb-814b-cd0e56ba64b1`::moving from state preparing -> state finished >Thread-29610::DEBUG::2012-12-02 14:44:45,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29610::DEBUG::2012-12-02 14:44:45,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29610::DEBUG::2012-12-02 14:44:45,062::task::957::TaskManager.Task::(_decref) Task=`667a5077-e74c-4bdb-814b-cd0e56ba64b1`::ref 0 aborting False >Thread-29616::DEBUG::2012-12-02 14:44:53,802::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29616::DEBUG::2012-12-02 14:44:53,803::task::568::TaskManager.Task::(_updateState) Task=`ff6693c7-98a4-4569-b31d-ffde36aa658e`::moving from state init -> state preparing >Thread-29616::INFO::2012-12-02 14:44:53,803::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29616::INFO::2012-12-02 14:44:53,803::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29616::DEBUG::2012-12-02 14:44:53,803::task::1151::TaskManager.Task::(prepare) Task=`ff6693c7-98a4-4569-b31d-ffde36aa658e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29616::DEBUG::2012-12-02 14:44:53,803::task::568::TaskManager.Task::(_updateState) Task=`ff6693c7-98a4-4569-b31d-ffde36aa658e`::moving from state preparing -> state finished >Thread-29616::DEBUG::2012-12-02 14:44:53,803::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29616::DEBUG::2012-12-02 14:44:53,803::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29616::DEBUG::2012-12-02 14:44:53,804::task::957::TaskManager.Task::(_decref) Task=`ff6693c7-98a4-4569-b31d-ffde36aa658e`::ref 0 aborting False >Thread-29617::DEBUG::2012-12-02 14:44:53,806::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29617::DEBUG::2012-12-02 14:44:53,806::task::568::TaskManager.Task::(_updateState) Task=`6e7a38cc-36b5-47c5-b218-6f0edda77456`::moving from state init -> state preparing >Thread-29617::INFO::2012-12-02 14:44:53,806::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29617::DEBUG::2012-12-02 14:44:53,807::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2bcd3ecc-5f34-4ce3-8aa4-4aef2dd26a47`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29617::DEBUG::2012-12-02 14:44:53,807::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29617::DEBUG::2012-12-02 14:44:53,807::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29617::DEBUG::2012-12-02 14:44:53,807::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2bcd3ecc-5f34-4ce3-8aa4-4aef2dd26a47`::Granted request >Thread-29617::DEBUG::2012-12-02 14:44:53,807::task::794::TaskManager.Task::(resourceAcquired) Task=`6e7a38cc-36b5-47c5-b218-6f0edda77456`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29617::DEBUG::2012-12-02 14:44:53,807::task::957::TaskManager.Task::(_decref) Task=`6e7a38cc-36b5-47c5-b218-6f0edda77456`::ref 1 aborting False >Thread-29617::INFO::2012-12-02 14:44:53,811::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29617::DEBUG::2012-12-02 14:44:53,812::task::1151::TaskManager.Task::(prepare) Task=`6e7a38cc-36b5-47c5-b218-6f0edda77456`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29617::DEBUG::2012-12-02 14:44:53,812::task::568::TaskManager.Task::(_updateState) Task=`6e7a38cc-36b5-47c5-b218-6f0edda77456`::moving from state preparing -> state finished >Thread-29617::DEBUG::2012-12-02 14:44:53,812::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29617::DEBUG::2012-12-02 14:44:53,812::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29617::DEBUG::2012-12-02 14:44:53,812::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29617::DEBUG::2012-12-02 14:44:53,812::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29617::DEBUG::2012-12-02 14:44:53,812::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29617::DEBUG::2012-12-02 14:44:53,812::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29617::DEBUG::2012-12-02 14:44:53,812::task::957::TaskManager.Task::(_decref) Task=`6e7a38cc-36b5-47c5-b218-6f0edda77456`::ref 0 aborting False >Thread-29618::DEBUG::2012-12-02 14:44:55,176::task::568::TaskManager.Task::(_updateState) Task=`8cd35ec6-d997-4caf-a47b-769bc8c8d196`::moving from state init -> state preparing >Thread-29618::INFO::2012-12-02 14:44:55,177::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29618::INFO::2012-12-02 14:44:55,177::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00892901420593', 'lastCheck': 1354452290.521785, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00177097320557', 'lastCheck': 1354452286.961632, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163817405701', 'lastCheck': 1354452287.372817, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0039529800415', 'lastCheck': 1354452287.0694129, 'code': 0, 'valid': True}} >Thread-29618::DEBUG::2012-12-02 14:44:55,177::task::1151::TaskManager.Task::(prepare) Task=`8cd35ec6-d997-4caf-a47b-769bc8c8d196`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00892901420593', 'lastCheck': 1354452290.521785, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00177097320557', 'lastCheck': 1354452286.961632, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00163817405701', 'lastCheck': 1354452287.372817, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0039529800415', 'lastCheck': 1354452287.0694129, 'code': 0, 'valid': True}} >Thread-29618::DEBUG::2012-12-02 14:44:55,177::task::568::TaskManager.Task::(_updateState) Task=`8cd35ec6-d997-4caf-a47b-769bc8c8d196`::moving from state preparing -> state finished >Thread-29618::DEBUG::2012-12-02 14:44:55,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29618::DEBUG::2012-12-02 14:44:55,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29618::DEBUG::2012-12-02 14:44:55,177::task::957::TaskManager.Task::(_decref) Task=`8cd35ec6-d997-4caf-a47b-769bc8c8d196`::ref 0 aborting False >Thread-29624::DEBUG::2012-12-02 14:45:03,940::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29624::DEBUG::2012-12-02 14:45:03,940::task::568::TaskManager.Task::(_updateState) Task=`af823bf7-5922-453a-b7af-c7e524c7bf02`::moving from state init -> state preparing >Thread-29624::INFO::2012-12-02 14:45:03,940::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29624::INFO::2012-12-02 14:45:03,941::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29624::DEBUG::2012-12-02 14:45:03,941::task::1151::TaskManager.Task::(prepare) Task=`af823bf7-5922-453a-b7af-c7e524c7bf02`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29624::DEBUG::2012-12-02 14:45:03,941::task::568::TaskManager.Task::(_updateState) Task=`af823bf7-5922-453a-b7af-c7e524c7bf02`::moving from state preparing -> state finished >Thread-29624::DEBUG::2012-12-02 14:45:03,941::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29624::DEBUG::2012-12-02 14:45:03,941::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29624::DEBUG::2012-12-02 14:45:03,941::task::957::TaskManager.Task::(_decref) Task=`af823bf7-5922-453a-b7af-c7e524c7bf02`::ref 0 aborting False >Thread-29625::DEBUG::2012-12-02 14:45:03,944::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29625::DEBUG::2012-12-02 14:45:03,944::task::568::TaskManager.Task::(_updateState) Task=`a7f90916-692f-4146-8348-9c1a70145f93`::moving from state init -> state preparing >Thread-29625::INFO::2012-12-02 14:45:03,944::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29625::DEBUG::2012-12-02 14:45:03,945::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9a7f9dec-a9f0-48c5-9817-a4e5b6634468`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29625::DEBUG::2012-12-02 14:45:03,945::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29625::DEBUG::2012-12-02 14:45:03,945::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29625::DEBUG::2012-12-02 14:45:03,945::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9a7f9dec-a9f0-48c5-9817-a4e5b6634468`::Granted request >Thread-29625::DEBUG::2012-12-02 14:45:03,945::task::794::TaskManager.Task::(resourceAcquired) Task=`a7f90916-692f-4146-8348-9c1a70145f93`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29625::DEBUG::2012-12-02 14:45:03,945::task::957::TaskManager.Task::(_decref) Task=`a7f90916-692f-4146-8348-9c1a70145f93`::ref 1 aborting False >Thread-29625::INFO::2012-12-02 14:45:03,949::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29625::DEBUG::2012-12-02 14:45:03,950::task::1151::TaskManager.Task::(prepare) Task=`a7f90916-692f-4146-8348-9c1a70145f93`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29625::DEBUG::2012-12-02 14:45:03,950::task::568::TaskManager.Task::(_updateState) Task=`a7f90916-692f-4146-8348-9c1a70145f93`::moving from state preparing -> state finished >Thread-29625::DEBUG::2012-12-02 14:45:03,950::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29625::DEBUG::2012-12-02 14:45:03,950::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29625::DEBUG::2012-12-02 14:45:03,950::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29625::DEBUG::2012-12-02 14:45:03,950::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29625::DEBUG::2012-12-02 14:45:03,950::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29625::DEBUG::2012-12-02 14:45:03,950::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29625::DEBUG::2012-12-02 14:45:03,950::task::957::TaskManager.Task::(_decref) Task=`a7f90916-692f-4146-8348-9c1a70145f93`::ref 0 aborting False >Thread-29626::DEBUG::2012-12-02 14:45:05,277::task::568::TaskManager.Task::(_updateState) Task=`e0b54516-1cf4-4580-a142-a74cbe709912`::moving from state init -> state preparing >Thread-29626::INFO::2012-12-02 14:45:05,277::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29626::INFO::2012-12-02 14:45:05,277::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00896811485291', 'lastCheck': 1354452300.5393829, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00178694725037', 'lastCheck': 1354452296.9731679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00132203102112', 'lastCheck': 1354452297.381253, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00741910934448', 'lastCheck': 1354452297.087935, 'code': 0, 'valid': True}} >Thread-29626::DEBUG::2012-12-02 14:45:05,277::task::1151::TaskManager.Task::(prepare) Task=`e0b54516-1cf4-4580-a142-a74cbe709912`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00896811485291', 'lastCheck': 1354452300.5393829, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00178694725037', 'lastCheck': 1354452296.9731679, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00132203102112', 'lastCheck': 1354452297.381253, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00741910934448', 'lastCheck': 1354452297.087935, 'code': 0, 'valid': True}} >Thread-29626::DEBUG::2012-12-02 14:45:05,277::task::568::TaskManager.Task::(_updateState) Task=`e0b54516-1cf4-4580-a142-a74cbe709912`::moving from state preparing -> state finished >Thread-29626::DEBUG::2012-12-02 14:45:05,278::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29626::DEBUG::2012-12-02 14:45:05,278::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29626::DEBUG::2012-12-02 14:45:05,278::task::957::TaskManager.Task::(_decref) Task=`e0b54516-1cf4-4580-a142-a74cbe709912`::ref 0 aborting False >Thread-29632::DEBUG::2012-12-02 14:45:14,119::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29632::DEBUG::2012-12-02 14:45:14,120::task::568::TaskManager.Task::(_updateState) Task=`2b4aa40e-910a-4078-ab3d-a5e5344eae89`::moving from state init -> state preparing >Thread-29632::INFO::2012-12-02 14:45:14,120::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29632::INFO::2012-12-02 14:45:14,120::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29632::DEBUG::2012-12-02 14:45:14,120::task::1151::TaskManager.Task::(prepare) Task=`2b4aa40e-910a-4078-ab3d-a5e5344eae89`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29632::DEBUG::2012-12-02 14:45:14,120::task::568::TaskManager.Task::(_updateState) Task=`2b4aa40e-910a-4078-ab3d-a5e5344eae89`::moving from state preparing -> state finished >Thread-29632::DEBUG::2012-12-02 14:45:14,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29632::DEBUG::2012-12-02 14:45:14,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29632::DEBUG::2012-12-02 14:45:14,121::task::957::TaskManager.Task::(_decref) Task=`2b4aa40e-910a-4078-ab3d-a5e5344eae89`::ref 0 aborting False >Thread-29633::DEBUG::2012-12-02 14:45:14,123::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29633::DEBUG::2012-12-02 14:45:14,124::task::568::TaskManager.Task::(_updateState) Task=`1202e876-b059-4cf4-a6b9-a1d944993b43`::moving from state init -> state preparing >Thread-29633::INFO::2012-12-02 14:45:14,124::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29633::DEBUG::2012-12-02 14:45:14,124::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e95583e7-dd2e-4970-93ed-73f418beb94a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29633::DEBUG::2012-12-02 14:45:14,124::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29633::DEBUG::2012-12-02 14:45:14,124::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29633::DEBUG::2012-12-02 14:45:14,125::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e95583e7-dd2e-4970-93ed-73f418beb94a`::Granted request >Thread-29633::DEBUG::2012-12-02 14:45:14,125::task::794::TaskManager.Task::(resourceAcquired) Task=`1202e876-b059-4cf4-a6b9-a1d944993b43`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29633::DEBUG::2012-12-02 14:45:14,125::task::957::TaskManager.Task::(_decref) Task=`1202e876-b059-4cf4-a6b9-a1d944993b43`::ref 1 aborting False >Thread-29633::INFO::2012-12-02 14:45:14,129::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29633::DEBUG::2012-12-02 14:45:14,130::task::1151::TaskManager.Task::(prepare) Task=`1202e876-b059-4cf4-a6b9-a1d944993b43`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29633::DEBUG::2012-12-02 14:45:14,130::task::568::TaskManager.Task::(_updateState) Task=`1202e876-b059-4cf4-a6b9-a1d944993b43`::moving from state preparing -> state finished >Thread-29633::DEBUG::2012-12-02 14:45:14,130::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29633::DEBUG::2012-12-02 14:45:14,130::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29633::DEBUG::2012-12-02 14:45:14,130::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29633::DEBUG::2012-12-02 14:45:14,130::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29633::DEBUG::2012-12-02 14:45:14,130::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29633::DEBUG::2012-12-02 14:45:14,130::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29633::DEBUG::2012-12-02 14:45:14,130::task::957::TaskManager.Task::(_decref) Task=`1202e876-b059-4cf4-a6b9-a1d944993b43`::ref 0 aborting False >Thread-29634::DEBUG::2012-12-02 14:45:15,345::task::568::TaskManager.Task::(_updateState) Task=`9c85b1b8-afac-4843-bf95-39bc40c2aedf`::moving from state init -> state preparing >Thread-29634::INFO::2012-12-02 14:45:15,345::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29634::INFO::2012-12-02 14:45:15,345::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0079391002655', 'lastCheck': 1354452310.5577829, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180602073669', 'lastCheck': 1354452306.984988, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00166606903076', 'lastCheck': 1354452307.3914659, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0039370059967', 'lastCheck': 1354452307.100091, 'code': 0, 'valid': True}} >Thread-29634::DEBUG::2012-12-02 14:45:15,346::task::1151::TaskManager.Task::(prepare) Task=`9c85b1b8-afac-4843-bf95-39bc40c2aedf`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0079391002655', 'lastCheck': 1354452310.5577829, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00180602073669', 'lastCheck': 1354452306.984988, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00166606903076', 'lastCheck': 1354452307.3914659, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0039370059967', 'lastCheck': 1354452307.100091, 'code': 0, 'valid': True}} >Thread-29634::DEBUG::2012-12-02 14:45:15,346::task::568::TaskManager.Task::(_updateState) Task=`9c85b1b8-afac-4843-bf95-39bc40c2aedf`::moving from state preparing -> state finished >Thread-29634::DEBUG::2012-12-02 14:45:15,346::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29634::DEBUG::2012-12-02 14:45:15,346::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29634::DEBUG::2012-12-02 14:45:15,346::task::957::TaskManager.Task::(_decref) Task=`9c85b1b8-afac-4843-bf95-39bc40c2aedf`::ref 0 aborting False >Thread-29640::DEBUG::2012-12-02 14:45:24,257::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29640::DEBUG::2012-12-02 14:45:24,258::task::568::TaskManager.Task::(_updateState) Task=`8a6c8eb2-f8f5-45d9-9a4a-d6bc8ce66920`::moving from state init -> state preparing >Thread-29640::INFO::2012-12-02 14:45:24,258::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29640::INFO::2012-12-02 14:45:24,258::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29640::DEBUG::2012-12-02 14:45:24,258::task::1151::TaskManager.Task::(prepare) Task=`8a6c8eb2-f8f5-45d9-9a4a-d6bc8ce66920`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29640::DEBUG::2012-12-02 14:45:24,258::task::568::TaskManager.Task::(_updateState) Task=`8a6c8eb2-f8f5-45d9-9a4a-d6bc8ce66920`::moving from state preparing -> state finished >Thread-29640::DEBUG::2012-12-02 14:45:24,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29640::DEBUG::2012-12-02 14:45:24,259::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29640::DEBUG::2012-12-02 14:45:24,259::task::957::TaskManager.Task::(_decref) Task=`8a6c8eb2-f8f5-45d9-9a4a-d6bc8ce66920`::ref 0 aborting False >Thread-29641::DEBUG::2012-12-02 14:45:24,261::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29641::DEBUG::2012-12-02 14:45:24,261::task::568::TaskManager.Task::(_updateState) Task=`f8757041-bf4c-4cfe-b009-01edf287f9b9`::moving from state init -> state preparing >Thread-29641::INFO::2012-12-02 14:45:24,261::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29641::DEBUG::2012-12-02 14:45:24,262::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5dd64ca2-d096-4066-9b1e-6983e6cbe588`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29641::DEBUG::2012-12-02 14:45:24,262::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29641::DEBUG::2012-12-02 14:45:24,262::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29641::DEBUG::2012-12-02 14:45:24,262::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5dd64ca2-d096-4066-9b1e-6983e6cbe588`::Granted request >Thread-29641::DEBUG::2012-12-02 14:45:24,263::task::794::TaskManager.Task::(resourceAcquired) Task=`f8757041-bf4c-4cfe-b009-01edf287f9b9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29641::DEBUG::2012-12-02 14:45:24,263::task::957::TaskManager.Task::(_decref) Task=`f8757041-bf4c-4cfe-b009-01edf287f9b9`::ref 1 aborting False >Thread-29641::INFO::2012-12-02 14:45:24,268::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29641::DEBUG::2012-12-02 14:45:24,268::task::1151::TaskManager.Task::(prepare) Task=`f8757041-bf4c-4cfe-b009-01edf287f9b9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29641::DEBUG::2012-12-02 14:45:24,268::task::568::TaskManager.Task::(_updateState) Task=`f8757041-bf4c-4cfe-b009-01edf287f9b9`::moving from state preparing -> state finished >Thread-29641::DEBUG::2012-12-02 14:45:24,268::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29641::DEBUG::2012-12-02 14:45:24,268::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29641::DEBUG::2012-12-02 14:45:24,269::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29641::DEBUG::2012-12-02 14:45:24,269::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29641::DEBUG::2012-12-02 14:45:24,269::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29641::DEBUG::2012-12-02 14:45:24,269::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29641::DEBUG::2012-12-02 14:45:24,269::task::957::TaskManager.Task::(_decref) Task=`f8757041-bf4c-4cfe-b009-01edf287f9b9`::ref 0 aborting False >Thread-29642::DEBUG::2012-12-02 14:45:25,485::task::568::TaskManager.Task::(_updateState) Task=`c55cc3fe-2187-4ca4-8189-b8d27894bfe2`::moving from state init -> state preparing >Thread-29642::INFO::2012-12-02 14:45:25,485::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29642::INFO::2012-12-02 14:45:25,485::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00739312171936', 'lastCheck': 1354452320.574645, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00330495834351', 'lastCheck': 1354452316.9991109, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022919178009', 'lastCheck': 1354452317.402993, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00894021987915', 'lastCheck': 1354452317.117861, 'code': 0, 'valid': True}} >Thread-29642::DEBUG::2012-12-02 14:45:25,485::task::1151::TaskManager.Task::(prepare) Task=`c55cc3fe-2187-4ca4-8189-b8d27894bfe2`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00739312171936', 'lastCheck': 1354452320.574645, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00330495834351', 'lastCheck': 1354452316.9991109, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022919178009', 'lastCheck': 1354452317.402993, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00894021987915', 'lastCheck': 1354452317.117861, 'code': 0, 'valid': True}} >Thread-29642::DEBUG::2012-12-02 14:45:25,485::task::568::TaskManager.Task::(_updateState) Task=`c55cc3fe-2187-4ca4-8189-b8d27894bfe2`::moving from state preparing -> state finished >Thread-29642::DEBUG::2012-12-02 14:45:25,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29642::DEBUG::2012-12-02 14:45:25,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29642::DEBUG::2012-12-02 14:45:25,486::task::957::TaskManager.Task::(_decref) Task=`c55cc3fe-2187-4ca4-8189-b8d27894bfe2`::ref 0 aborting False >Thread-29648::DEBUG::2012-12-02 14:45:34,419::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29648::DEBUG::2012-12-02 14:45:34,420::task::568::TaskManager.Task::(_updateState) Task=`6cbb2892-0819-4ebf-a1f0-afe22dc0e48f`::moving from state init -> state preparing >Thread-29648::INFO::2012-12-02 14:45:34,420::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29648::INFO::2012-12-02 14:45:34,420::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29648::DEBUG::2012-12-02 14:45:34,420::task::1151::TaskManager.Task::(prepare) Task=`6cbb2892-0819-4ebf-a1f0-afe22dc0e48f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29648::DEBUG::2012-12-02 14:45:34,420::task::568::TaskManager.Task::(_updateState) Task=`6cbb2892-0819-4ebf-a1f0-afe22dc0e48f`::moving from state preparing -> state finished >Thread-29648::DEBUG::2012-12-02 14:45:34,421::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29648::DEBUG::2012-12-02 14:45:34,421::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29648::DEBUG::2012-12-02 14:45:34,421::task::957::TaskManager.Task::(_decref) Task=`6cbb2892-0819-4ebf-a1f0-afe22dc0e48f`::ref 0 aborting False >Thread-29649::DEBUG::2012-12-02 14:45:34,423::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29649::DEBUG::2012-12-02 14:45:34,424::task::568::TaskManager.Task::(_updateState) Task=`39fc2a01-9476-4702-a36a-e7d915c9cee0`::moving from state init -> state preparing >Thread-29649::INFO::2012-12-02 14:45:34,424::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29649::DEBUG::2012-12-02 14:45:34,424::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0effcc0a-69b5-48c6-8126-3dbba89f66a3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29649::DEBUG::2012-12-02 14:45:34,424::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29649::DEBUG::2012-12-02 14:45:34,424::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29649::DEBUG::2012-12-02 14:45:34,424::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0effcc0a-69b5-48c6-8126-3dbba89f66a3`::Granted request >Thread-29649::DEBUG::2012-12-02 14:45:34,425::task::794::TaskManager.Task::(resourceAcquired) Task=`39fc2a01-9476-4702-a36a-e7d915c9cee0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29649::DEBUG::2012-12-02 14:45:34,425::task::957::TaskManager.Task::(_decref) Task=`39fc2a01-9476-4702-a36a-e7d915c9cee0`::ref 1 aborting False >Thread-29649::INFO::2012-12-02 14:45:34,431::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29649::DEBUG::2012-12-02 14:45:34,431::task::1151::TaskManager.Task::(prepare) Task=`39fc2a01-9476-4702-a36a-e7d915c9cee0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29649::DEBUG::2012-12-02 14:45:34,432::task::568::TaskManager.Task::(_updateState) Task=`39fc2a01-9476-4702-a36a-e7d915c9cee0`::moving from state preparing -> state finished >Thread-29649::DEBUG::2012-12-02 14:45:34,432::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29649::DEBUG::2012-12-02 14:45:34,432::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29649::DEBUG::2012-12-02 14:45:34,432::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29649::DEBUG::2012-12-02 14:45:34,432::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29649::DEBUG::2012-12-02 14:45:34,432::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29649::DEBUG::2012-12-02 14:45:34,432::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29649::DEBUG::2012-12-02 14:45:34,433::task::957::TaskManager.Task::(_decref) Task=`39fc2a01-9476-4702-a36a-e7d915c9cee0`::ref 0 aborting False >Thread-29650::DEBUG::2012-12-02 14:45:35,618::task::568::TaskManager.Task::(_updateState) Task=`a2bffb03-d47c-455a-ab8b-ddd6b281b96f`::moving from state init -> state preparing >Thread-29650::INFO::2012-12-02 14:45:35,618::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29650::INFO::2012-12-02 14:45:35,618::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00776505470276', 'lastCheck': 1354452330.5919621, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00325107574463', 'lastCheck': 1354452327.0197589, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00158596038818', 'lastCheck': 1354452327.4118259, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0075409412384', 'lastCheck': 1354452327.136157, 'code': 0, 'valid': True}} >Thread-29650::DEBUG::2012-12-02 14:45:35,618::task::1151::TaskManager.Task::(prepare) Task=`a2bffb03-d47c-455a-ab8b-ddd6b281b96f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00776505470276', 'lastCheck': 1354452330.5919621, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00325107574463', 'lastCheck': 1354452327.0197589, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00158596038818', 'lastCheck': 1354452327.4118259, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0075409412384', 'lastCheck': 1354452327.136157, 'code': 0, 'valid': True}} >Thread-29650::DEBUG::2012-12-02 14:45:35,618::task::568::TaskManager.Task::(_updateState) Task=`a2bffb03-d47c-455a-ab8b-ddd6b281b96f`::moving from state preparing -> state finished >Thread-29650::DEBUG::2012-12-02 14:45:35,619::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29650::DEBUG::2012-12-02 14:45:35,619::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29650::DEBUG::2012-12-02 14:45:35,619::task::957::TaskManager.Task::(_decref) Task=`a2bffb03-d47c-455a-ab8b-ddd6b281b96f`::ref 0 aborting False >Thread-29656::DEBUG::2012-12-02 14:45:44,563::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29656::DEBUG::2012-12-02 14:45:44,564::task::568::TaskManager.Task::(_updateState) Task=`3f507a08-90bb-47c5-8f57-be830dae065e`::moving from state init -> state preparing >Thread-29656::INFO::2012-12-02 14:45:44,564::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29656::INFO::2012-12-02 14:45:44,564::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29656::DEBUG::2012-12-02 14:45:44,564::task::1151::TaskManager.Task::(prepare) Task=`3f507a08-90bb-47c5-8f57-be830dae065e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29656::DEBUG::2012-12-02 14:45:44,564::task::568::TaskManager.Task::(_updateState) Task=`3f507a08-90bb-47c5-8f57-be830dae065e`::moving from state preparing -> state finished >Thread-29656::DEBUG::2012-12-02 14:45:44,565::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29656::DEBUG::2012-12-02 14:45:44,565::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29656::DEBUG::2012-12-02 14:45:44,565::task::957::TaskManager.Task::(_decref) Task=`3f507a08-90bb-47c5-8f57-be830dae065e`::ref 0 aborting False >Thread-29657::DEBUG::2012-12-02 14:45:44,567::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29657::DEBUG::2012-12-02 14:45:44,568::task::568::TaskManager.Task::(_updateState) Task=`18134a4e-7111-44bd-899e-46af079e484b`::moving from state init -> state preparing >Thread-29657::INFO::2012-12-02 14:45:44,568::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29657::DEBUG::2012-12-02 14:45:44,568::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`def84818-1902-44d9-b5ac-6f174f8027e4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29657::DEBUG::2012-12-02 14:45:44,568::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29657::DEBUG::2012-12-02 14:45:44,568::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29657::DEBUG::2012-12-02 14:45:44,568::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`def84818-1902-44d9-b5ac-6f174f8027e4`::Granted request >Thread-29657::DEBUG::2012-12-02 14:45:44,568::task::794::TaskManager.Task::(resourceAcquired) Task=`18134a4e-7111-44bd-899e-46af079e484b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29657::DEBUG::2012-12-02 14:45:44,568::task::957::TaskManager.Task::(_decref) Task=`18134a4e-7111-44bd-899e-46af079e484b`::ref 1 aborting False >Thread-29657::INFO::2012-12-02 14:45:44,573::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29657::DEBUG::2012-12-02 14:45:44,573::task::1151::TaskManager.Task::(prepare) Task=`18134a4e-7111-44bd-899e-46af079e484b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29657::DEBUG::2012-12-02 14:45:44,573::task::568::TaskManager.Task::(_updateState) Task=`18134a4e-7111-44bd-899e-46af079e484b`::moving from state preparing -> state finished >Thread-29657::DEBUG::2012-12-02 14:45:44,573::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29657::DEBUG::2012-12-02 14:45:44,573::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29657::DEBUG::2012-12-02 14:45:44,573::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29657::DEBUG::2012-12-02 14:45:44,573::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29657::DEBUG::2012-12-02 14:45:44,573::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29657::DEBUG::2012-12-02 14:45:44,574::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29657::DEBUG::2012-12-02 14:45:44,574::task::957::TaskManager.Task::(_decref) Task=`18134a4e-7111-44bd-899e-46af079e484b`::ref 0 aborting False >Thread-29658::DEBUG::2012-12-02 14:45:45,796::task::568::TaskManager.Task::(_updateState) Task=`277131ff-4b68-41b4-8ee5-5ef06fc46452`::moving from state init -> state preparing >Thread-29658::INFO::2012-12-02 14:45:45,796::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29658::INFO::2012-12-02 14:45:45,796::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0094051361084', 'lastCheck': 1354452340.6162961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00216102600098', 'lastCheck': 1354452337.0312729, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00250601768494', 'lastCheck': 1354452337.423382, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00711917877197', 'lastCheck': 1354452337.1521931, 'code': 0, 'valid': True}} >Thread-29658::DEBUG::2012-12-02 14:45:45,796::task::1151::TaskManager.Task::(prepare) Task=`277131ff-4b68-41b4-8ee5-5ef06fc46452`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0094051361084', 'lastCheck': 1354452340.6162961, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00216102600098', 'lastCheck': 1354452337.0312729, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00250601768494', 'lastCheck': 1354452337.423382, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00711917877197', 'lastCheck': 1354452337.1521931, 'code': 0, 'valid': True}} >Thread-29658::DEBUG::2012-12-02 14:45:45,796::task::568::TaskManager.Task::(_updateState) Task=`277131ff-4b68-41b4-8ee5-5ef06fc46452`::moving from state preparing -> state finished >Thread-29658::DEBUG::2012-12-02 14:45:45,797::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29658::DEBUG::2012-12-02 14:45:45,797::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29658::DEBUG::2012-12-02 14:45:45,797::task::957::TaskManager.Task::(_decref) Task=`277131ff-4b68-41b4-8ee5-5ef06fc46452`::ref 0 aborting False >Thread-29664::DEBUG::2012-12-02 14:45:54,691::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29664::DEBUG::2012-12-02 14:45:54,691::task::568::TaskManager.Task::(_updateState) Task=`95efc5ae-a09e-46a7-b926-fa39bf4fc91f`::moving from state init -> state preparing >Thread-29664::INFO::2012-12-02 14:45:54,691::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29664::INFO::2012-12-02 14:45:54,691::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29664::DEBUG::2012-12-02 14:45:54,691::task::1151::TaskManager.Task::(prepare) Task=`95efc5ae-a09e-46a7-b926-fa39bf4fc91f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29664::DEBUG::2012-12-02 14:45:54,692::task::568::TaskManager.Task::(_updateState) Task=`95efc5ae-a09e-46a7-b926-fa39bf4fc91f`::moving from state preparing -> state finished >Thread-29664::DEBUG::2012-12-02 14:45:54,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29664::DEBUG::2012-12-02 14:45:54,692::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29664::DEBUG::2012-12-02 14:45:54,692::task::957::TaskManager.Task::(_decref) Task=`95efc5ae-a09e-46a7-b926-fa39bf4fc91f`::ref 0 aborting False >Thread-29665::DEBUG::2012-12-02 14:45:54,694::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29665::DEBUG::2012-12-02 14:45:54,695::task::568::TaskManager.Task::(_updateState) Task=`1823c5cb-24f3-433d-9a24-d062cdf09bde`::moving from state init -> state preparing >Thread-29665::INFO::2012-12-02 14:45:54,695::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29665::DEBUG::2012-12-02 14:45:54,695::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cab6bf8c-4a35-4ba0-8df1-cf7c2b74a065`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29665::DEBUG::2012-12-02 14:45:54,695::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29665::DEBUG::2012-12-02 14:45:54,695::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29665::DEBUG::2012-12-02 14:45:54,695::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cab6bf8c-4a35-4ba0-8df1-cf7c2b74a065`::Granted request >Thread-29665::DEBUG::2012-12-02 14:45:54,696::task::794::TaskManager.Task::(resourceAcquired) Task=`1823c5cb-24f3-433d-9a24-d062cdf09bde`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29665::DEBUG::2012-12-02 14:45:54,696::task::957::TaskManager.Task::(_decref) Task=`1823c5cb-24f3-433d-9a24-d062cdf09bde`::ref 1 aborting False >Thread-29665::INFO::2012-12-02 14:45:54,701::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29665::DEBUG::2012-12-02 14:45:54,701::task::1151::TaskManager.Task::(prepare) Task=`1823c5cb-24f3-433d-9a24-d062cdf09bde`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29665::DEBUG::2012-12-02 14:45:54,701::task::568::TaskManager.Task::(_updateState) Task=`1823c5cb-24f3-433d-9a24-d062cdf09bde`::moving from state preparing -> state finished >Thread-29665::DEBUG::2012-12-02 14:45:54,701::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29665::DEBUG::2012-12-02 14:45:54,701::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29665::DEBUG::2012-12-02 14:45:54,702::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29665::DEBUG::2012-12-02 14:45:54,702::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29665::DEBUG::2012-12-02 14:45:54,702::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29665::DEBUG::2012-12-02 14:45:54,702::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29665::DEBUG::2012-12-02 14:45:54,702::task::957::TaskManager.Task::(_decref) Task=`1823c5cb-24f3-433d-9a24-d062cdf09bde`::ref 0 aborting False >Thread-29666::DEBUG::2012-12-02 14:45:55,864::task::568::TaskManager.Task::(_updateState) Task=`583aed0b-b7d6-4c13-919c-54c7929e7011`::moving from state init -> state preparing >Thread-29666::INFO::2012-12-02 14:45:55,865::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29666::INFO::2012-12-02 14:45:55,865::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794792175293', 'lastCheck': 1354452350.6333809, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00212693214417', 'lastCheck': 1354452347.0465641, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00184798240662', 'lastCheck': 1354452347.4319811, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778293609619', 'lastCheck': 1354452347.169935, 'code': 0, 'valid': True}} >Thread-29666::DEBUG::2012-12-02 14:45:55,865::task::1151::TaskManager.Task::(prepare) Task=`583aed0b-b7d6-4c13-919c-54c7929e7011`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00794792175293', 'lastCheck': 1354452350.6333809, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00212693214417', 'lastCheck': 1354452347.0465641, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00184798240662', 'lastCheck': 1354452347.4319811, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00778293609619', 'lastCheck': 1354452347.169935, 'code': 0, 'valid': True}} >Thread-29666::DEBUG::2012-12-02 14:45:55,865::task::568::TaskManager.Task::(_updateState) Task=`583aed0b-b7d6-4c13-919c-54c7929e7011`::moving from state preparing -> state finished >Thread-29666::DEBUG::2012-12-02 14:45:55,865::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29666::DEBUG::2012-12-02 14:45:55,865::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29666::DEBUG::2012-12-02 14:45:55,866::task::957::TaskManager.Task::(_decref) Task=`583aed0b-b7d6-4c13-919c-54c7929e7011`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:45:57,432::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:45:57,432::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:45:57,432::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:45:57,636::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:45:57,638::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:45:57,638::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:45:57,673::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:45:57,673::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:45:57,675::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:45:57,677::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:45:57,678::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:45:57,678::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-11821::DEBUG::2012-12-02 14:46:00,633::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:46:00,633::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:46:00,634::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:46:00,843::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:46:00,844::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:46:00,845::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:46:00,880::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:46:00,880::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:46:00,887::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:46:00,889::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:46:00,890::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:46:00,890::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-29794::DEBUG::2012-12-02 14:46:04,826::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29794::DEBUG::2012-12-02 14:46:04,826::task::568::TaskManager.Task::(_updateState) Task=`b4e819c0-557c-4d3b-a3c5-578b06a68e7d`::moving from state init -> state preparing >Thread-29794::INFO::2012-12-02 14:46:04,826::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29794::INFO::2012-12-02 14:46:04,826::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29794::DEBUG::2012-12-02 14:46:04,826::task::1151::TaskManager.Task::(prepare) Task=`b4e819c0-557c-4d3b-a3c5-578b06a68e7d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29794::DEBUG::2012-12-02 14:46:04,827::task::568::TaskManager.Task::(_updateState) Task=`b4e819c0-557c-4d3b-a3c5-578b06a68e7d`::moving from state preparing -> state finished >Thread-29794::DEBUG::2012-12-02 14:46:04,827::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29794::DEBUG::2012-12-02 14:46:04,827::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29794::DEBUG::2012-12-02 14:46:04,827::task::957::TaskManager.Task::(_decref) Task=`b4e819c0-557c-4d3b-a3c5-578b06a68e7d`::ref 0 aborting False >Thread-29795::DEBUG::2012-12-02 14:46:04,829::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29795::DEBUG::2012-12-02 14:46:04,830::task::568::TaskManager.Task::(_updateState) Task=`2ccf0820-d9b3-4858-84ca-41d2e1bd03d5`::moving from state init -> state preparing >Thread-29795::INFO::2012-12-02 14:46:04,830::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29795::DEBUG::2012-12-02 14:46:04,830::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`79657ff3-e7d0-453e-9f25-c7153dbd43ff`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29795::DEBUG::2012-12-02 14:46:04,830::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29795::DEBUG::2012-12-02 14:46:04,830::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29795::DEBUG::2012-12-02 14:46:04,831::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`79657ff3-e7d0-453e-9f25-c7153dbd43ff`::Granted request >Thread-29795::DEBUG::2012-12-02 14:46:04,831::task::794::TaskManager.Task::(resourceAcquired) Task=`2ccf0820-d9b3-4858-84ca-41d2e1bd03d5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29795::DEBUG::2012-12-02 14:46:04,831::task::957::TaskManager.Task::(_decref) Task=`2ccf0820-d9b3-4858-84ca-41d2e1bd03d5`::ref 1 aborting False >Thread-29795::INFO::2012-12-02 14:46:04,837::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29795::DEBUG::2012-12-02 14:46:04,837::task::1151::TaskManager.Task::(prepare) Task=`2ccf0820-d9b3-4858-84ca-41d2e1bd03d5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29795::DEBUG::2012-12-02 14:46:04,837::task::568::TaskManager.Task::(_updateState) Task=`2ccf0820-d9b3-4858-84ca-41d2e1bd03d5`::moving from state preparing -> state finished >Thread-29795::DEBUG::2012-12-02 14:46:04,837::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29795::DEBUG::2012-12-02 14:46:04,837::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29795::DEBUG::2012-12-02 14:46:04,837::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29795::DEBUG::2012-12-02 14:46:04,837::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29795::DEBUG::2012-12-02 14:46:04,837::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29795::DEBUG::2012-12-02 14:46:04,837::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29795::DEBUG::2012-12-02 14:46:04,838::task::957::TaskManager.Task::(_decref) Task=`2ccf0820-d9b3-4858-84ca-41d2e1bd03d5`::ref 0 aborting False >Thread-29796::DEBUG::2012-12-02 14:46:05,923::task::568::TaskManager.Task::(_updateState) Task=`1efd3e3c-318e-448d-8325-f4d1ba14562c`::moving from state init -> state preparing >Thread-29796::INFO::2012-12-02 14:46:05,923::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29796::INFO::2012-12-02 14:46:05,923::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0067138671875', 'lastCheck': 1354452360.905607, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00219798088074', 'lastCheck': 1354452357.0587461, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00196599960327', 'lastCheck': 1354452357.688344, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00905704498291', 'lastCheck': 1354452357.187458, 'code': 0, 'valid': True}} >Thread-29796::DEBUG::2012-12-02 14:46:05,923::task::1151::TaskManager.Task::(prepare) Task=`1efd3e3c-318e-448d-8325-f4d1ba14562c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0067138671875', 'lastCheck': 1354452360.905607, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00219798088074', 'lastCheck': 1354452357.0587461, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00196599960327', 'lastCheck': 1354452357.688344, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00905704498291', 'lastCheck': 1354452357.187458, 'code': 0, 'valid': True}} >Thread-29796::DEBUG::2012-12-02 14:46:05,923::task::568::TaskManager.Task::(_updateState) Task=`1efd3e3c-318e-448d-8325-f4d1ba14562c`::moving from state preparing -> state finished >Thread-29796::DEBUG::2012-12-02 14:46:05,923::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29796::DEBUG::2012-12-02 14:46:05,924::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29796::DEBUG::2012-12-02 14:46:05,924::task::957::TaskManager.Task::(_decref) Task=`1efd3e3c-318e-448d-8325-f4d1ba14562c`::ref 0 aborting False >Thread-11822::DEBUG::2012-12-02 14:46:07,058::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:46:07,059::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:46:07,059::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:46:07,268::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:46:07,269::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:46:07,269::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:46:07,305::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:46:07,305::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:46:07,308::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:46:07,311::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:46:07,311::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:46:07,311::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:46:07,311::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:46:07,312::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:46:07,315::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:46:07,523::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:46:07,524::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:46:07,525::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:46:07,560::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:46:07,560::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:46:07,570::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:46:07,572::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:46:07,572::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:46:07,573::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-29924::DEBUG::2012-12-02 14:46:15,092::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29924::DEBUG::2012-12-02 14:46:15,093::task::568::TaskManager.Task::(_updateState) Task=`b1ed51d5-7e68-445f-9c40-20f23a6b3045`::moving from state init -> state preparing >Thread-29924::INFO::2012-12-02 14:46:15,093::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29924::INFO::2012-12-02 14:46:15,093::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29924::DEBUG::2012-12-02 14:46:15,093::task::1151::TaskManager.Task::(prepare) Task=`b1ed51d5-7e68-445f-9c40-20f23a6b3045`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29924::DEBUG::2012-12-02 14:46:15,093::task::568::TaskManager.Task::(_updateState) Task=`b1ed51d5-7e68-445f-9c40-20f23a6b3045`::moving from state preparing -> state finished >Thread-29924::DEBUG::2012-12-02 14:46:15,093::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29924::DEBUG::2012-12-02 14:46:15,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29924::DEBUG::2012-12-02 14:46:15,094::task::957::TaskManager.Task::(_decref) Task=`b1ed51d5-7e68-445f-9c40-20f23a6b3045`::ref 0 aborting False >Thread-29925::DEBUG::2012-12-02 14:46:15,096::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29925::DEBUG::2012-12-02 14:46:15,096::task::568::TaskManager.Task::(_updateState) Task=`096ae8a0-efb5-49a8-9c7c-d4081ede51ef`::moving from state init -> state preparing >Thread-29925::INFO::2012-12-02 14:46:15,097::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29925::DEBUG::2012-12-02 14:46:15,097::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e6f6271-2bf1-48c4-b0a9-8ff445af6e2f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29925::DEBUG::2012-12-02 14:46:15,097::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29925::DEBUG::2012-12-02 14:46:15,097::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29925::DEBUG::2012-12-02 14:46:15,097::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`3e6f6271-2bf1-48c4-b0a9-8ff445af6e2f`::Granted request >Thread-29925::DEBUG::2012-12-02 14:46:15,098::task::794::TaskManager.Task::(resourceAcquired) Task=`096ae8a0-efb5-49a8-9c7c-d4081ede51ef`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29925::DEBUG::2012-12-02 14:46:15,098::task::957::TaskManager.Task::(_decref) Task=`096ae8a0-efb5-49a8-9c7c-d4081ede51ef`::ref 1 aborting False >Thread-29925::INFO::2012-12-02 14:46:15,104::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29925::DEBUG::2012-12-02 14:46:15,104::task::1151::TaskManager.Task::(prepare) Task=`096ae8a0-efb5-49a8-9c7c-d4081ede51ef`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29925::DEBUG::2012-12-02 14:46:15,105::task::568::TaskManager.Task::(_updateState) Task=`096ae8a0-efb5-49a8-9c7c-d4081ede51ef`::moving from state preparing -> state finished >Thread-29925::DEBUG::2012-12-02 14:46:15,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29925::DEBUG::2012-12-02 14:46:15,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29925::DEBUG::2012-12-02 14:46:15,105::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29925::DEBUG::2012-12-02 14:46:15,105::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29925::DEBUG::2012-12-02 14:46:15,105::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29925::DEBUG::2012-12-02 14:46:15,105::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29925::DEBUG::2012-12-02 14:46:15,105::task::957::TaskManager.Task::(_decref) Task=`096ae8a0-efb5-49a8-9c7c-d4081ede51ef`::ref 0 aborting False >Thread-29926::DEBUG::2012-12-02 14:46:16,019::task::568::TaskManager.Task::(_updateState) Task=`2bd9d969-b6bb-4bcd-860d-a8d686d8e701`::moving from state init -> state preparing >Thread-29926::INFO::2012-12-02 14:46:16,020::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29926::INFO::2012-12-02 14:46:16,020::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00848984718323', 'lastCheck': 1354452370.9240401, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0026159286499', 'lastCheck': 1354452367.5818281, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00167202949524', 'lastCheck': 1354452367.698545, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00639390945435', 'lastCheck': 1354452367.589993, 'code': 0, 'valid': True}} >Thread-29926::DEBUG::2012-12-02 14:46:16,020::task::1151::TaskManager.Task::(prepare) Task=`2bd9d969-b6bb-4bcd-860d-a8d686d8e701`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00848984718323', 'lastCheck': 1354452370.9240401, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0026159286499', 'lastCheck': 1354452367.5818281, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00167202949524', 'lastCheck': 1354452367.698545, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00639390945435', 'lastCheck': 1354452367.589993, 'code': 0, 'valid': True}} >Thread-29926::DEBUG::2012-12-02 14:46:16,020::task::568::TaskManager.Task::(_updateState) Task=`2bd9d969-b6bb-4bcd-860d-a8d686d8e701`::moving from state preparing -> state finished >Thread-29926::DEBUG::2012-12-02 14:46:16,020::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29926::DEBUG::2012-12-02 14:46:16,020::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29926::DEBUG::2012-12-02 14:46:16,021::task::957::TaskManager.Task::(_decref) Task=`2bd9d969-b6bb-4bcd-860d-a8d686d8e701`::ref 0 aborting False >Thread-29932::DEBUG::2012-12-02 14:46:25,245::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29932::DEBUG::2012-12-02 14:46:25,246::task::568::TaskManager.Task::(_updateState) Task=`ea7a804f-bff9-4643-bc03-f10021300925`::moving from state init -> state preparing >Thread-29932::INFO::2012-12-02 14:46:25,246::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29932::INFO::2012-12-02 14:46:25,246::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29932::DEBUG::2012-12-02 14:46:25,246::task::1151::TaskManager.Task::(prepare) Task=`ea7a804f-bff9-4643-bc03-f10021300925`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29932::DEBUG::2012-12-02 14:46:25,246::task::568::TaskManager.Task::(_updateState) Task=`ea7a804f-bff9-4643-bc03-f10021300925`::moving from state preparing -> state finished >Thread-29932::DEBUG::2012-12-02 14:46:25,246::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29932::DEBUG::2012-12-02 14:46:25,247::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29932::DEBUG::2012-12-02 14:46:25,247::task::957::TaskManager.Task::(_decref) Task=`ea7a804f-bff9-4643-bc03-f10021300925`::ref 0 aborting False >Thread-29933::DEBUG::2012-12-02 14:46:25,249::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29933::DEBUG::2012-12-02 14:46:25,249::task::568::TaskManager.Task::(_updateState) Task=`8804d343-27e7-4011-ac8d-29cbe9da9cb6`::moving from state init -> state preparing >Thread-29933::INFO::2012-12-02 14:46:25,250::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29933::DEBUG::2012-12-02 14:46:25,250::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`69748bb2-1dd9-40b3-a385-4fd5f42ed29b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29933::DEBUG::2012-12-02 14:46:25,250::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29933::DEBUG::2012-12-02 14:46:25,250::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29933::DEBUG::2012-12-02 14:46:25,250::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`69748bb2-1dd9-40b3-a385-4fd5f42ed29b`::Granted request >Thread-29933::DEBUG::2012-12-02 14:46:25,251::task::794::TaskManager.Task::(resourceAcquired) Task=`8804d343-27e7-4011-ac8d-29cbe9da9cb6`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29933::DEBUG::2012-12-02 14:46:25,251::task::957::TaskManager.Task::(_decref) Task=`8804d343-27e7-4011-ac8d-29cbe9da9cb6`::ref 1 aborting False >Thread-29933::INFO::2012-12-02 14:46:25,256::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29933::DEBUG::2012-12-02 14:46:25,256::task::1151::TaskManager.Task::(prepare) Task=`8804d343-27e7-4011-ac8d-29cbe9da9cb6`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29933::DEBUG::2012-12-02 14:46:25,257::task::568::TaskManager.Task::(_updateState) Task=`8804d343-27e7-4011-ac8d-29cbe9da9cb6`::moving from state preparing -> state finished >Thread-29933::DEBUG::2012-12-02 14:46:25,257::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29933::DEBUG::2012-12-02 14:46:25,257::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29933::DEBUG::2012-12-02 14:46:25,257::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29933::DEBUG::2012-12-02 14:46:25,257::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29933::DEBUG::2012-12-02 14:46:25,257::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29933::DEBUG::2012-12-02 14:46:25,257::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29933::DEBUG::2012-12-02 14:46:25,257::task::957::TaskManager.Task::(_decref) Task=`8804d343-27e7-4011-ac8d-29cbe9da9cb6`::ref 0 aborting False >Thread-29934::DEBUG::2012-12-02 14:46:26,148::task::568::TaskManager.Task::(_updateState) Task=`071383d1-c1b1-4f46-9710-5e3f3c77395f`::moving from state init -> state preparing >Thread-29934::INFO::2012-12-02 14:46:26,149::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29934::INFO::2012-12-02 14:46:26,149::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0135300159454', 'lastCheck': 1354452380.9480259, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00612688064575', 'lastCheck': 1354452377.6059179, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00340509414673', 'lastCheck': 1354452377.712352, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0116889476776', 'lastCheck': 1354452377.612633, 'code': 0, 'valid': True}} >Thread-29934::DEBUG::2012-12-02 14:46:26,149::task::1151::TaskManager.Task::(prepare) Task=`071383d1-c1b1-4f46-9710-5e3f3c77395f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0135300159454', 'lastCheck': 1354452380.9480259, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00612688064575', 'lastCheck': 1354452377.6059179, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00340509414673', 'lastCheck': 1354452377.712352, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0116889476776', 'lastCheck': 1354452377.612633, 'code': 0, 'valid': True}} >Thread-29934::DEBUG::2012-12-02 14:46:26,149::task::568::TaskManager.Task::(_updateState) Task=`071383d1-c1b1-4f46-9710-5e3f3c77395f`::moving from state preparing -> state finished >Thread-29934::DEBUG::2012-12-02 14:46:26,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29934::DEBUG::2012-12-02 14:46:26,149::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29934::DEBUG::2012-12-02 14:46:26,149::task::957::TaskManager.Task::(_decref) Task=`071383d1-c1b1-4f46-9710-5e3f3c77395f`::ref 0 aborting False >Thread-29940::DEBUG::2012-12-02 14:46:35,405::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29940::DEBUG::2012-12-02 14:46:35,405::task::568::TaskManager.Task::(_updateState) Task=`95dabe9b-c94f-49d1-a467-53ef0c1f35a9`::moving from state init -> state preparing >Thread-29940::INFO::2012-12-02 14:46:35,405::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29940::INFO::2012-12-02 14:46:35,406::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29940::DEBUG::2012-12-02 14:46:35,406::task::1151::TaskManager.Task::(prepare) Task=`95dabe9b-c94f-49d1-a467-53ef0c1f35a9`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29940::DEBUG::2012-12-02 14:46:35,406::task::568::TaskManager.Task::(_updateState) Task=`95dabe9b-c94f-49d1-a467-53ef0c1f35a9`::moving from state preparing -> state finished >Thread-29940::DEBUG::2012-12-02 14:46:35,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29940::DEBUG::2012-12-02 14:46:35,406::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29940::DEBUG::2012-12-02 14:46:35,406::task::957::TaskManager.Task::(_decref) Task=`95dabe9b-c94f-49d1-a467-53ef0c1f35a9`::ref 0 aborting False >Thread-29941::DEBUG::2012-12-02 14:46:35,409::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29941::DEBUG::2012-12-02 14:46:35,410::task::568::TaskManager.Task::(_updateState) Task=`5eb6148d-6212-4912-a1e5-b9568417fd87`::moving from state init -> state preparing >Thread-29941::INFO::2012-12-02 14:46:35,410::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29941::DEBUG::2012-12-02 14:46:35,410::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4cc3534d-e54d-4734-a67b-9449230af270`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29941::DEBUG::2012-12-02 14:46:35,410::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29941::DEBUG::2012-12-02 14:46:35,411::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29941::DEBUG::2012-12-02 14:46:35,411::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4cc3534d-e54d-4734-a67b-9449230af270`::Granted request >Thread-29941::DEBUG::2012-12-02 14:46:35,411::task::794::TaskManager.Task::(resourceAcquired) Task=`5eb6148d-6212-4912-a1e5-b9568417fd87`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29941::DEBUG::2012-12-02 14:46:35,411::task::957::TaskManager.Task::(_decref) Task=`5eb6148d-6212-4912-a1e5-b9568417fd87`::ref 1 aborting False >Thread-29941::INFO::2012-12-02 14:46:35,417::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29941::DEBUG::2012-12-02 14:46:35,417::task::1151::TaskManager.Task::(prepare) Task=`5eb6148d-6212-4912-a1e5-b9568417fd87`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29941::DEBUG::2012-12-02 14:46:35,417::task::568::TaskManager.Task::(_updateState) Task=`5eb6148d-6212-4912-a1e5-b9568417fd87`::moving from state preparing -> state finished >Thread-29941::DEBUG::2012-12-02 14:46:35,417::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29941::DEBUG::2012-12-02 14:46:35,417::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29941::DEBUG::2012-12-02 14:46:35,417::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29941::DEBUG::2012-12-02 14:46:35,417::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29941::DEBUG::2012-12-02 14:46:35,418::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29941::DEBUG::2012-12-02 14:46:35,418::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29941::DEBUG::2012-12-02 14:46:35,418::task::957::TaskManager.Task::(_decref) Task=`5eb6148d-6212-4912-a1e5-b9568417fd87`::ref 0 aborting False >Thread-29942::DEBUG::2012-12-02 14:46:36,235::task::568::TaskManager.Task::(_updateState) Task=`a8bcccb4-1dbb-4eb0-beba-56b472de8847`::moving from state init -> state preparing >Thread-29942::INFO::2012-12-02 14:46:36,235::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29942::INFO::2012-12-02 14:46:36,235::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00778102874756', 'lastCheck': 1354452390.9656751, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00350880622864', 'lastCheck': 1354452387.6259501, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00166082382202', 'lastCheck': 1354452387.721683, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842905044556', 'lastCheck': 1354452387.630821, 'code': 0, 'valid': True}} >Thread-29942::DEBUG::2012-12-02 14:46:36,235::task::1151::TaskManager.Task::(prepare) Task=`a8bcccb4-1dbb-4eb0-beba-56b472de8847`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00778102874756', 'lastCheck': 1354452390.9656751, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00350880622864', 'lastCheck': 1354452387.6259501, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00166082382202', 'lastCheck': 1354452387.721683, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00842905044556', 'lastCheck': 1354452387.630821, 'code': 0, 'valid': True}} >Thread-29942::DEBUG::2012-12-02 14:46:36,235::task::568::TaskManager.Task::(_updateState) Task=`a8bcccb4-1dbb-4eb0-beba-56b472de8847`::moving from state preparing -> state finished >Thread-29942::DEBUG::2012-12-02 14:46:36,235::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29942::DEBUG::2012-12-02 14:46:36,236::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29942::DEBUG::2012-12-02 14:46:36,236::task::957::TaskManager.Task::(_decref) Task=`a8bcccb4-1dbb-4eb0-beba-56b472de8847`::ref 0 aborting False >Thread-29948::DEBUG::2012-12-02 14:46:45,595::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29948::DEBUG::2012-12-02 14:46:45,596::task::568::TaskManager.Task::(_updateState) Task=`638e5ff7-b231-4be7-9141-448260e4a257`::moving from state init -> state preparing >Thread-29948::INFO::2012-12-02 14:46:45,596::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29948::INFO::2012-12-02 14:46:45,596::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29948::DEBUG::2012-12-02 14:46:45,596::task::1151::TaskManager.Task::(prepare) Task=`638e5ff7-b231-4be7-9141-448260e4a257`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29948::DEBUG::2012-12-02 14:46:45,596::task::568::TaskManager.Task::(_updateState) Task=`638e5ff7-b231-4be7-9141-448260e4a257`::moving from state preparing -> state finished >Thread-29948::DEBUG::2012-12-02 14:46:45,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29948::DEBUG::2012-12-02 14:46:45,597::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29948::DEBUG::2012-12-02 14:46:45,597::task::957::TaskManager.Task::(_decref) Task=`638e5ff7-b231-4be7-9141-448260e4a257`::ref 0 aborting False >Thread-29949::DEBUG::2012-12-02 14:46:45,599::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29949::DEBUG::2012-12-02 14:46:45,600::task::568::TaskManager.Task::(_updateState) Task=`926f27e7-9476-4d34-b191-ffd1c1261d90`::moving from state init -> state preparing >Thread-29949::INFO::2012-12-02 14:46:45,600::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29949::DEBUG::2012-12-02 14:46:45,600::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4de215a4-5647-4121-a6d6-896cb490b89e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29949::DEBUG::2012-12-02 14:46:45,600::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29949::DEBUG::2012-12-02 14:46:45,600::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29949::DEBUG::2012-12-02 14:46:45,600::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4de215a4-5647-4121-a6d6-896cb490b89e`::Granted request >Thread-29949::DEBUG::2012-12-02 14:46:45,601::task::794::TaskManager.Task::(resourceAcquired) Task=`926f27e7-9476-4d34-b191-ffd1c1261d90`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29949::DEBUG::2012-12-02 14:46:45,601::task::957::TaskManager.Task::(_decref) Task=`926f27e7-9476-4d34-b191-ffd1c1261d90`::ref 1 aborting False >Thread-29949::INFO::2012-12-02 14:46:45,607::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29949::DEBUG::2012-12-02 14:46:45,607::task::1151::TaskManager.Task::(prepare) Task=`926f27e7-9476-4d34-b191-ffd1c1261d90`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29949::DEBUG::2012-12-02 14:46:45,607::task::568::TaskManager.Task::(_updateState) Task=`926f27e7-9476-4d34-b191-ffd1c1261d90`::moving from state preparing -> state finished >Thread-29949::DEBUG::2012-12-02 14:46:45,607::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29949::DEBUG::2012-12-02 14:46:45,607::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29949::DEBUG::2012-12-02 14:46:45,607::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29949::DEBUG::2012-12-02 14:46:45,608::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29949::DEBUG::2012-12-02 14:46:45,608::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29949::DEBUG::2012-12-02 14:46:45,608::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29949::DEBUG::2012-12-02 14:46:45,608::task::957::TaskManager.Task::(_decref) Task=`926f27e7-9476-4d34-b191-ffd1c1261d90`::ref 0 aborting False >Thread-29950::DEBUG::2012-12-02 14:46:46,375::task::568::TaskManager.Task::(_updateState) Task=`70776d7c-6c0e-42cc-a284-6a42b946f7f3`::moving from state init -> state preparing >Thread-29950::INFO::2012-12-02 14:46:46,375::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29950::INFO::2012-12-02 14:46:46,375::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00970196723938', 'lastCheck': 1354452400.9878421, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00359010696411', 'lastCheck': 1354452397.6408601, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00282502174377', 'lastCheck': 1354452397.7338741, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00793194770813', 'lastCheck': 1354452397.649564, 'code': 0, 'valid': True}} >Thread-29950::DEBUG::2012-12-02 14:46:46,375::task::1151::TaskManager.Task::(prepare) Task=`70776d7c-6c0e-42cc-a284-6a42b946f7f3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00970196723938', 'lastCheck': 1354452400.9878421, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00359010696411', 'lastCheck': 1354452397.6408601, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00282502174377', 'lastCheck': 1354452397.7338741, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00793194770813', 'lastCheck': 1354452397.649564, 'code': 0, 'valid': True}} >Thread-29950::DEBUG::2012-12-02 14:46:46,375::task::568::TaskManager.Task::(_updateState) Task=`70776d7c-6c0e-42cc-a284-6a42b946f7f3`::moving from state preparing -> state finished >Thread-29950::DEBUG::2012-12-02 14:46:46,376::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29950::DEBUG::2012-12-02 14:46:46,376::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29950::DEBUG::2012-12-02 14:46:46,376::task::957::TaskManager.Task::(_decref) Task=`70776d7c-6c0e-42cc-a284-6a42b946f7f3`::ref 0 aborting False >Thread-29956::DEBUG::2012-12-02 14:46:55,715::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29956::DEBUG::2012-12-02 14:46:55,716::task::568::TaskManager.Task::(_updateState) Task=`f918e732-ee45-4593-8f42-9345b31c7e7d`::moving from state init -> state preparing >Thread-29956::INFO::2012-12-02 14:46:55,716::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29956::INFO::2012-12-02 14:46:55,716::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29956::DEBUG::2012-12-02 14:46:55,716::task::1151::TaskManager.Task::(prepare) Task=`f918e732-ee45-4593-8f42-9345b31c7e7d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29956::DEBUG::2012-12-02 14:46:55,717::task::568::TaskManager.Task::(_updateState) Task=`f918e732-ee45-4593-8f42-9345b31c7e7d`::moving from state preparing -> state finished >Thread-29956::DEBUG::2012-12-02 14:46:55,717::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29956::DEBUG::2012-12-02 14:46:55,717::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29956::DEBUG::2012-12-02 14:46:55,717::task::957::TaskManager.Task::(_decref) Task=`f918e732-ee45-4593-8f42-9345b31c7e7d`::ref 0 aborting False >Thread-29957::DEBUG::2012-12-02 14:46:55,719::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29957::DEBUG::2012-12-02 14:46:55,720::task::568::TaskManager.Task::(_updateState) Task=`4646ecb2-1dc6-4d00-bee5-830c2ed10eb5`::moving from state init -> state preparing >Thread-29957::INFO::2012-12-02 14:46:55,720::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29957::DEBUG::2012-12-02 14:46:55,720::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`478e64cb-1565-4c0f-9c62-84f66bf32aae`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29957::DEBUG::2012-12-02 14:46:55,720::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29957::DEBUG::2012-12-02 14:46:55,720::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29957::DEBUG::2012-12-02 14:46:55,720::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`478e64cb-1565-4c0f-9c62-84f66bf32aae`::Granted request >Thread-29957::DEBUG::2012-12-02 14:46:55,721::task::794::TaskManager.Task::(resourceAcquired) Task=`4646ecb2-1dc6-4d00-bee5-830c2ed10eb5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29957::DEBUG::2012-12-02 14:46:55,721::task::957::TaskManager.Task::(_decref) Task=`4646ecb2-1dc6-4d00-bee5-830c2ed10eb5`::ref 1 aborting False >Thread-29957::INFO::2012-12-02 14:46:55,727::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29957::DEBUG::2012-12-02 14:46:55,727::task::1151::TaskManager.Task::(prepare) Task=`4646ecb2-1dc6-4d00-bee5-830c2ed10eb5`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29957::DEBUG::2012-12-02 14:46:55,727::task::568::TaskManager.Task::(_updateState) Task=`4646ecb2-1dc6-4d00-bee5-830c2ed10eb5`::moving from state preparing -> state finished >Thread-29957::DEBUG::2012-12-02 14:46:55,727::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29957::DEBUG::2012-12-02 14:46:55,727::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29957::DEBUG::2012-12-02 14:46:55,727::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29957::DEBUG::2012-12-02 14:46:55,727::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29957::DEBUG::2012-12-02 14:46:55,727::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29957::DEBUG::2012-12-02 14:46:55,727::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29957::DEBUG::2012-12-02 14:46:55,727::task::957::TaskManager.Task::(_decref) Task=`4646ecb2-1dc6-4d00-bee5-830c2ed10eb5`::ref 0 aborting False >Thread-29958::DEBUG::2012-12-02 14:46:56,484::task::568::TaskManager.Task::(_updateState) Task=`d6682eba-d14f-4ee4-bf55-9320b40c00c4`::moving from state init -> state preparing >Thread-29958::INFO::2012-12-02 14:46:56,484::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29958::INFO::2012-12-02 14:46:56,484::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00740599632263', 'lastCheck': 1354452411.004699, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0027642250061', 'lastCheck': 1354452407.660301, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00156998634338', 'lastCheck': 1354452407.742703, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00803399085999', 'lastCheck': 1354452407.6708961, 'code': 0, 'valid': True}} >Thread-29958::DEBUG::2012-12-02 14:46:56,484::task::1151::TaskManager.Task::(prepare) Task=`d6682eba-d14f-4ee4-bf55-9320b40c00c4`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00740599632263', 'lastCheck': 1354452411.004699, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0027642250061', 'lastCheck': 1354452407.660301, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00156998634338', 'lastCheck': 1354452407.742703, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00803399085999', 'lastCheck': 1354452407.6708961, 'code': 0, 'valid': True}} >Thread-29958::DEBUG::2012-12-02 14:46:56,484::task::568::TaskManager.Task::(_updateState) Task=`d6682eba-d14f-4ee4-bf55-9320b40c00c4`::moving from state preparing -> state finished >Thread-29958::DEBUG::2012-12-02 14:46:56,484::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29958::DEBUG::2012-12-02 14:46:56,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29958::DEBUG::2012-12-02 14:46:56,485::task::957::TaskManager.Task::(_decref) Task=`d6682eba-d14f-4ee4-bf55-9320b40c00c4`::ref 0 aborting False >Thread-29964::DEBUG::2012-12-02 14:47:05,847::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29964::DEBUG::2012-12-02 14:47:05,847::task::568::TaskManager.Task::(_updateState) Task=`c39c8036-b0ad-45c9-9297-f535cce382e6`::moving from state init -> state preparing >Thread-29964::INFO::2012-12-02 14:47:05,847::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29964::INFO::2012-12-02 14:47:05,847::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29964::DEBUG::2012-12-02 14:47:05,848::task::1151::TaskManager.Task::(prepare) Task=`c39c8036-b0ad-45c9-9297-f535cce382e6`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29964::DEBUG::2012-12-02 14:47:05,848::task::568::TaskManager.Task::(_updateState) Task=`c39c8036-b0ad-45c9-9297-f535cce382e6`::moving from state preparing -> state finished >Thread-29964::DEBUG::2012-12-02 14:47:05,848::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29964::DEBUG::2012-12-02 14:47:05,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29964::DEBUG::2012-12-02 14:47:05,848::task::957::TaskManager.Task::(_decref) Task=`c39c8036-b0ad-45c9-9297-f535cce382e6`::ref 0 aborting False >Thread-29965::DEBUG::2012-12-02 14:47:05,851::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29965::DEBUG::2012-12-02 14:47:05,851::task::568::TaskManager.Task::(_updateState) Task=`75d31a1f-d528-4e49-8b46-201cf7546e1a`::moving from state init -> state preparing >Thread-29965::INFO::2012-12-02 14:47:05,851::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29965::DEBUG::2012-12-02 14:47:05,851::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7b687800-777a-4b07-83f8-dfe09beeab0b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29965::DEBUG::2012-12-02 14:47:05,852::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29965::DEBUG::2012-12-02 14:47:05,852::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29965::DEBUG::2012-12-02 14:47:05,852::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7b687800-777a-4b07-83f8-dfe09beeab0b`::Granted request >Thread-29965::DEBUG::2012-12-02 14:47:05,852::task::794::TaskManager.Task::(resourceAcquired) Task=`75d31a1f-d528-4e49-8b46-201cf7546e1a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29965::DEBUG::2012-12-02 14:47:05,852::task::957::TaskManager.Task::(_decref) Task=`75d31a1f-d528-4e49-8b46-201cf7546e1a`::ref 1 aborting False >Thread-29965::INFO::2012-12-02 14:47:05,857::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29965::DEBUG::2012-12-02 14:47:05,858::task::1151::TaskManager.Task::(prepare) Task=`75d31a1f-d528-4e49-8b46-201cf7546e1a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29965::DEBUG::2012-12-02 14:47:05,858::task::568::TaskManager.Task::(_updateState) Task=`75d31a1f-d528-4e49-8b46-201cf7546e1a`::moving from state preparing -> state finished >Thread-29965::DEBUG::2012-12-02 14:47:05,858::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29965::DEBUG::2012-12-02 14:47:05,858::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29965::DEBUG::2012-12-02 14:47:05,858::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29965::DEBUG::2012-12-02 14:47:05,858::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29965::DEBUG::2012-12-02 14:47:05,858::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29965::DEBUG::2012-12-02 14:47:05,858::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29965::DEBUG::2012-12-02 14:47:05,858::task::957::TaskManager.Task::(_decref) Task=`75d31a1f-d528-4e49-8b46-201cf7546e1a`::ref 0 aborting False >Thread-29966::DEBUG::2012-12-02 14:47:06,590::task::568::TaskManager.Task::(_updateState) Task=`52444204-fa0d-4d96-ac0c-88c72dfa08a2`::moving from state init -> state preparing >Thread-29966::INFO::2012-12-02 14:47:06,590::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29966::INFO::2012-12-02 14:47:06,590::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0115880966187', 'lastCheck': 1354452421.0264161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0036940574646', 'lastCheck': 1354452417.6781471, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191497802734', 'lastCheck': 1354452417.7521639, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00817203521729', 'lastCheck': 1354452417.6885469, 'code': 0, 'valid': True}} >Thread-29966::DEBUG::2012-12-02 14:47:06,590::task::1151::TaskManager.Task::(prepare) Task=`52444204-fa0d-4d96-ac0c-88c72dfa08a2`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0115880966187', 'lastCheck': 1354452421.0264161, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0036940574646', 'lastCheck': 1354452417.6781471, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191497802734', 'lastCheck': 1354452417.7521639, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00817203521729', 'lastCheck': 1354452417.6885469, 'code': 0, 'valid': True}} >Thread-29966::DEBUG::2012-12-02 14:47:06,591::task::568::TaskManager.Task::(_updateState) Task=`52444204-fa0d-4d96-ac0c-88c72dfa08a2`::moving from state preparing -> state finished >Thread-29966::DEBUG::2012-12-02 14:47:06,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29966::DEBUG::2012-12-02 14:47:06,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29966::DEBUG::2012-12-02 14:47:06,591::task::957::TaskManager.Task::(_decref) Task=`52444204-fa0d-4d96-ac0c-88c72dfa08a2`::ref 0 aborting False >Thread-29972::DEBUG::2012-12-02 14:47:15,975::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29972::DEBUG::2012-12-02 14:47:15,975::task::568::TaskManager.Task::(_updateState) Task=`162ac330-fdee-4836-87c1-28e48b696fc3`::moving from state init -> state preparing >Thread-29972::INFO::2012-12-02 14:47:15,976::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29972::INFO::2012-12-02 14:47:15,976::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29972::DEBUG::2012-12-02 14:47:15,976::task::1151::TaskManager.Task::(prepare) Task=`162ac330-fdee-4836-87c1-28e48b696fc3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29972::DEBUG::2012-12-02 14:47:15,976::task::568::TaskManager.Task::(_updateState) Task=`162ac330-fdee-4836-87c1-28e48b696fc3`::moving from state preparing -> state finished >Thread-29972::DEBUG::2012-12-02 14:47:15,976::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29972::DEBUG::2012-12-02 14:47:15,976::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29972::DEBUG::2012-12-02 14:47:15,977::task::957::TaskManager.Task::(_decref) Task=`162ac330-fdee-4836-87c1-28e48b696fc3`::ref 0 aborting False >Thread-29973::DEBUG::2012-12-02 14:47:15,979::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29973::DEBUG::2012-12-02 14:47:15,979::task::568::TaskManager.Task::(_updateState) Task=`ffe090d6-0612-43dc-8286-70116e2a83a9`::moving from state init -> state preparing >Thread-29973::INFO::2012-12-02 14:47:15,980::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29973::DEBUG::2012-12-02 14:47:15,980::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`867012f8-edb7-464c-9ce6-1feceaa71da2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29973::DEBUG::2012-12-02 14:47:15,980::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29973::DEBUG::2012-12-02 14:47:15,980::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29973::DEBUG::2012-12-02 14:47:15,980::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`867012f8-edb7-464c-9ce6-1feceaa71da2`::Granted request >Thread-29973::DEBUG::2012-12-02 14:47:15,981::task::794::TaskManager.Task::(resourceAcquired) Task=`ffe090d6-0612-43dc-8286-70116e2a83a9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29973::DEBUG::2012-12-02 14:47:15,981::task::957::TaskManager.Task::(_decref) Task=`ffe090d6-0612-43dc-8286-70116e2a83a9`::ref 1 aborting False >Thread-29973::INFO::2012-12-02 14:47:15,986::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29973::DEBUG::2012-12-02 14:47:15,987::task::1151::TaskManager.Task::(prepare) Task=`ffe090d6-0612-43dc-8286-70116e2a83a9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29973::DEBUG::2012-12-02 14:47:15,987::task::568::TaskManager.Task::(_updateState) Task=`ffe090d6-0612-43dc-8286-70116e2a83a9`::moving from state preparing -> state finished >Thread-29973::DEBUG::2012-12-02 14:47:15,987::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29973::DEBUG::2012-12-02 14:47:15,987::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29973::DEBUG::2012-12-02 14:47:15,987::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29973::DEBUG::2012-12-02 14:47:15,987::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29973::DEBUG::2012-12-02 14:47:15,987::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29973::DEBUG::2012-12-02 14:47:15,987::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29973::DEBUG::2012-12-02 14:47:15,987::task::957::TaskManager.Task::(_decref) Task=`ffe090d6-0612-43dc-8286-70116e2a83a9`::ref 0 aborting False >Thread-29974::DEBUG::2012-12-02 14:47:16,702::task::568::TaskManager.Task::(_updateState) Task=`25313966-a677-4223-9418-7bf5a533f368`::moving from state init -> state preparing >Thread-29974::INFO::2012-12-02 14:47:16,703::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29974::INFO::2012-12-02 14:47:16,703::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0144131183624', 'lastCheck': 1354452431.054297, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00223994255066', 'lastCheck': 1354452427.6971309, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181317329407', 'lastCheck': 1354452427.761719, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00806999206543', 'lastCheck': 1354452427.70751, 'code': 0, 'valid': True}} >Thread-29974::DEBUG::2012-12-02 14:47:16,703::task::1151::TaskManager.Task::(prepare) Task=`25313966-a677-4223-9418-7bf5a533f368`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0144131183624', 'lastCheck': 1354452431.054297, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00223994255066', 'lastCheck': 1354452427.6971309, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181317329407', 'lastCheck': 1354452427.761719, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00806999206543', 'lastCheck': 1354452427.70751, 'code': 0, 'valid': True}} >Thread-29974::DEBUG::2012-12-02 14:47:16,703::task::568::TaskManager.Task::(_updateState) Task=`25313966-a677-4223-9418-7bf5a533f368`::moving from state preparing -> state finished >Thread-29974::DEBUG::2012-12-02 14:47:16,703::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29974::DEBUG::2012-12-02 14:47:16,703::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29974::DEBUG::2012-12-02 14:47:16,703::task::957::TaskManager.Task::(_decref) Task=`25313966-a677-4223-9418-7bf5a533f368`::ref 0 aborting False >Thread-29980::DEBUG::2012-12-02 14:47:26,155::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29980::DEBUG::2012-12-02 14:47:26,155::task::568::TaskManager.Task::(_updateState) Task=`19243aae-0a8b-4396-a5db-db7ce3c7eb7a`::moving from state init -> state preparing >Thread-29980::INFO::2012-12-02 14:47:26,155::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29980::INFO::2012-12-02 14:47:26,156::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29980::DEBUG::2012-12-02 14:47:26,156::task::1151::TaskManager.Task::(prepare) Task=`19243aae-0a8b-4396-a5db-db7ce3c7eb7a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29980::DEBUG::2012-12-02 14:47:26,156::task::568::TaskManager.Task::(_updateState) Task=`19243aae-0a8b-4396-a5db-db7ce3c7eb7a`::moving from state preparing -> state finished >Thread-29980::DEBUG::2012-12-02 14:47:26,156::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29980::DEBUG::2012-12-02 14:47:26,156::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29980::DEBUG::2012-12-02 14:47:26,156::task::957::TaskManager.Task::(_decref) Task=`19243aae-0a8b-4396-a5db-db7ce3c7eb7a`::ref 0 aborting False >Thread-29981::DEBUG::2012-12-02 14:47:26,159::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29981::DEBUG::2012-12-02 14:47:26,159::task::568::TaskManager.Task::(_updateState) Task=`61c11718-ec00-4094-8e4a-644721a5eb49`::moving from state init -> state preparing >Thread-29981::INFO::2012-12-02 14:47:26,159::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29981::DEBUG::2012-12-02 14:47:26,159::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2e5bc3b2-2234-4c26-9335-b904346a56f7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29981::DEBUG::2012-12-02 14:47:26,160::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29981::DEBUG::2012-12-02 14:47:26,160::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29981::DEBUG::2012-12-02 14:47:26,160::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`2e5bc3b2-2234-4c26-9335-b904346a56f7`::Granted request >Thread-29981::DEBUG::2012-12-02 14:47:26,160::task::794::TaskManager.Task::(resourceAcquired) Task=`61c11718-ec00-4094-8e4a-644721a5eb49`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29981::DEBUG::2012-12-02 14:47:26,160::task::957::TaskManager.Task::(_decref) Task=`61c11718-ec00-4094-8e4a-644721a5eb49`::ref 1 aborting False >Thread-29981::INFO::2012-12-02 14:47:26,166::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29981::DEBUG::2012-12-02 14:47:26,166::task::1151::TaskManager.Task::(prepare) Task=`61c11718-ec00-4094-8e4a-644721a5eb49`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29981::DEBUG::2012-12-02 14:47:26,166::task::568::TaskManager.Task::(_updateState) Task=`61c11718-ec00-4094-8e4a-644721a5eb49`::moving from state preparing -> state finished >Thread-29981::DEBUG::2012-12-02 14:47:26,166::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29981::DEBUG::2012-12-02 14:47:26,167::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29981::DEBUG::2012-12-02 14:47:26,167::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29981::DEBUG::2012-12-02 14:47:26,167::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29981::DEBUG::2012-12-02 14:47:26,167::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29981::DEBUG::2012-12-02 14:47:26,167::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29981::DEBUG::2012-12-02 14:47:26,167::task::957::TaskManager.Task::(_decref) Task=`61c11718-ec00-4094-8e4a-644721a5eb49`::ref 0 aborting False >Thread-29982::DEBUG::2012-12-02 14:47:26,784::task::568::TaskManager.Task::(_updateState) Task=`e07e4b7b-2757-42f2-b867-0c6a5828e8aa`::moving from state init -> state preparing >Thread-29982::INFO::2012-12-02 14:47:26,784::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29982::INFO::2012-12-02 14:47:26,784::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0079460144043', 'lastCheck': 1354452441.070174, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00387001037598', 'lastCheck': 1354452437.7157431, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00256299972534', 'lastCheck': 1354452437.7730141, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00685381889343', 'lastCheck': 1354452437.724859, 'code': 0, 'valid': True}} >Thread-29982::DEBUG::2012-12-02 14:47:26,784::task::1151::TaskManager.Task::(prepare) Task=`e07e4b7b-2757-42f2-b867-0c6a5828e8aa`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0079460144043', 'lastCheck': 1354452441.070174, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00387001037598', 'lastCheck': 1354452437.7157431, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00256299972534', 'lastCheck': 1354452437.7730141, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00685381889343', 'lastCheck': 1354452437.724859, 'code': 0, 'valid': True}} >Thread-29982::DEBUG::2012-12-02 14:47:26,784::task::568::TaskManager.Task::(_updateState) Task=`e07e4b7b-2757-42f2-b867-0c6a5828e8aa`::moving from state preparing -> state finished >Thread-29982::DEBUG::2012-12-02 14:47:26,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29982::DEBUG::2012-12-02 14:47:26,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29982::DEBUG::2012-12-02 14:47:26,785::task::957::TaskManager.Task::(_decref) Task=`e07e4b7b-2757-42f2-b867-0c6a5828e8aa`::ref 0 aborting False >Thread-29988::DEBUG::2012-12-02 14:47:36,311::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29988::DEBUG::2012-12-02 14:47:36,311::task::568::TaskManager.Task::(_updateState) Task=`ed91a727-6baf-4222-a24b-e3b424abadfc`::moving from state init -> state preparing >Thread-29988::INFO::2012-12-02 14:47:36,311::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29988::INFO::2012-12-02 14:47:36,312::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29988::DEBUG::2012-12-02 14:47:36,312::task::1151::TaskManager.Task::(prepare) Task=`ed91a727-6baf-4222-a24b-e3b424abadfc`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29988::DEBUG::2012-12-02 14:47:36,312::task::568::TaskManager.Task::(_updateState) Task=`ed91a727-6baf-4222-a24b-e3b424abadfc`::moving from state preparing -> state finished >Thread-29988::DEBUG::2012-12-02 14:47:36,312::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29988::DEBUG::2012-12-02 14:47:36,312::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29988::DEBUG::2012-12-02 14:47:36,312::task::957::TaskManager.Task::(_decref) Task=`ed91a727-6baf-4222-a24b-e3b424abadfc`::ref 0 aborting False >Thread-29989::DEBUG::2012-12-02 14:47:36,315::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29989::DEBUG::2012-12-02 14:47:36,315::task::568::TaskManager.Task::(_updateState) Task=`d7908ea4-1804-42ef-b8a4-c4890b81ec52`::moving from state init -> state preparing >Thread-29989::INFO::2012-12-02 14:47:36,315::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29989::DEBUG::2012-12-02 14:47:36,315::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e0bc0393-f7bf-4748-9ad9-f4f6de05f7df`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29989::DEBUG::2012-12-02 14:47:36,315::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29989::DEBUG::2012-12-02 14:47:36,316::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29989::DEBUG::2012-12-02 14:47:36,316::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e0bc0393-f7bf-4748-9ad9-f4f6de05f7df`::Granted request >Thread-29989::DEBUG::2012-12-02 14:47:36,316::task::794::TaskManager.Task::(resourceAcquired) Task=`d7908ea4-1804-42ef-b8a4-c4890b81ec52`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29989::DEBUG::2012-12-02 14:47:36,316::task::957::TaskManager.Task::(_decref) Task=`d7908ea4-1804-42ef-b8a4-c4890b81ec52`::ref 1 aborting False >Thread-29989::INFO::2012-12-02 14:47:36,321::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29989::DEBUG::2012-12-02 14:47:36,321::task::1151::TaskManager.Task::(prepare) Task=`d7908ea4-1804-42ef-b8a4-c4890b81ec52`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29989::DEBUG::2012-12-02 14:47:36,321::task::568::TaskManager.Task::(_updateState) Task=`d7908ea4-1804-42ef-b8a4-c4890b81ec52`::moving from state preparing -> state finished >Thread-29989::DEBUG::2012-12-02 14:47:36,321::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29989::DEBUG::2012-12-02 14:47:36,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29989::DEBUG::2012-12-02 14:47:36,321::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29989::DEBUG::2012-12-02 14:47:36,321::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29989::DEBUG::2012-12-02 14:47:36,321::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29989::DEBUG::2012-12-02 14:47:36,321::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29989::DEBUG::2012-12-02 14:47:36,321::task::957::TaskManager.Task::(_decref) Task=`d7908ea4-1804-42ef-b8a4-c4890b81ec52`::ref 0 aborting False >Thread-29990::DEBUG::2012-12-02 14:47:36,979::task::568::TaskManager.Task::(_updateState) Task=`ae4b2bc4-a388-4314-b9d8-45dc4db0c767`::moving from state init -> state preparing >Thread-29990::INFO::2012-12-02 14:47:36,979::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29990::INFO::2012-12-02 14:47:36,979::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00774788856506', 'lastCheck': 1354452451.0890629, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00389099121094', 'lastCheck': 1354452447.736053, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183391571045', 'lastCheck': 1354452447.7824309, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0086498260498', 'lastCheck': 1354452447.7438271, 'code': 0, 'valid': True}} >Thread-29990::DEBUG::2012-12-02 14:47:36,979::task::1151::TaskManager.Task::(prepare) Task=`ae4b2bc4-a388-4314-b9d8-45dc4db0c767`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00774788856506', 'lastCheck': 1354452451.0890629, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00389099121094', 'lastCheck': 1354452447.736053, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00183391571045', 'lastCheck': 1354452447.7824309, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0086498260498', 'lastCheck': 1354452447.7438271, 'code': 0, 'valid': True}} >Thread-29990::DEBUG::2012-12-02 14:47:36,979::task::568::TaskManager.Task::(_updateState) Task=`ae4b2bc4-a388-4314-b9d8-45dc4db0c767`::moving from state preparing -> state finished >Thread-29990::DEBUG::2012-12-02 14:47:36,980::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29990::DEBUG::2012-12-02 14:47:36,980::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29990::DEBUG::2012-12-02 14:47:36,980::task::957::TaskManager.Task::(_decref) Task=`ae4b2bc4-a388-4314-b9d8-45dc4db0c767`::ref 0 aborting False >Thread-29996::DEBUG::2012-12-02 14:47:46,448::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29996::DEBUG::2012-12-02 14:47:46,448::task::568::TaskManager.Task::(_updateState) Task=`a960d0bb-f67f-4287-8b67-923fa13d3e4f`::moving from state init -> state preparing >Thread-29996::INFO::2012-12-02 14:47:46,448::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29996::INFO::2012-12-02 14:47:46,448::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29996::DEBUG::2012-12-02 14:47:46,449::task::1151::TaskManager.Task::(prepare) Task=`a960d0bb-f67f-4287-8b67-923fa13d3e4f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-29996::DEBUG::2012-12-02 14:47:46,449::task::568::TaskManager.Task::(_updateState) Task=`a960d0bb-f67f-4287-8b67-923fa13d3e4f`::moving from state preparing -> state finished >Thread-29996::DEBUG::2012-12-02 14:47:46,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29996::DEBUG::2012-12-02 14:47:46,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29996::DEBUG::2012-12-02 14:47:46,449::task::957::TaskManager.Task::(_decref) Task=`a960d0bb-f67f-4287-8b67-923fa13d3e4f`::ref 0 aborting False >Thread-29997::DEBUG::2012-12-02 14:47:46,451::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-29997::DEBUG::2012-12-02 14:47:46,452::task::568::TaskManager.Task::(_updateState) Task=`48d14ed0-fe27-4c45-925c-94660b688e11`::moving from state init -> state preparing >Thread-29997::INFO::2012-12-02 14:47:46,452::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-29997::DEBUG::2012-12-02 14:47:46,452::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`62f96608-2f0f-4fd2-8926-0b24f6a10773`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-29997::DEBUG::2012-12-02 14:47:46,452::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-29997::DEBUG::2012-12-02 14:47:46,452::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-29997::DEBUG::2012-12-02 14:47:46,453::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`62f96608-2f0f-4fd2-8926-0b24f6a10773`::Granted request >Thread-29997::DEBUG::2012-12-02 14:47:46,453::task::794::TaskManager.Task::(resourceAcquired) Task=`48d14ed0-fe27-4c45-925c-94660b688e11`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-29997::DEBUG::2012-12-02 14:47:46,453::task::957::TaskManager.Task::(_decref) Task=`48d14ed0-fe27-4c45-925c-94660b688e11`::ref 1 aborting False >Thread-29997::INFO::2012-12-02 14:47:46,459::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29997::DEBUG::2012-12-02 14:47:46,460::task::1151::TaskManager.Task::(prepare) Task=`48d14ed0-fe27-4c45-925c-94660b688e11`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-29997::DEBUG::2012-12-02 14:47:46,460::task::568::TaskManager.Task::(_updateState) Task=`48d14ed0-fe27-4c45-925c-94660b688e11`::moving from state preparing -> state finished >Thread-29997::DEBUG::2012-12-02 14:47:46,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-29997::DEBUG::2012-12-02 14:47:46,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29997::DEBUG::2012-12-02 14:47:46,460::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-29997::DEBUG::2012-12-02 14:47:46,460::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-29997::DEBUG::2012-12-02 14:47:46,461::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-29997::DEBUG::2012-12-02 14:47:46,461::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-29997::DEBUG::2012-12-02 14:47:46,461::task::957::TaskManager.Task::(_decref) Task=`48d14ed0-fe27-4c45-925c-94660b688e11`::ref 0 aborting False >Thread-29998::DEBUG::2012-12-02 14:47:47,092::task::568::TaskManager.Task::(_updateState) Task=`867fe58e-8d55-4187-8059-eb9156749704`::moving from state init -> state preparing >Thread-29998::INFO::2012-12-02 14:47:47,092::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29998::INFO::2012-12-02 14:47:47,092::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00869297981262', 'lastCheck': 1354452461.1127889, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0035719871521', 'lastCheck': 1354452457.751086, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00270700454712', 'lastCheck': 1354452457.793983, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00769901275635', 'lastCheck': 1354452457.7598591, 'code': 0, 'valid': True}} >Thread-29998::DEBUG::2012-12-02 14:47:47,092::task::1151::TaskManager.Task::(prepare) Task=`867fe58e-8d55-4187-8059-eb9156749704`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00869297981262', 'lastCheck': 1354452461.1127889, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0035719871521', 'lastCheck': 1354452457.751086, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00270700454712', 'lastCheck': 1354452457.793983, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00769901275635', 'lastCheck': 1354452457.7598591, 'code': 0, 'valid': True}} >Thread-29998::DEBUG::2012-12-02 14:47:47,092::task::568::TaskManager.Task::(_updateState) Task=`867fe58e-8d55-4187-8059-eb9156749704`::moving from state preparing -> state finished >Thread-29998::DEBUG::2012-12-02 14:47:47,093::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29998::DEBUG::2012-12-02 14:47:47,093::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-29998::DEBUG::2012-12-02 14:47:47,093::task::957::TaskManager.Task::(_decref) Task=`867fe58e-8d55-4187-8059-eb9156749704`::ref 0 aborting False >Thread-30004::DEBUG::2012-12-02 14:47:56,585::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30004::DEBUG::2012-12-02 14:47:56,586::task::568::TaskManager.Task::(_updateState) Task=`901f8645-c960-463c-9170-a0705e54267d`::moving from state init -> state preparing >Thread-30004::INFO::2012-12-02 14:47:56,586::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30004::INFO::2012-12-02 14:47:56,586::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30004::DEBUG::2012-12-02 14:47:56,586::task::1151::TaskManager.Task::(prepare) Task=`901f8645-c960-463c-9170-a0705e54267d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30004::DEBUG::2012-12-02 14:47:56,586::task::568::TaskManager.Task::(_updateState) Task=`901f8645-c960-463c-9170-a0705e54267d`::moving from state preparing -> state finished >Thread-30004::DEBUG::2012-12-02 14:47:56,587::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30004::DEBUG::2012-12-02 14:47:56,587::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30004::DEBUG::2012-12-02 14:47:56,587::task::957::TaskManager.Task::(_decref) Task=`901f8645-c960-463c-9170-a0705e54267d`::ref 0 aborting False >Thread-30005::DEBUG::2012-12-02 14:47:56,589::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30005::DEBUG::2012-12-02 14:47:56,590::task::568::TaskManager.Task::(_updateState) Task=`51913753-73a1-444d-97cf-16825a134b1b`::moving from state init -> state preparing >Thread-30005::INFO::2012-12-02 14:47:56,590::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30005::DEBUG::2012-12-02 14:47:56,590::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6fc81313-99c4-4af0-a382-39ac87a34a62`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30005::DEBUG::2012-12-02 14:47:56,590::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30005::DEBUG::2012-12-02 14:47:56,590::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30005::DEBUG::2012-12-02 14:47:56,591::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6fc81313-99c4-4af0-a382-39ac87a34a62`::Granted request >Thread-30005::DEBUG::2012-12-02 14:47:56,591::task::794::TaskManager.Task::(resourceAcquired) Task=`51913753-73a1-444d-97cf-16825a134b1b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30005::DEBUG::2012-12-02 14:47:56,591::task::957::TaskManager.Task::(_decref) Task=`51913753-73a1-444d-97cf-16825a134b1b`::ref 1 aborting False >Thread-30005::INFO::2012-12-02 14:47:56,597::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30005::DEBUG::2012-12-02 14:47:56,597::task::1151::TaskManager.Task::(prepare) Task=`51913753-73a1-444d-97cf-16825a134b1b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30005::DEBUG::2012-12-02 14:47:56,597::task::568::TaskManager.Task::(_updateState) Task=`51913753-73a1-444d-97cf-16825a134b1b`::moving from state preparing -> state finished >Thread-30005::DEBUG::2012-12-02 14:47:56,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30005::DEBUG::2012-12-02 14:47:56,597::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30005::DEBUG::2012-12-02 14:47:56,597::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30005::DEBUG::2012-12-02 14:47:56,598::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30005::DEBUG::2012-12-02 14:47:56,598::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30005::DEBUG::2012-12-02 14:47:56,598::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30005::DEBUG::2012-12-02 14:47:56,598::task::957::TaskManager.Task::(_decref) Task=`51913753-73a1-444d-97cf-16825a134b1b`::ref 0 aborting False >Thread-30006::DEBUG::2012-12-02 14:47:57,227::task::568::TaskManager.Task::(_updateState) Task=`e03b483c-3315-464d-ba1c-8b796dac5cb1`::moving from state init -> state preparing >Thread-30006::INFO::2012-12-02 14:47:57,227::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30006::INFO::2012-12-02 14:47:57,227::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00672507286072', 'lastCheck': 1354452471.1275411, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0023341178894', 'lastCheck': 1354452467.7692399, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181078910828', 'lastCheck': 1354452467.802916, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00789403915405', 'lastCheck': 1354452467.7810249, 'code': 0, 'valid': True}} >Thread-30006::DEBUG::2012-12-02 14:47:57,227::task::1151::TaskManager.Task::(prepare) Task=`e03b483c-3315-464d-ba1c-8b796dac5cb1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00672507286072', 'lastCheck': 1354452471.1275411, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0023341178894', 'lastCheck': 1354452467.7692399, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00181078910828', 'lastCheck': 1354452467.802916, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00789403915405', 'lastCheck': 1354452467.7810249, 'code': 0, 'valid': True}} >Thread-30006::DEBUG::2012-12-02 14:47:57,228::task::568::TaskManager.Task::(_updateState) Task=`e03b483c-3315-464d-ba1c-8b796dac5cb1`::moving from state preparing -> state finished >Thread-30006::DEBUG::2012-12-02 14:47:57,228::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30006::DEBUG::2012-12-02 14:47:57,228::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30006::DEBUG::2012-12-02 14:47:57,228::task::957::TaskManager.Task::(_decref) Task=`e03b483c-3315-464d-ba1c-8b796dac5cb1`::ref 0 aborting False >Thread-30012::DEBUG::2012-12-02 14:48:06,723::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30012::DEBUG::2012-12-02 14:48:06,723::task::568::TaskManager.Task::(_updateState) Task=`40712a97-4de6-4965-8a1b-eb3fad146901`::moving from state init -> state preparing >Thread-30012::INFO::2012-12-02 14:48:06,723::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30012::INFO::2012-12-02 14:48:06,724::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30012::DEBUG::2012-12-02 14:48:06,724::task::1151::TaskManager.Task::(prepare) Task=`40712a97-4de6-4965-8a1b-eb3fad146901`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30012::DEBUG::2012-12-02 14:48:06,724::task::568::TaskManager.Task::(_updateState) Task=`40712a97-4de6-4965-8a1b-eb3fad146901`::moving from state preparing -> state finished >Thread-30012::DEBUG::2012-12-02 14:48:06,724::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30012::DEBUG::2012-12-02 14:48:06,724::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30012::DEBUG::2012-12-02 14:48:06,724::task::957::TaskManager.Task::(_decref) Task=`40712a97-4de6-4965-8a1b-eb3fad146901`::ref 0 aborting False >Thread-30013::DEBUG::2012-12-02 14:48:06,726::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30013::DEBUG::2012-12-02 14:48:06,727::task::568::TaskManager.Task::(_updateState) Task=`322b7f42-c157-40a2-9113-1b5f31c2fa87`::moving from state init -> state preparing >Thread-30013::INFO::2012-12-02 14:48:06,727::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30013::DEBUG::2012-12-02 14:48:06,727::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c51cd426-e8fb-4d67-b188-5c7d8aa0bd21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30013::DEBUG::2012-12-02 14:48:06,727::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30013::DEBUG::2012-12-02 14:48:06,727::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30013::DEBUG::2012-12-02 14:48:06,727::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c51cd426-e8fb-4d67-b188-5c7d8aa0bd21`::Granted request >Thread-30013::DEBUG::2012-12-02 14:48:06,727::task::794::TaskManager.Task::(resourceAcquired) Task=`322b7f42-c157-40a2-9113-1b5f31c2fa87`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30013::DEBUG::2012-12-02 14:48:06,728::task::957::TaskManager.Task::(_decref) Task=`322b7f42-c157-40a2-9113-1b5f31c2fa87`::ref 1 aborting False >Thread-30013::INFO::2012-12-02 14:48:06,732::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30013::DEBUG::2012-12-02 14:48:06,732::task::1151::TaskManager.Task::(prepare) Task=`322b7f42-c157-40a2-9113-1b5f31c2fa87`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30013::DEBUG::2012-12-02 14:48:06,732::task::568::TaskManager.Task::(_updateState) Task=`322b7f42-c157-40a2-9113-1b5f31c2fa87`::moving from state preparing -> state finished >Thread-30013::DEBUG::2012-12-02 14:48:06,732::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30013::DEBUG::2012-12-02 14:48:06,732::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30013::DEBUG::2012-12-02 14:48:06,732::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30013::DEBUG::2012-12-02 14:48:06,733::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30013::DEBUG::2012-12-02 14:48:06,733::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30013::DEBUG::2012-12-02 14:48:06,733::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30013::DEBUG::2012-12-02 14:48:06,733::task::957::TaskManager.Task::(_decref) Task=`322b7f42-c157-40a2-9113-1b5f31c2fa87`::ref 0 aborting False >Thread-30014::DEBUG::2012-12-02 14:48:07,370::task::568::TaskManager.Task::(_updateState) Task=`235ece47-80e9-41d6-9173-96147553d8c0`::moving from state init -> state preparing >Thread-30014::INFO::2012-12-02 14:48:07,370::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30014::INFO::2012-12-02 14:48:07,370::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104751586914', 'lastCheck': 1354452481.1475461, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00220894813538', 'lastCheck': 1354452477.7884591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00232815742493', 'lastCheck': 1354452477.8136401, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00891995429993', 'lastCheck': 1354452477.8000481, 'code': 0, 'valid': True}} >Thread-30014::DEBUG::2012-12-02 14:48:07,370::task::1151::TaskManager.Task::(prepare) Task=`235ece47-80e9-41d6-9173-96147553d8c0`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0104751586914', 'lastCheck': 1354452481.1475461, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00220894813538', 'lastCheck': 1354452477.7884591, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00232815742493', 'lastCheck': 1354452477.8136401, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00891995429993', 'lastCheck': 1354452477.8000481, 'code': 0, 'valid': True}} >Thread-30014::DEBUG::2012-12-02 14:48:07,370::task::568::TaskManager.Task::(_updateState) Task=`235ece47-80e9-41d6-9173-96147553d8c0`::moving from state preparing -> state finished >Thread-30014::DEBUG::2012-12-02 14:48:07,371::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30014::DEBUG::2012-12-02 14:48:07,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30014::DEBUG::2012-12-02 14:48:07,371::task::957::TaskManager.Task::(_decref) Task=`235ece47-80e9-41d6-9173-96147553d8c0`::ref 0 aborting False >Thread-30020::DEBUG::2012-12-02 14:48:16,885::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30020::DEBUG::2012-12-02 14:48:16,886::task::568::TaskManager.Task::(_updateState) Task=`a552ab33-7414-4f58-ae87-4edbd855ae99`::moving from state init -> state preparing >Thread-30020::INFO::2012-12-02 14:48:16,886::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30020::INFO::2012-12-02 14:48:16,886::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30020::DEBUG::2012-12-02 14:48:16,886::task::1151::TaskManager.Task::(prepare) Task=`a552ab33-7414-4f58-ae87-4edbd855ae99`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30020::DEBUG::2012-12-02 14:48:16,886::task::568::TaskManager.Task::(_updateState) Task=`a552ab33-7414-4f58-ae87-4edbd855ae99`::moving from state preparing -> state finished >Thread-30020::DEBUG::2012-12-02 14:48:16,887::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30020::DEBUG::2012-12-02 14:48:16,887::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30020::DEBUG::2012-12-02 14:48:16,887::task::957::TaskManager.Task::(_decref) Task=`a552ab33-7414-4f58-ae87-4edbd855ae99`::ref 0 aborting False >Thread-30021::DEBUG::2012-12-02 14:48:16,889::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30021::DEBUG::2012-12-02 14:48:16,890::task::568::TaskManager.Task::(_updateState) Task=`c26bfb03-4104-4616-aa73-64468296fbff`::moving from state init -> state preparing >Thread-30021::INFO::2012-12-02 14:48:16,890::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30021::DEBUG::2012-12-02 14:48:16,890::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1a028eea-1a53-4ccf-a84c-5f698c15e244`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30021::DEBUG::2012-12-02 14:48:16,890::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30021::DEBUG::2012-12-02 14:48:16,890::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30021::DEBUG::2012-12-02 14:48:16,891::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1a028eea-1a53-4ccf-a84c-5f698c15e244`::Granted request >Thread-30021::DEBUG::2012-12-02 14:48:16,891::task::794::TaskManager.Task::(resourceAcquired) Task=`c26bfb03-4104-4616-aa73-64468296fbff`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30021::DEBUG::2012-12-02 14:48:16,891::task::957::TaskManager.Task::(_decref) Task=`c26bfb03-4104-4616-aa73-64468296fbff`::ref 1 aborting False >Thread-30021::INFO::2012-12-02 14:48:16,897::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30021::DEBUG::2012-12-02 14:48:16,897::task::1151::TaskManager.Task::(prepare) Task=`c26bfb03-4104-4616-aa73-64468296fbff`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30021::DEBUG::2012-12-02 14:48:16,897::task::568::TaskManager.Task::(_updateState) Task=`c26bfb03-4104-4616-aa73-64468296fbff`::moving from state preparing -> state finished >Thread-30021::DEBUG::2012-12-02 14:48:16,897::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30021::DEBUG::2012-12-02 14:48:16,897::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30021::DEBUG::2012-12-02 14:48:16,897::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30021::DEBUG::2012-12-02 14:48:16,897::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30021::DEBUG::2012-12-02 14:48:16,897::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30021::DEBUG::2012-12-02 14:48:16,898::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30021::DEBUG::2012-12-02 14:48:16,898::task::957::TaskManager.Task::(_decref) Task=`c26bfb03-4104-4616-aa73-64468296fbff`::ref 0 aborting False >Thread-30022::DEBUG::2012-12-02 14:48:17,430::task::568::TaskManager.Task::(_updateState) Task=`7f12bef7-f160-47e3-87f1-9fda65727708`::moving from state init -> state preparing >Thread-30022::INFO::2012-12-02 14:48:17,431::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30022::INFO::2012-12-02 14:48:17,431::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00772285461426', 'lastCheck': 1354452491.166708, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266599655151', 'lastCheck': 1354452487.807117, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178098678589', 'lastCheck': 1354452487.825772, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0115139484406', 'lastCheck': 1354452487.8228619, 'code': 0, 'valid': True}} >Thread-30022::DEBUG::2012-12-02 14:48:17,431::task::1151::TaskManager.Task::(prepare) Task=`7f12bef7-f160-47e3-87f1-9fda65727708`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00772285461426', 'lastCheck': 1354452491.166708, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00266599655151', 'lastCheck': 1354452487.807117, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00178098678589', 'lastCheck': 1354452487.825772, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0115139484406', 'lastCheck': 1354452487.8228619, 'code': 0, 'valid': True}} >Thread-30022::DEBUG::2012-12-02 14:48:17,431::task::568::TaskManager.Task::(_updateState) Task=`7f12bef7-f160-47e3-87f1-9fda65727708`::moving from state preparing -> state finished >Thread-30022::DEBUG::2012-12-02 14:48:17,431::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30022::DEBUG::2012-12-02 14:48:17,431::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30022::DEBUG::2012-12-02 14:48:17,431::task::957::TaskManager.Task::(_decref) Task=`7f12bef7-f160-47e3-87f1-9fda65727708`::ref 0 aborting False >Thread-30028::DEBUG::2012-12-02 14:48:27,024::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30028::DEBUG::2012-12-02 14:48:27,024::task::568::TaskManager.Task::(_updateState) Task=`f23364ce-ba19-48dd-a4f1-c27968db26fb`::moving from state init -> state preparing >Thread-30028::INFO::2012-12-02 14:48:27,024::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30028::INFO::2012-12-02 14:48:27,024::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30028::DEBUG::2012-12-02 14:48:27,025::task::1151::TaskManager.Task::(prepare) Task=`f23364ce-ba19-48dd-a4f1-c27968db26fb`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30028::DEBUG::2012-12-02 14:48:27,025::task::568::TaskManager.Task::(_updateState) Task=`f23364ce-ba19-48dd-a4f1-c27968db26fb`::moving from state preparing -> state finished >Thread-30028::DEBUG::2012-12-02 14:48:27,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30028::DEBUG::2012-12-02 14:48:27,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30028::DEBUG::2012-12-02 14:48:27,025::task::957::TaskManager.Task::(_decref) Task=`f23364ce-ba19-48dd-a4f1-c27968db26fb`::ref 0 aborting False >Thread-30029::DEBUG::2012-12-02 14:48:27,027::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30029::DEBUG::2012-12-02 14:48:27,028::task::568::TaskManager.Task::(_updateState) Task=`3776ab17-6538-4681-8833-3685cc90274f`::moving from state init -> state preparing >Thread-30029::INFO::2012-12-02 14:48:27,028::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30029::DEBUG::2012-12-02 14:48:27,028::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1a7e2dd9-5b41-473d-8053-c961b5f4d681`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30029::DEBUG::2012-12-02 14:48:27,028::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30029::DEBUG::2012-12-02 14:48:27,028::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30029::DEBUG::2012-12-02 14:48:27,028::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1a7e2dd9-5b41-473d-8053-c961b5f4d681`::Granted request >Thread-30029::DEBUG::2012-12-02 14:48:27,029::task::794::TaskManager.Task::(resourceAcquired) Task=`3776ab17-6538-4681-8833-3685cc90274f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30029::DEBUG::2012-12-02 14:48:27,029::task::957::TaskManager.Task::(_decref) Task=`3776ab17-6538-4681-8833-3685cc90274f`::ref 1 aborting False >Thread-30029::INFO::2012-12-02 14:48:27,033::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30029::DEBUG::2012-12-02 14:48:27,033::task::1151::TaskManager.Task::(prepare) Task=`3776ab17-6538-4681-8833-3685cc90274f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30029::DEBUG::2012-12-02 14:48:27,033::task::568::TaskManager.Task::(_updateState) Task=`3776ab17-6538-4681-8833-3685cc90274f`::moving from state preparing -> state finished >Thread-30029::DEBUG::2012-12-02 14:48:27,034::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30029::DEBUG::2012-12-02 14:48:27,034::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30029::DEBUG::2012-12-02 14:48:27,034::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30029::DEBUG::2012-12-02 14:48:27,034::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30029::DEBUG::2012-12-02 14:48:27,034::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30029::DEBUG::2012-12-02 14:48:27,034::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30029::DEBUG::2012-12-02 14:48:27,034::task::957::TaskManager.Task::(_decref) Task=`3776ab17-6538-4681-8833-3685cc90274f`::ref 0 aborting False >Thread-30030::DEBUG::2012-12-02 14:48:27,602::task::568::TaskManager.Task::(_updateState) Task=`4d3ddae1-08a3-48e1-b648-4b17c944c21f`::moving from state init -> state preparing >Thread-30030::INFO::2012-12-02 14:48:27,602::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30030::INFO::2012-12-02 14:48:27,603::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00739097595215', 'lastCheck': 1354452501.1831441, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00303912162781', 'lastCheck': 1354452497.820713, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00319695472717', 'lastCheck': 1354452497.8431039, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812292098999', 'lastCheck': 1354452497.8434429, 'code': 0, 'valid': True}} >Thread-30030::DEBUG::2012-12-02 14:48:27,603::task::1151::TaskManager.Task::(prepare) Task=`4d3ddae1-08a3-48e1-b648-4b17c944c21f`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00739097595215', 'lastCheck': 1354452501.1831441, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00303912162781', 'lastCheck': 1354452497.820713, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00319695472717', 'lastCheck': 1354452497.8431039, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812292098999', 'lastCheck': 1354452497.8434429, 'code': 0, 'valid': True}} >Thread-30030::DEBUG::2012-12-02 14:48:27,603::task::568::TaskManager.Task::(_updateState) Task=`4d3ddae1-08a3-48e1-b648-4b17c944c21f`::moving from state preparing -> state finished >Thread-30030::DEBUG::2012-12-02 14:48:27,603::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30030::DEBUG::2012-12-02 14:48:27,603::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30030::DEBUG::2012-12-02 14:48:27,603::task::957::TaskManager.Task::(_decref) Task=`4d3ddae1-08a3-48e1-b648-4b17c944c21f`::ref 0 aborting False >Thread-30036::DEBUG::2012-12-02 14:48:37,202::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30036::DEBUG::2012-12-02 14:48:37,202::task::568::TaskManager.Task::(_updateState) Task=`8c40b3c9-1cf7-445d-96b8-26bd72fa3cd9`::moving from state init -> state preparing >Thread-30036::INFO::2012-12-02 14:48:37,202::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30036::INFO::2012-12-02 14:48:37,203::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30036::DEBUG::2012-12-02 14:48:37,203::task::1151::TaskManager.Task::(prepare) Task=`8c40b3c9-1cf7-445d-96b8-26bd72fa3cd9`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30036::DEBUG::2012-12-02 14:48:37,203::task::568::TaskManager.Task::(_updateState) Task=`8c40b3c9-1cf7-445d-96b8-26bd72fa3cd9`::moving from state preparing -> state finished >Thread-30036::DEBUG::2012-12-02 14:48:37,203::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30036::DEBUG::2012-12-02 14:48:37,203::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30036::DEBUG::2012-12-02 14:48:37,203::task::957::TaskManager.Task::(_decref) Task=`8c40b3c9-1cf7-445d-96b8-26bd72fa3cd9`::ref 0 aborting False >Thread-30037::DEBUG::2012-12-02 14:48:37,206::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30037::DEBUG::2012-12-02 14:48:37,206::task::568::TaskManager.Task::(_updateState) Task=`30d38e27-7e92-429d-83b5-8625aa4a6521`::moving from state init -> state preparing >Thread-30037::INFO::2012-12-02 14:48:37,206::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30037::DEBUG::2012-12-02 14:48:37,207::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`705a5263-dbec-45e2-9d35-0d9d8bb25d23`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30037::DEBUG::2012-12-02 14:48:37,207::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30037::DEBUG::2012-12-02 14:48:37,207::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30037::DEBUG::2012-12-02 14:48:37,207::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`705a5263-dbec-45e2-9d35-0d9d8bb25d23`::Granted request >Thread-30037::DEBUG::2012-12-02 14:48:37,207::task::794::TaskManager.Task::(resourceAcquired) Task=`30d38e27-7e92-429d-83b5-8625aa4a6521`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30037::DEBUG::2012-12-02 14:48:37,207::task::957::TaskManager.Task::(_decref) Task=`30d38e27-7e92-429d-83b5-8625aa4a6521`::ref 1 aborting False >Thread-30037::INFO::2012-12-02 14:48:37,214::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30037::DEBUG::2012-12-02 14:48:37,214::task::1151::TaskManager.Task::(prepare) Task=`30d38e27-7e92-429d-83b5-8625aa4a6521`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30037::DEBUG::2012-12-02 14:48:37,214::task::568::TaskManager.Task::(_updateState) Task=`30d38e27-7e92-429d-83b5-8625aa4a6521`::moving from state preparing -> state finished >Thread-30037::DEBUG::2012-12-02 14:48:37,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30037::DEBUG::2012-12-02 14:48:37,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30037::DEBUG::2012-12-02 14:48:37,215::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30037::DEBUG::2012-12-02 14:48:37,215::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30037::DEBUG::2012-12-02 14:48:37,215::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30037::DEBUG::2012-12-02 14:48:37,215::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30037::DEBUG::2012-12-02 14:48:37,215::task::957::TaskManager.Task::(_decref) Task=`30d38e27-7e92-429d-83b5-8625aa4a6521`::ref 0 aborting False >Thread-30038::DEBUG::2012-12-02 14:48:37,781::task::568::TaskManager.Task::(_updateState) Task=`da00154c-0b4c-4468-b015-885d6880018d`::moving from state init -> state preparing >Thread-30038::INFO::2012-12-02 14:48:37,781::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30038::INFO::2012-12-02 14:48:37,781::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00717902183533', 'lastCheck': 1354452511.1985099, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00199794769287', 'lastCheck': 1354452507.835664, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00220394134521', 'lastCheck': 1354452507.86169, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828003883362', 'lastCheck': 1354452507.8676939, 'code': 0, 'valid': True}} >Thread-30038::DEBUG::2012-12-02 14:48:37,781::task::1151::TaskManager.Task::(prepare) Task=`da00154c-0b4c-4468-b015-885d6880018d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00717902183533', 'lastCheck': 1354452511.1985099, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00199794769287', 'lastCheck': 1354452507.835664, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00220394134521', 'lastCheck': 1354452507.86169, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00828003883362', 'lastCheck': 1354452507.8676939, 'code': 0, 'valid': True}} >Thread-30038::DEBUG::2012-12-02 14:48:37,782::task::568::TaskManager.Task::(_updateState) Task=`da00154c-0b4c-4468-b015-885d6880018d`::moving from state preparing -> state finished >Thread-30038::DEBUG::2012-12-02 14:48:37,782::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30038::DEBUG::2012-12-02 14:48:37,782::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30038::DEBUG::2012-12-02 14:48:37,782::task::957::TaskManager.Task::(_decref) Task=`da00154c-0b4c-4468-b015-885d6880018d`::ref 0 aborting False >Thread-30044::DEBUG::2012-12-02 14:48:47,348::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30044::DEBUG::2012-12-02 14:48:47,349::task::568::TaskManager.Task::(_updateState) Task=`f0707e8c-439e-4064-bb23-005c2efd5ca1`::moving from state init -> state preparing >Thread-30044::INFO::2012-12-02 14:48:47,349::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30044::INFO::2012-12-02 14:48:47,349::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30044::DEBUG::2012-12-02 14:48:47,349::task::1151::TaskManager.Task::(prepare) Task=`f0707e8c-439e-4064-bb23-005c2efd5ca1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30044::DEBUG::2012-12-02 14:48:47,349::task::568::TaskManager.Task::(_updateState) Task=`f0707e8c-439e-4064-bb23-005c2efd5ca1`::moving from state preparing -> state finished >Thread-30044::DEBUG::2012-12-02 14:48:47,349::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30044::DEBUG::2012-12-02 14:48:47,350::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30044::DEBUG::2012-12-02 14:48:47,350::task::957::TaskManager.Task::(_decref) Task=`f0707e8c-439e-4064-bb23-005c2efd5ca1`::ref 0 aborting False >Thread-30045::DEBUG::2012-12-02 14:48:47,352::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30045::DEBUG::2012-12-02 14:48:47,352::task::568::TaskManager.Task::(_updateState) Task=`2ce84086-49eb-4d13-a73a-82ecfd6f70ad`::moving from state init -> state preparing >Thread-30045::INFO::2012-12-02 14:48:47,353::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30045::DEBUG::2012-12-02 14:48:47,353::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`93018c46-3d7f-429a-aa4b-c4a2b38c2aaf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30045::DEBUG::2012-12-02 14:48:47,353::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30045::DEBUG::2012-12-02 14:48:47,353::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30045::DEBUG::2012-12-02 14:48:47,353::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`93018c46-3d7f-429a-aa4b-c4a2b38c2aaf`::Granted request >Thread-30045::DEBUG::2012-12-02 14:48:47,354::task::794::TaskManager.Task::(resourceAcquired) Task=`2ce84086-49eb-4d13-a73a-82ecfd6f70ad`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30045::DEBUG::2012-12-02 14:48:47,354::task::957::TaskManager.Task::(_decref) Task=`2ce84086-49eb-4d13-a73a-82ecfd6f70ad`::ref 1 aborting False >Thread-30045::INFO::2012-12-02 14:48:47,359::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30045::DEBUG::2012-12-02 14:48:47,359::task::1151::TaskManager.Task::(prepare) Task=`2ce84086-49eb-4d13-a73a-82ecfd6f70ad`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30045::DEBUG::2012-12-02 14:48:47,359::task::568::TaskManager.Task::(_updateState) Task=`2ce84086-49eb-4d13-a73a-82ecfd6f70ad`::moving from state preparing -> state finished >Thread-30045::DEBUG::2012-12-02 14:48:47,359::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30045::DEBUG::2012-12-02 14:48:47,359::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30045::DEBUG::2012-12-02 14:48:47,359::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30045::DEBUG::2012-12-02 14:48:47,359::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30045::DEBUG::2012-12-02 14:48:47,359::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30045::DEBUG::2012-12-02 14:48:47,359::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30045::DEBUG::2012-12-02 14:48:47,359::task::957::TaskManager.Task::(_decref) Task=`2ce84086-49eb-4d13-a73a-82ecfd6f70ad`::ref 0 aborting False >Thread-30046::DEBUG::2012-12-02 14:48:47,933::task::568::TaskManager.Task::(_updateState) Task=`67d9fe1e-d464-42b5-b441-344c05199ce5`::moving from state init -> state preparing >Thread-30046::INFO::2012-12-02 14:48:47,933::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30046::INFO::2012-12-02 14:48:47,933::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00807309150696', 'lastCheck': 1354452521.220608, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00199389457703', 'lastCheck': 1354452527.863641, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199699401855', 'lastCheck': 1354452527.8871551, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.114732980728', 'lastCheck': 1354452517.9914131, 'code': 0, 'valid': True}} >Thread-30046::DEBUG::2012-12-02 14:48:47,933::task::1151::TaskManager.Task::(prepare) Task=`67d9fe1e-d464-42b5-b441-344c05199ce5`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00807309150696', 'lastCheck': 1354452521.220608, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00199389457703', 'lastCheck': 1354452527.863641, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00199699401855', 'lastCheck': 1354452527.8871551, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.114732980728', 'lastCheck': 1354452517.9914131, 'code': 0, 'valid': True}} >Thread-30046::DEBUG::2012-12-02 14:48:47,933::task::568::TaskManager.Task::(_updateState) Task=`67d9fe1e-d464-42b5-b441-344c05199ce5`::moving from state preparing -> state finished >Thread-30046::DEBUG::2012-12-02 14:48:47,934::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30046::DEBUG::2012-12-02 14:48:47,934::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30046::DEBUG::2012-12-02 14:48:47,934::task::957::TaskManager.Task::(_decref) Task=`67d9fe1e-d464-42b5-b441-344c05199ce5`::ref 0 aborting False >Thread-30052::DEBUG::2012-12-02 14:48:57,452::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30052::DEBUG::2012-12-02 14:48:57,453::task::568::TaskManager.Task::(_updateState) Task=`cfb606e7-2e38-4d49-afaa-479f0bc05925`::moving from state init -> state preparing >Thread-30052::INFO::2012-12-02 14:48:57,453::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30052::INFO::2012-12-02 14:48:57,453::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30052::DEBUG::2012-12-02 14:48:57,453::task::1151::TaskManager.Task::(prepare) Task=`cfb606e7-2e38-4d49-afaa-479f0bc05925`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30052::DEBUG::2012-12-02 14:48:57,453::task::568::TaskManager.Task::(_updateState) Task=`cfb606e7-2e38-4d49-afaa-479f0bc05925`::moving from state preparing -> state finished >Thread-30052::DEBUG::2012-12-02 14:48:57,453::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30052::DEBUG::2012-12-02 14:48:57,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30052::DEBUG::2012-12-02 14:48:57,454::task::957::TaskManager.Task::(_decref) Task=`cfb606e7-2e38-4d49-afaa-479f0bc05925`::ref 0 aborting False >Thread-30053::DEBUG::2012-12-02 14:48:57,456::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30053::DEBUG::2012-12-02 14:48:57,457::task::568::TaskManager.Task::(_updateState) Task=`203b9297-5c02-4b75-a698-d096f895a1c6`::moving from state init -> state preparing >Thread-30053::INFO::2012-12-02 14:48:57,457::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30053::DEBUG::2012-12-02 14:48:57,457::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8375ab03-1cc4-4975-bbb7-4afec7249252`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30053::DEBUG::2012-12-02 14:48:57,457::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30053::DEBUG::2012-12-02 14:48:57,457::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30053::DEBUG::2012-12-02 14:48:57,457::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8375ab03-1cc4-4975-bbb7-4afec7249252`::Granted request >Thread-30053::DEBUG::2012-12-02 14:48:57,458::task::794::TaskManager.Task::(resourceAcquired) Task=`203b9297-5c02-4b75-a698-d096f895a1c6`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30053::DEBUG::2012-12-02 14:48:57,458::task::957::TaskManager.Task::(_decref) Task=`203b9297-5c02-4b75-a698-d096f895a1c6`::ref 1 aborting False >Thread-30053::INFO::2012-12-02 14:48:57,464::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30053::DEBUG::2012-12-02 14:48:57,464::task::1151::TaskManager.Task::(prepare) Task=`203b9297-5c02-4b75-a698-d096f895a1c6`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30053::DEBUG::2012-12-02 14:48:57,465::task::568::TaskManager.Task::(_updateState) Task=`203b9297-5c02-4b75-a698-d096f895a1c6`::moving from state preparing -> state finished >Thread-30053::DEBUG::2012-12-02 14:48:57,465::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30053::DEBUG::2012-12-02 14:48:57,465::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30053::DEBUG::2012-12-02 14:48:57,465::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30053::DEBUG::2012-12-02 14:48:57,465::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30053::DEBUG::2012-12-02 14:48:57,465::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30053::DEBUG::2012-12-02 14:48:57,465::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30053::DEBUG::2012-12-02 14:48:57,466::task::957::TaskManager.Task::(_decref) Task=`203b9297-5c02-4b75-a698-d096f895a1c6`::ref 0 aborting False >Thread-30054::DEBUG::2012-12-02 14:48:58,055::task::568::TaskManager.Task::(_updateState) Task=`ca562d12-ce73-43ef-a153-fe450c58a182`::moving from state init -> state preparing >Thread-30054::INFO::2012-12-02 14:48:58,056::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30054::INFO::2012-12-02 14:48:58,056::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00771188735962', 'lastCheck': 1354452531.2373481, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00364685058594', 'lastCheck': 1354452537.8798211, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00174880027771', 'lastCheck': 1354452537.8977721, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0100309848785', 'lastCheck': 1354452538.0307009, 'code': 0, 'valid': True}} >Thread-30054::DEBUG::2012-12-02 14:48:58,056::task::1151::TaskManager.Task::(prepare) Task=`ca562d12-ce73-43ef-a153-fe450c58a182`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00771188735962', 'lastCheck': 1354452531.2373481, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00364685058594', 'lastCheck': 1354452537.8798211, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00174880027771', 'lastCheck': 1354452537.8977721, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0100309848785', 'lastCheck': 1354452538.0307009, 'code': 0, 'valid': True}} >Thread-30054::DEBUG::2012-12-02 14:48:58,056::task::568::TaskManager.Task::(_updateState) Task=`ca562d12-ce73-43ef-a153-fe450c58a182`::moving from state preparing -> state finished >Thread-30054::DEBUG::2012-12-02 14:48:58,056::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30054::DEBUG::2012-12-02 14:48:58,056::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30054::DEBUG::2012-12-02 14:48:58,056::task::957::TaskManager.Task::(_decref) Task=`ca562d12-ce73-43ef-a153-fe450c58a182`::ref 0 aborting False >Thread-30060::DEBUG::2012-12-02 14:49:07,590::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30060::DEBUG::2012-12-02 14:49:07,590::task::568::TaskManager.Task::(_updateState) Task=`535013bb-9ef9-429d-8321-03e9a8eafd3c`::moving from state init -> state preparing >Thread-30060::INFO::2012-12-02 14:49:07,590::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30060::INFO::2012-12-02 14:49:07,591::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30060::DEBUG::2012-12-02 14:49:07,591::task::1151::TaskManager.Task::(prepare) Task=`535013bb-9ef9-429d-8321-03e9a8eafd3c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30060::DEBUG::2012-12-02 14:49:07,591::task::568::TaskManager.Task::(_updateState) Task=`535013bb-9ef9-429d-8321-03e9a8eafd3c`::moving from state preparing -> state finished >Thread-30060::DEBUG::2012-12-02 14:49:07,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30060::DEBUG::2012-12-02 14:49:07,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30060::DEBUG::2012-12-02 14:49:07,591::task::957::TaskManager.Task::(_decref) Task=`535013bb-9ef9-429d-8321-03e9a8eafd3c`::ref 0 aborting False >Thread-30061::DEBUG::2012-12-02 14:49:07,594::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30061::DEBUG::2012-12-02 14:49:07,594::task::568::TaskManager.Task::(_updateState) Task=`662655cb-2649-4b27-9b16-018eb8099c3d`::moving from state init -> state preparing >Thread-30061::INFO::2012-12-02 14:49:07,594::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30061::DEBUG::2012-12-02 14:49:07,595::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0140b6b3-8505-40f3-bb1b-a01eaf315b51`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30061::DEBUG::2012-12-02 14:49:07,595::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30061::DEBUG::2012-12-02 14:49:07,595::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30061::DEBUG::2012-12-02 14:49:07,595::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0140b6b3-8505-40f3-bb1b-a01eaf315b51`::Granted request >Thread-30061::DEBUG::2012-12-02 14:49:07,595::task::794::TaskManager.Task::(resourceAcquired) Task=`662655cb-2649-4b27-9b16-018eb8099c3d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30061::DEBUG::2012-12-02 14:49:07,595::task::957::TaskManager.Task::(_decref) Task=`662655cb-2649-4b27-9b16-018eb8099c3d`::ref 1 aborting False >Thread-30061::INFO::2012-12-02 14:49:07,600::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30061::DEBUG::2012-12-02 14:49:07,600::task::1151::TaskManager.Task::(prepare) Task=`662655cb-2649-4b27-9b16-018eb8099c3d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30061::DEBUG::2012-12-02 14:49:07,600::task::568::TaskManager.Task::(_updateState) Task=`662655cb-2649-4b27-9b16-018eb8099c3d`::moving from state preparing -> state finished >Thread-30061::DEBUG::2012-12-02 14:49:07,600::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30061::DEBUG::2012-12-02 14:49:07,600::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30061::DEBUG::2012-12-02 14:49:07,600::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30061::DEBUG::2012-12-02 14:49:07,601::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30061::DEBUG::2012-12-02 14:49:07,601::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30061::DEBUG::2012-12-02 14:49:07,601::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30061::DEBUG::2012-12-02 14:49:07,601::task::957::TaskManager.Task::(_decref) Task=`662655cb-2649-4b27-9b16-018eb8099c3d`::ref 0 aborting False >Thread-30062::DEBUG::2012-12-02 14:49:08,168::task::568::TaskManager.Task::(_updateState) Task=`58bf0a99-fba0-42c5-92fc-e44c6fa55093`::moving from state init -> state preparing >Thread-30062::INFO::2012-12-02 14:49:08,168::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30062::INFO::2012-12-02 14:49:08,168::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0147149562836', 'lastCheck': 1354452541.260983, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00179696083069', 'lastCheck': 1354452547.8909831, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00160884857178', 'lastCheck': 1354452547.9057219, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0497720241547', 'lastCheck': 1354452548.0889721, 'code': 0, 'valid': True}} >Thread-30062::DEBUG::2012-12-02 14:49:08,169::task::1151::TaskManager.Task::(prepare) Task=`58bf0a99-fba0-42c5-92fc-e44c6fa55093`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0147149562836', 'lastCheck': 1354452541.260983, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00179696083069', 'lastCheck': 1354452547.8909831, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00160884857178', 'lastCheck': 1354452547.9057219, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0497720241547', 'lastCheck': 1354452548.0889721, 'code': 0, 'valid': True}} >Thread-30062::DEBUG::2012-12-02 14:49:08,169::task::568::TaskManager.Task::(_updateState) Task=`58bf0a99-fba0-42c5-92fc-e44c6fa55093`::moving from state preparing -> state finished >Thread-30062::DEBUG::2012-12-02 14:49:08,169::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30062::DEBUG::2012-12-02 14:49:08,169::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30062::DEBUG::2012-12-02 14:49:08,169::task::957::TaskManager.Task::(_decref) Task=`58bf0a99-fba0-42c5-92fc-e44c6fa55093`::ref 0 aborting False >Thread-30068::DEBUG::2012-12-02 14:49:17,728::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30068::DEBUG::2012-12-02 14:49:17,728::task::568::TaskManager.Task::(_updateState) Task=`7fd44eb0-7c9c-4ed3-8bb2-b6db309670e0`::moving from state init -> state preparing >Thread-30068::INFO::2012-12-02 14:49:17,728::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30068::INFO::2012-12-02 14:49:17,728::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30068::DEBUG::2012-12-02 14:49:17,729::task::1151::TaskManager.Task::(prepare) Task=`7fd44eb0-7c9c-4ed3-8bb2-b6db309670e0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30068::DEBUG::2012-12-02 14:49:17,729::task::568::TaskManager.Task::(_updateState) Task=`7fd44eb0-7c9c-4ed3-8bb2-b6db309670e0`::moving from state preparing -> state finished >Thread-30068::DEBUG::2012-12-02 14:49:17,729::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30068::DEBUG::2012-12-02 14:49:17,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30068::DEBUG::2012-12-02 14:49:17,729::task::957::TaskManager.Task::(_decref) Task=`7fd44eb0-7c9c-4ed3-8bb2-b6db309670e0`::ref 0 aborting False >Thread-30069::DEBUG::2012-12-02 14:49:17,732::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30069::DEBUG::2012-12-02 14:49:17,732::task::568::TaskManager.Task::(_updateState) Task=`da7494ca-521b-4a9a-9610-37de5e15ab36`::moving from state init -> state preparing >Thread-30069::INFO::2012-12-02 14:49:17,732::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30069::DEBUG::2012-12-02 14:49:17,732::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`12ec451c-5ef9-4bb8-ac30-d89edf248060`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30069::DEBUG::2012-12-02 14:49:17,732::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30069::DEBUG::2012-12-02 14:49:17,733::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30069::DEBUG::2012-12-02 14:49:17,733::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`12ec451c-5ef9-4bb8-ac30-d89edf248060`::Granted request >Thread-30069::DEBUG::2012-12-02 14:49:17,733::task::794::TaskManager.Task::(resourceAcquired) Task=`da7494ca-521b-4a9a-9610-37de5e15ab36`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30069::DEBUG::2012-12-02 14:49:17,733::task::957::TaskManager.Task::(_decref) Task=`da7494ca-521b-4a9a-9610-37de5e15ab36`::ref 1 aborting False >Thread-30069::INFO::2012-12-02 14:49:17,738::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30069::DEBUG::2012-12-02 14:49:17,738::task::1151::TaskManager.Task::(prepare) Task=`da7494ca-521b-4a9a-9610-37de5e15ab36`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30069::DEBUG::2012-12-02 14:49:17,739::task::568::TaskManager.Task::(_updateState) Task=`da7494ca-521b-4a9a-9610-37de5e15ab36`::moving from state preparing -> state finished >Thread-30069::DEBUG::2012-12-02 14:49:17,739::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30069::DEBUG::2012-12-02 14:49:17,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30069::DEBUG::2012-12-02 14:49:17,739::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30069::DEBUG::2012-12-02 14:49:17,739::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30069::DEBUG::2012-12-02 14:49:17,739::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30069::DEBUG::2012-12-02 14:49:17,739::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30069::DEBUG::2012-12-02 14:49:17,739::task::957::TaskManager.Task::(_decref) Task=`da7494ca-521b-4a9a-9610-37de5e15ab36`::ref 0 aborting False >Thread-30070::DEBUG::2012-12-02 14:49:18,306::task::568::TaskManager.Task::(_updateState) Task=`ca917f0b-9178-494b-9e00-6bb5387f3aab`::moving from state init -> state preparing >Thread-30070::INFO::2012-12-02 14:49:18,306::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30070::INFO::2012-12-02 14:49:18,306::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00868201255798', 'lastCheck': 1354452551.2784319, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0031418800354', 'lastCheck': 1354452557.904057, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00196599960327', 'lastCheck': 1354452557.9165101, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00322294235229', 'lastCheck': 1354452558.100312, 'code': 0, 'valid': True}} >Thread-30070::DEBUG::2012-12-02 14:49:18,306::task::1151::TaskManager.Task::(prepare) Task=`ca917f0b-9178-494b-9e00-6bb5387f3aab`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00868201255798', 'lastCheck': 1354452551.2784319, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0031418800354', 'lastCheck': 1354452557.904057, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00196599960327', 'lastCheck': 1354452557.9165101, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00322294235229', 'lastCheck': 1354452558.100312, 'code': 0, 'valid': True}} >Thread-30070::DEBUG::2012-12-02 14:49:18,306::task::568::TaskManager.Task::(_updateState) Task=`ca917f0b-9178-494b-9e00-6bb5387f3aab`::moving from state preparing -> state finished >Thread-30070::DEBUG::2012-12-02 14:49:18,306::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30070::DEBUG::2012-12-02 14:49:18,307::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30070::DEBUG::2012-12-02 14:49:18,307::task::957::TaskManager.Task::(_decref) Task=`ca917f0b-9178-494b-9e00-6bb5387f3aab`::ref 0 aborting False >Thread-30076::DEBUG::2012-12-02 14:49:27,882::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30076::DEBUG::2012-12-02 14:49:27,883::task::568::TaskManager.Task::(_updateState) Task=`df1e94d0-05dd-4d2a-84a6-be3f352840b3`::moving from state init -> state preparing >Thread-30076::INFO::2012-12-02 14:49:27,883::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30076::INFO::2012-12-02 14:49:27,883::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30076::DEBUG::2012-12-02 14:49:27,883::task::1151::TaskManager.Task::(prepare) Task=`df1e94d0-05dd-4d2a-84a6-be3f352840b3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30076::DEBUG::2012-12-02 14:49:27,884::task::568::TaskManager.Task::(_updateState) Task=`df1e94d0-05dd-4d2a-84a6-be3f352840b3`::moving from state preparing -> state finished >Thread-30076::DEBUG::2012-12-02 14:49:27,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30076::DEBUG::2012-12-02 14:49:27,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30076::DEBUG::2012-12-02 14:49:27,884::task::957::TaskManager.Task::(_decref) Task=`df1e94d0-05dd-4d2a-84a6-be3f352840b3`::ref 0 aborting False >Thread-30077::DEBUG::2012-12-02 14:49:27,886::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30077::DEBUG::2012-12-02 14:49:27,887::task::568::TaskManager.Task::(_updateState) Task=`c6decec9-03ef-4422-a38a-9173ffa27a22`::moving from state init -> state preparing >Thread-30077::INFO::2012-12-02 14:49:27,887::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30077::DEBUG::2012-12-02 14:49:27,887::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`54d4ed08-8c3b-49c3-b951-03439fb5e6a5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30077::DEBUG::2012-12-02 14:49:27,887::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30077::DEBUG::2012-12-02 14:49:27,887::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30077::DEBUG::2012-12-02 14:49:27,887::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`54d4ed08-8c3b-49c3-b951-03439fb5e6a5`::Granted request >Thread-30077::DEBUG::2012-12-02 14:49:27,887::task::794::TaskManager.Task::(resourceAcquired) Task=`c6decec9-03ef-4422-a38a-9173ffa27a22`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30077::DEBUG::2012-12-02 14:49:27,888::task::957::TaskManager.Task::(_decref) Task=`c6decec9-03ef-4422-a38a-9173ffa27a22`::ref 1 aborting False >Thread-30077::INFO::2012-12-02 14:49:27,892::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30077::DEBUG::2012-12-02 14:49:27,892::task::1151::TaskManager.Task::(prepare) Task=`c6decec9-03ef-4422-a38a-9173ffa27a22`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30077::DEBUG::2012-12-02 14:49:27,892::task::568::TaskManager.Task::(_updateState) Task=`c6decec9-03ef-4422-a38a-9173ffa27a22`::moving from state preparing -> state finished >Thread-30077::DEBUG::2012-12-02 14:49:27,892::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30077::DEBUG::2012-12-02 14:49:27,892::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30077::DEBUG::2012-12-02 14:49:27,892::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30077::DEBUG::2012-12-02 14:49:27,892::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30077::DEBUG::2012-12-02 14:49:27,893::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30077::DEBUG::2012-12-02 14:49:27,893::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30077::DEBUG::2012-12-02 14:49:27,893::task::957::TaskManager.Task::(_decref) Task=`c6decec9-03ef-4422-a38a-9173ffa27a22`::ref 0 aborting False >Thread-30078::DEBUG::2012-12-02 14:49:28,612::task::568::TaskManager.Task::(_updateState) Task=`8ed8ccb7-9477-4432-bc23-1e015f3c5e8a`::moving from state init -> state preparing >Thread-30078::INFO::2012-12-02 14:49:28,612::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30078::INFO::2012-12-02 14:49:28,612::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0067310333252', 'lastCheck': 1354452561.294049, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00212407112122', 'lastCheck': 1354452567.9172029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177311897278', 'lastCheck': 1354452567.9271049, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00937795639038', 'lastCheck': 1354452568.1180711, 'code': 0, 'valid': True}} >Thread-30078::DEBUG::2012-12-02 14:49:28,613::task::1151::TaskManager.Task::(prepare) Task=`8ed8ccb7-9477-4432-bc23-1e015f3c5e8a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0067310333252', 'lastCheck': 1354452561.294049, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00212407112122', 'lastCheck': 1354452567.9172029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177311897278', 'lastCheck': 1354452567.9271049, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00937795639038', 'lastCheck': 1354452568.1180711, 'code': 0, 'valid': True}} >Thread-30078::DEBUG::2012-12-02 14:49:28,613::task::568::TaskManager.Task::(_updateState) Task=`8ed8ccb7-9477-4432-bc23-1e015f3c5e8a`::moving from state preparing -> state finished >Thread-30078::DEBUG::2012-12-02 14:49:28,613::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30078::DEBUG::2012-12-02 14:49:28,613::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30078::DEBUG::2012-12-02 14:49:28,613::task::957::TaskManager.Task::(_decref) Task=`8ed8ccb7-9477-4432-bc23-1e015f3c5e8a`::ref 0 aborting False >Thread-30084::DEBUG::2012-12-02 14:49:38,020::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30084::DEBUG::2012-12-02 14:49:38,020::task::568::TaskManager.Task::(_updateState) Task=`12b8af7b-1d9c-4e13-a981-1605f6ca7418`::moving from state init -> state preparing >Thread-30084::INFO::2012-12-02 14:49:38,021::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30084::INFO::2012-12-02 14:49:38,021::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30084::DEBUG::2012-12-02 14:49:38,021::task::1151::TaskManager.Task::(prepare) Task=`12b8af7b-1d9c-4e13-a981-1605f6ca7418`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30084::DEBUG::2012-12-02 14:49:38,021::task::568::TaskManager.Task::(_updateState) Task=`12b8af7b-1d9c-4e13-a981-1605f6ca7418`::moving from state preparing -> state finished >Thread-30084::DEBUG::2012-12-02 14:49:38,021::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30084::DEBUG::2012-12-02 14:49:38,021::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30084::DEBUG::2012-12-02 14:49:38,021::task::957::TaskManager.Task::(_decref) Task=`12b8af7b-1d9c-4e13-a981-1605f6ca7418`::ref 0 aborting False >Thread-30085::DEBUG::2012-12-02 14:49:38,024::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30085::DEBUG::2012-12-02 14:49:38,024::task::568::TaskManager.Task::(_updateState) Task=`8af90898-17ea-4132-aeec-a00cc0c1ec69`::moving from state init -> state preparing >Thread-30085::INFO::2012-12-02 14:49:38,024::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30085::DEBUG::2012-12-02 14:49:38,025::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`90ae7f98-ccf5-43da-9799-db947d56a387`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30085::DEBUG::2012-12-02 14:49:38,025::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30085::DEBUG::2012-12-02 14:49:38,025::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30085::DEBUG::2012-12-02 14:49:38,025::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`90ae7f98-ccf5-43da-9799-db947d56a387`::Granted request >Thread-30085::DEBUG::2012-12-02 14:49:38,025::task::794::TaskManager.Task::(resourceAcquired) Task=`8af90898-17ea-4132-aeec-a00cc0c1ec69`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30085::DEBUG::2012-12-02 14:49:38,026::task::957::TaskManager.Task::(_decref) Task=`8af90898-17ea-4132-aeec-a00cc0c1ec69`::ref 1 aborting False >Thread-30085::INFO::2012-12-02 14:49:38,030::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30085::DEBUG::2012-12-02 14:49:38,030::task::1151::TaskManager.Task::(prepare) Task=`8af90898-17ea-4132-aeec-a00cc0c1ec69`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30085::DEBUG::2012-12-02 14:49:38,030::task::568::TaskManager.Task::(_updateState) Task=`8af90898-17ea-4132-aeec-a00cc0c1ec69`::moving from state preparing -> state finished >Thread-30085::DEBUG::2012-12-02 14:49:38,030::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30085::DEBUG::2012-12-02 14:49:38,030::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30085::DEBUG::2012-12-02 14:49:38,030::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30085::DEBUG::2012-12-02 14:49:38,030::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30085::DEBUG::2012-12-02 14:49:38,031::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30085::DEBUG::2012-12-02 14:49:38,031::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30085::DEBUG::2012-12-02 14:49:38,031::task::957::TaskManager.Task::(_decref) Task=`8af90898-17ea-4132-aeec-a00cc0c1ec69`::ref 0 aborting False >Thread-30086::DEBUG::2012-12-02 14:49:38,696::task::568::TaskManager.Task::(_updateState) Task=`1902644e-4f9e-4bd8-8ef2-00bfa324fdb1`::moving from state init -> state preparing >Thread-30086::INFO::2012-12-02 14:49:38,696::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30086::INFO::2012-12-02 14:49:38,697::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0065131187439', 'lastCheck': 1354452571.3081739, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00500798225403', 'lastCheck': 1354452577.9369099, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00239896774292', 'lastCheck': 1354452577.9418631, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00904607772827', 'lastCheck': 1354452578.1356809, 'code': 0, 'valid': True}} >Thread-30086::DEBUG::2012-12-02 14:49:38,697::task::1151::TaskManager.Task::(prepare) Task=`1902644e-4f9e-4bd8-8ef2-00bfa324fdb1`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0065131187439', 'lastCheck': 1354452571.3081739, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00500798225403', 'lastCheck': 1354452577.9369099, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00239896774292', 'lastCheck': 1354452577.9418631, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00904607772827', 'lastCheck': 1354452578.1356809, 'code': 0, 'valid': True}} >Thread-30086::DEBUG::2012-12-02 14:49:38,697::task::568::TaskManager.Task::(_updateState) Task=`1902644e-4f9e-4bd8-8ef2-00bfa324fdb1`::moving from state preparing -> state finished >Thread-30086::DEBUG::2012-12-02 14:49:38,697::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30086::DEBUG::2012-12-02 14:49:38,697::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30086::DEBUG::2012-12-02 14:49:38,697::task::957::TaskManager.Task::(_decref) Task=`1902644e-4f9e-4bd8-8ef2-00bfa324fdb1`::ref 0 aborting False >Thread-30092::DEBUG::2012-12-02 14:49:48,123::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30092::DEBUG::2012-12-02 14:49:48,124::task::568::TaskManager.Task::(_updateState) Task=`e3b472db-2076-43b8-a0f3-7c2b1d1153c1`::moving from state init -> state preparing >Thread-30092::INFO::2012-12-02 14:49:48,124::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30092::INFO::2012-12-02 14:49:48,124::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30092::DEBUG::2012-12-02 14:49:48,124::task::1151::TaskManager.Task::(prepare) Task=`e3b472db-2076-43b8-a0f3-7c2b1d1153c1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30092::DEBUG::2012-12-02 14:49:48,124::task::568::TaskManager.Task::(_updateState) Task=`e3b472db-2076-43b8-a0f3-7c2b1d1153c1`::moving from state preparing -> state finished >Thread-30092::DEBUG::2012-12-02 14:49:48,125::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30092::DEBUG::2012-12-02 14:49:48,125::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30092::DEBUG::2012-12-02 14:49:48,125::task::957::TaskManager.Task::(_decref) Task=`e3b472db-2076-43b8-a0f3-7c2b1d1153c1`::ref 0 aborting False >Thread-30093::DEBUG::2012-12-02 14:49:48,127::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30093::DEBUG::2012-12-02 14:49:48,128::task::568::TaskManager.Task::(_updateState) Task=`6fe2398c-2c4f-4cbb-9366-62ac6daf35a4`::moving from state init -> state preparing >Thread-30093::INFO::2012-12-02 14:49:48,128::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30093::DEBUG::2012-12-02 14:49:48,128::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f9b88739-4aab-4d9d-a05a-487374976680`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30093::DEBUG::2012-12-02 14:49:48,128::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30093::DEBUG::2012-12-02 14:49:48,128::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30093::DEBUG::2012-12-02 14:49:48,129::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`f9b88739-4aab-4d9d-a05a-487374976680`::Granted request >Thread-30093::DEBUG::2012-12-02 14:49:48,129::task::794::TaskManager.Task::(resourceAcquired) Task=`6fe2398c-2c4f-4cbb-9366-62ac6daf35a4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30093::DEBUG::2012-12-02 14:49:48,129::task::957::TaskManager.Task::(_decref) Task=`6fe2398c-2c4f-4cbb-9366-62ac6daf35a4`::ref 1 aborting False >Thread-30093::INFO::2012-12-02 14:49:48,135::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30093::DEBUG::2012-12-02 14:49:48,136::task::1151::TaskManager.Task::(prepare) Task=`6fe2398c-2c4f-4cbb-9366-62ac6daf35a4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30093::DEBUG::2012-12-02 14:49:48,136::task::568::TaskManager.Task::(_updateState) Task=`6fe2398c-2c4f-4cbb-9366-62ac6daf35a4`::moving from state preparing -> state finished >Thread-30093::DEBUG::2012-12-02 14:49:48,136::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30093::DEBUG::2012-12-02 14:49:48,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30093::DEBUG::2012-12-02 14:49:48,139::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30093::DEBUG::2012-12-02 14:49:48,140::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30093::DEBUG::2012-12-02 14:49:48,142::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30093::DEBUG::2012-12-02 14:49:48,143::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30093::DEBUG::2012-12-02 14:49:48,143::task::957::TaskManager.Task::(_decref) Task=`6fe2398c-2c4f-4cbb-9366-62ac6daf35a4`::ref 0 aborting False >Thread-30094::DEBUG::2012-12-02 14:49:48,763::task::568::TaskManager.Task::(_updateState) Task=`bb6cf84a-c4c8-4ed4-b339-8d99f733a729`::moving from state init -> state preparing >Thread-30094::INFO::2012-12-02 14:49:48,764::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30094::INFO::2012-12-02 14:49:48,764::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00989294052124', 'lastCheck': 1354452581.329515, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00286102294922', 'lastCheck': 1354452587.952631, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00154304504395', 'lastCheck': 1354452587.9532161, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00853085517883', 'lastCheck': 1354452588.157887, 'code': 0, 'valid': True}} >Thread-30094::DEBUG::2012-12-02 14:49:48,764::task::1151::TaskManager.Task::(prepare) Task=`bb6cf84a-c4c8-4ed4-b339-8d99f733a729`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00989294052124', 'lastCheck': 1354452581.329515, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00286102294922', 'lastCheck': 1354452587.952631, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00154304504395', 'lastCheck': 1354452587.9532161, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00853085517883', 'lastCheck': 1354452588.157887, 'code': 0, 'valid': True}} >Thread-30094::DEBUG::2012-12-02 14:49:48,764::task::568::TaskManager.Task::(_updateState) Task=`bb6cf84a-c4c8-4ed4-b339-8d99f733a729`::moving from state preparing -> state finished >Thread-30094::DEBUG::2012-12-02 14:49:48,764::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30094::DEBUG::2012-12-02 14:49:48,764::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30094::DEBUG::2012-12-02 14:49:48,764::task::957::TaskManager.Task::(_decref) Task=`bb6cf84a-c4c8-4ed4-b339-8d99f733a729`::ref 0 aborting False >Thread-30100::DEBUG::2012-12-02 14:49:58,274::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30100::DEBUG::2012-12-02 14:49:58,275::task::568::TaskManager.Task::(_updateState) Task=`9bef0d16-d24c-4499-8be6-e8119b1f08c5`::moving from state init -> state preparing >Thread-30100::INFO::2012-12-02 14:49:58,275::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30100::INFO::2012-12-02 14:49:58,275::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30100::DEBUG::2012-12-02 14:49:58,275::task::1151::TaskManager.Task::(prepare) Task=`9bef0d16-d24c-4499-8be6-e8119b1f08c5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30100::DEBUG::2012-12-02 14:49:58,275::task::568::TaskManager.Task::(_updateState) Task=`9bef0d16-d24c-4499-8be6-e8119b1f08c5`::moving from state preparing -> state finished >Thread-30100::DEBUG::2012-12-02 14:49:58,275::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30100::DEBUG::2012-12-02 14:49:58,275::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30100::DEBUG::2012-12-02 14:49:58,275::task::957::TaskManager.Task::(_decref) Task=`9bef0d16-d24c-4499-8be6-e8119b1f08c5`::ref 0 aborting False >Thread-30101::DEBUG::2012-12-02 14:49:58,278::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30101::DEBUG::2012-12-02 14:49:58,278::task::568::TaskManager.Task::(_updateState) Task=`cb9bcf06-5ec8-4682-a2e6-c2a0f0195bb1`::moving from state init -> state preparing >Thread-30101::INFO::2012-12-02 14:49:58,278::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30101::DEBUG::2012-12-02 14:49:58,279::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a147391-be31-4e56-b80d-91c80e86420f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30101::DEBUG::2012-12-02 14:49:58,279::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30101::DEBUG::2012-12-02 14:49:58,279::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30101::DEBUG::2012-12-02 14:49:58,279::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0a147391-be31-4e56-b80d-91c80e86420f`::Granted request >Thread-30101::DEBUG::2012-12-02 14:49:58,279::task::794::TaskManager.Task::(resourceAcquired) Task=`cb9bcf06-5ec8-4682-a2e6-c2a0f0195bb1`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30101::DEBUG::2012-12-02 14:49:58,279::task::957::TaskManager.Task::(_decref) Task=`cb9bcf06-5ec8-4682-a2e6-c2a0f0195bb1`::ref 1 aborting False >Thread-30101::INFO::2012-12-02 14:49:58,286::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30101::DEBUG::2012-12-02 14:49:58,286::task::1151::TaskManager.Task::(prepare) Task=`cb9bcf06-5ec8-4682-a2e6-c2a0f0195bb1`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30101::DEBUG::2012-12-02 14:49:58,286::task::568::TaskManager.Task::(_updateState) Task=`cb9bcf06-5ec8-4682-a2e6-c2a0f0195bb1`::moving from state preparing -> state finished >Thread-30101::DEBUG::2012-12-02 14:49:58,286::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30101::DEBUG::2012-12-02 14:49:58,286::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30101::DEBUG::2012-12-02 14:49:58,286::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30101::DEBUG::2012-12-02 14:49:58,286::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30101::DEBUG::2012-12-02 14:49:58,286::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30101::DEBUG::2012-12-02 14:49:58,286::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30101::DEBUG::2012-12-02 14:49:58,286::task::957::TaskManager.Task::(_decref) Task=`cb9bcf06-5ec8-4682-a2e6-c2a0f0195bb1`::ref 0 aborting False >Thread-30102::DEBUG::2012-12-02 14:49:58,865::task::568::TaskManager.Task::(_updateState) Task=`cdea4c74-12ef-4e3f-bb54-761cb52cfa66`::moving from state init -> state preparing >Thread-30102::INFO::2012-12-02 14:49:58,865::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30102::INFO::2012-12-02 14:49:58,865::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00793600082397', 'lastCheck': 1354452591.3467109, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00343489646912', 'lastCheck': 1354452597.9732399, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00162386894226', 'lastCheck': 1354452597.974133, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00989103317261', 'lastCheck': 1354452598.1775529, 'code': 0, 'valid': True}} >Thread-30102::DEBUG::2012-12-02 14:49:58,865::task::1151::TaskManager.Task::(prepare) Task=`cdea4c74-12ef-4e3f-bb54-761cb52cfa66`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00793600082397', 'lastCheck': 1354452591.3467109, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00343489646912', 'lastCheck': 1354452597.9732399, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00162386894226', 'lastCheck': 1354452597.974133, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00989103317261', 'lastCheck': 1354452598.1775529, 'code': 0, 'valid': True}} >Thread-30102::DEBUG::2012-12-02 14:49:58,865::task::568::TaskManager.Task::(_updateState) Task=`cdea4c74-12ef-4e3f-bb54-761cb52cfa66`::moving from state preparing -> state finished >Thread-30102::DEBUG::2012-12-02 14:49:58,865::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30102::DEBUG::2012-12-02 14:49:58,866::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30102::DEBUG::2012-12-02 14:49:58,866::task::957::TaskManager.Task::(_decref) Task=`cdea4c74-12ef-4e3f-bb54-761cb52cfa66`::ref 0 aborting False >Thread-30108::DEBUG::2012-12-02 14:50:08,407::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30108::DEBUG::2012-12-02 14:50:08,407::task::568::TaskManager.Task::(_updateState) Task=`3e7c7273-8780-488e-b7ec-d09309f23e7e`::moving from state init -> state preparing >Thread-30108::INFO::2012-12-02 14:50:08,407::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30108::INFO::2012-12-02 14:50:08,407::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30108::DEBUG::2012-12-02 14:50:08,408::task::1151::TaskManager.Task::(prepare) Task=`3e7c7273-8780-488e-b7ec-d09309f23e7e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30108::DEBUG::2012-12-02 14:50:08,408::task::568::TaskManager.Task::(_updateState) Task=`3e7c7273-8780-488e-b7ec-d09309f23e7e`::moving from state preparing -> state finished >Thread-30108::DEBUG::2012-12-02 14:50:08,408::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30108::DEBUG::2012-12-02 14:50:08,408::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30108::DEBUG::2012-12-02 14:50:08,408::task::957::TaskManager.Task::(_decref) Task=`3e7c7273-8780-488e-b7ec-d09309f23e7e`::ref 0 aborting False >Thread-30109::DEBUG::2012-12-02 14:50:08,411::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30109::DEBUG::2012-12-02 14:50:08,411::task::568::TaskManager.Task::(_updateState) Task=`0f6dd29e-11db-4a57-b2e2-e938e9220ee3`::moving from state init -> state preparing >Thread-30109::INFO::2012-12-02 14:50:08,411::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30109::DEBUG::2012-12-02 14:50:08,411::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cf37b642-a476-4d31-b561-cf803f1402cc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30109::DEBUG::2012-12-02 14:50:08,412::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30109::DEBUG::2012-12-02 14:50:08,412::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30109::DEBUG::2012-12-02 14:50:08,412::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cf37b642-a476-4d31-b561-cf803f1402cc`::Granted request >Thread-30109::DEBUG::2012-12-02 14:50:08,412::task::794::TaskManager.Task::(resourceAcquired) Task=`0f6dd29e-11db-4a57-b2e2-e938e9220ee3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30109::DEBUG::2012-12-02 14:50:08,412::task::957::TaskManager.Task::(_decref) Task=`0f6dd29e-11db-4a57-b2e2-e938e9220ee3`::ref 1 aborting False >Thread-30109::INFO::2012-12-02 14:50:08,418::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30109::DEBUG::2012-12-02 14:50:08,418::task::1151::TaskManager.Task::(prepare) Task=`0f6dd29e-11db-4a57-b2e2-e938e9220ee3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30109::DEBUG::2012-12-02 14:50:08,418::task::568::TaskManager.Task::(_updateState) Task=`0f6dd29e-11db-4a57-b2e2-e938e9220ee3`::moving from state preparing -> state finished >Thread-30109::DEBUG::2012-12-02 14:50:08,418::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30109::DEBUG::2012-12-02 14:50:08,418::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30109::DEBUG::2012-12-02 14:50:08,418::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30109::DEBUG::2012-12-02 14:50:08,418::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30109::DEBUG::2012-12-02 14:50:08,418::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30109::DEBUG::2012-12-02 14:50:08,418::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30109::DEBUG::2012-12-02 14:50:08,418::task::957::TaskManager.Task::(_decref) Task=`0f6dd29e-11db-4a57-b2e2-e938e9220ee3`::ref 0 aborting False >Thread-30110::DEBUG::2012-12-02 14:50:09,002::task::568::TaskManager.Task::(_updateState) Task=`c39b4bd5-2bd9-4179-ad17-0a658a27039d`::moving from state init -> state preparing >Thread-30110::INFO::2012-12-02 14:50:09,002::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30110::INFO::2012-12-02 14:50:09,002::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00843214988708', 'lastCheck': 1354452601.364747, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00376796722412', 'lastCheck': 1354452607.9954779, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00459098815918', 'lastCheck': 1354452608.0034449, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00906682014465', 'lastCheck': 1354452608.1969559, 'code': 0, 'valid': True}} >Thread-30110::DEBUG::2012-12-02 14:50:09,003::task::1151::TaskManager.Task::(prepare) Task=`c39b4bd5-2bd9-4179-ad17-0a658a27039d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00843214988708', 'lastCheck': 1354452601.364747, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00376796722412', 'lastCheck': 1354452607.9954779, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00459098815918', 'lastCheck': 1354452608.0034449, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00906682014465', 'lastCheck': 1354452608.1969559, 'code': 0, 'valid': True}} >Thread-30110::DEBUG::2012-12-02 14:50:09,003::task::568::TaskManager.Task::(_updateState) Task=`c39b4bd5-2bd9-4179-ad17-0a658a27039d`::moving from state preparing -> state finished >Thread-30110::DEBUG::2012-12-02 14:50:09,003::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30110::DEBUG::2012-12-02 14:50:09,003::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30110::DEBUG::2012-12-02 14:50:09,003::task::957::TaskManager.Task::(_decref) Task=`c39b4bd5-2bd9-4179-ad17-0a658a27039d`::ref 0 aborting False >Thread-30116::DEBUG::2012-12-02 14:50:18,553::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30116::DEBUG::2012-12-02 14:50:18,553::task::568::TaskManager.Task::(_updateState) Task=`ef809aa7-8572-4906-aa00-d18d9cf92829`::moving from state init -> state preparing >Thread-30116::INFO::2012-12-02 14:50:18,553::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30116::INFO::2012-12-02 14:50:18,554::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30116::DEBUG::2012-12-02 14:50:18,554::task::1151::TaskManager.Task::(prepare) Task=`ef809aa7-8572-4906-aa00-d18d9cf92829`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30116::DEBUG::2012-12-02 14:50:18,554::task::568::TaskManager.Task::(_updateState) Task=`ef809aa7-8572-4906-aa00-d18d9cf92829`::moving from state preparing -> state finished >Thread-30116::DEBUG::2012-12-02 14:50:18,554::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30116::DEBUG::2012-12-02 14:50:18,554::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30116::DEBUG::2012-12-02 14:50:18,554::task::957::TaskManager.Task::(_decref) Task=`ef809aa7-8572-4906-aa00-d18d9cf92829`::ref 0 aborting False >Thread-30117::DEBUG::2012-12-02 14:50:18,557::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30117::DEBUG::2012-12-02 14:50:18,557::task::568::TaskManager.Task::(_updateState) Task=`a7d1c76f-5ed9-4215-bdd3-8cf2b2039fdc`::moving from state init -> state preparing >Thread-30117::INFO::2012-12-02 14:50:18,557::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30117::DEBUG::2012-12-02 14:50:18,557::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9eafa816-6fb5-4589-8403-e8537e05c29e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30117::DEBUG::2012-12-02 14:50:18,557::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30117::DEBUG::2012-12-02 14:50:18,558::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30117::DEBUG::2012-12-02 14:50:18,558::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9eafa816-6fb5-4589-8403-e8537e05c29e`::Granted request >Thread-30117::DEBUG::2012-12-02 14:50:18,558::task::794::TaskManager.Task::(resourceAcquired) Task=`a7d1c76f-5ed9-4215-bdd3-8cf2b2039fdc`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30117::DEBUG::2012-12-02 14:50:18,558::task::957::TaskManager.Task::(_decref) Task=`a7d1c76f-5ed9-4215-bdd3-8cf2b2039fdc`::ref 1 aborting False >Thread-30117::INFO::2012-12-02 14:50:18,562::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30117::DEBUG::2012-12-02 14:50:18,562::task::1151::TaskManager.Task::(prepare) Task=`a7d1c76f-5ed9-4215-bdd3-8cf2b2039fdc`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30117::DEBUG::2012-12-02 14:50:18,562::task::568::TaskManager.Task::(_updateState) Task=`a7d1c76f-5ed9-4215-bdd3-8cf2b2039fdc`::moving from state preparing -> state finished >Thread-30117::DEBUG::2012-12-02 14:50:18,563::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30117::DEBUG::2012-12-02 14:50:18,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30117::DEBUG::2012-12-02 14:50:18,563::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30117::DEBUG::2012-12-02 14:50:18,563::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30117::DEBUG::2012-12-02 14:50:18,563::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30117::DEBUG::2012-12-02 14:50:18,563::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30117::DEBUG::2012-12-02 14:50:18,563::task::957::TaskManager.Task::(_decref) Task=`a7d1c76f-5ed9-4215-bdd3-8cf2b2039fdc`::ref 0 aborting False >Thread-30118::DEBUG::2012-12-02 14:50:19,146::task::568::TaskManager.Task::(_updateState) Task=`22104cec-f286-4b1b-a44d-213b582e9bf2`::moving from state init -> state preparing >Thread-30118::INFO::2012-12-02 14:50:19,146::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30118::INFO::2012-12-02 14:50:19,146::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00822782516479', 'lastCheck': 1354452611.3824279, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00556206703186', 'lastCheck': 1354452618.0147879, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00192189216614', 'lastCheck': 1354452618.0345459, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00744414329529', 'lastCheck': 1354452618.2132421, 'code': 0, 'valid': True}} >Thread-30118::DEBUG::2012-12-02 14:50:19,146::task::1151::TaskManager.Task::(prepare) Task=`22104cec-f286-4b1b-a44d-213b582e9bf2`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00822782516479', 'lastCheck': 1354452611.3824279, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00556206703186', 'lastCheck': 1354452618.0147879, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00192189216614', 'lastCheck': 1354452618.0345459, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00744414329529', 'lastCheck': 1354452618.2132421, 'code': 0, 'valid': True}} >Thread-30118::DEBUG::2012-12-02 14:50:19,146::task::568::TaskManager.Task::(_updateState) Task=`22104cec-f286-4b1b-a44d-213b582e9bf2`::moving from state preparing -> state finished >Thread-30118::DEBUG::2012-12-02 14:50:19,146::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30118::DEBUG::2012-12-02 14:50:19,147::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30118::DEBUG::2012-12-02 14:50:19,147::task::957::TaskManager.Task::(_decref) Task=`22104cec-f286-4b1b-a44d-213b582e9bf2`::ref 0 aborting False >Thread-30124::DEBUG::2012-12-02 14:50:28,758::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30124::DEBUG::2012-12-02 14:50:28,758::task::568::TaskManager.Task::(_updateState) Task=`324b1056-7c34-4b02-a83a-5625d0e2ba68`::moving from state init -> state preparing >Thread-30124::INFO::2012-12-02 14:50:28,758::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30124::INFO::2012-12-02 14:50:28,758::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30124::DEBUG::2012-12-02 14:50:28,759::task::1151::TaskManager.Task::(prepare) Task=`324b1056-7c34-4b02-a83a-5625d0e2ba68`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30124::DEBUG::2012-12-02 14:50:28,759::task::568::TaskManager.Task::(_updateState) Task=`324b1056-7c34-4b02-a83a-5625d0e2ba68`::moving from state preparing -> state finished >Thread-30124::DEBUG::2012-12-02 14:50:28,759::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30124::DEBUG::2012-12-02 14:50:28,759::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30124::DEBUG::2012-12-02 14:50:28,759::task::957::TaskManager.Task::(_decref) Task=`324b1056-7c34-4b02-a83a-5625d0e2ba68`::ref 0 aborting False >Thread-30125::DEBUG::2012-12-02 14:50:28,762::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30125::DEBUG::2012-12-02 14:50:28,762::task::568::TaskManager.Task::(_updateState) Task=`5b3994b5-3275-40e4-94f6-d4d2233537ee`::moving from state init -> state preparing >Thread-30125::INFO::2012-12-02 14:50:28,762::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30125::DEBUG::2012-12-02 14:50:28,763::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9c6586e5-250f-463c-8117-403a624e96e9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30125::DEBUG::2012-12-02 14:50:28,763::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30125::DEBUG::2012-12-02 14:50:28,763::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30125::DEBUG::2012-12-02 14:50:28,763::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9c6586e5-250f-463c-8117-403a624e96e9`::Granted request >Thread-30125::DEBUG::2012-12-02 14:50:28,763::task::794::TaskManager.Task::(resourceAcquired) Task=`5b3994b5-3275-40e4-94f6-d4d2233537ee`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30125::DEBUG::2012-12-02 14:50:28,763::task::957::TaskManager.Task::(_decref) Task=`5b3994b5-3275-40e4-94f6-d4d2233537ee`::ref 1 aborting False >Thread-30125::INFO::2012-12-02 14:50:28,768::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30125::DEBUG::2012-12-02 14:50:28,768::task::1151::TaskManager.Task::(prepare) Task=`5b3994b5-3275-40e4-94f6-d4d2233537ee`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30125::DEBUG::2012-12-02 14:50:28,768::task::568::TaskManager.Task::(_updateState) Task=`5b3994b5-3275-40e4-94f6-d4d2233537ee`::moving from state preparing -> state finished >Thread-30125::DEBUG::2012-12-02 14:50:28,769::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30125::DEBUG::2012-12-02 14:50:28,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30125::DEBUG::2012-12-02 14:50:28,769::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30125::DEBUG::2012-12-02 14:50:28,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30125::DEBUG::2012-12-02 14:50:28,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30125::DEBUG::2012-12-02 14:50:28,769::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30125::DEBUG::2012-12-02 14:50:28,769::task::957::TaskManager.Task::(_decref) Task=`5b3994b5-3275-40e4-94f6-d4d2233537ee`::ref 0 aborting False >Thread-30126::DEBUG::2012-12-02 14:50:29,231::task::568::TaskManager.Task::(_updateState) Task=`85b9e66c-761a-403f-acd6-d177a66e738a`::moving from state init -> state preparing >Thread-30126::INFO::2012-12-02 14:50:29,231::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30126::INFO::2012-12-02 14:50:29,232::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00749897956848', 'lastCheck': 1354452621.397989, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00147700309753', 'lastCheck': 1354452628.0269561, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238800048828', 'lastCheck': 1354452628.046011, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00723600387573', 'lastCheck': 1354452628.228085, 'code': 0, 'valid': True}} >Thread-30126::DEBUG::2012-12-02 14:50:29,232::task::1151::TaskManager.Task::(prepare) Task=`85b9e66c-761a-403f-acd6-d177a66e738a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00749897956848', 'lastCheck': 1354452621.397989, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00147700309753', 'lastCheck': 1354452628.0269561, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00238800048828', 'lastCheck': 1354452628.046011, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00723600387573', 'lastCheck': 1354452628.228085, 'code': 0, 'valid': True}} >Thread-30126::DEBUG::2012-12-02 14:50:29,232::task::568::TaskManager.Task::(_updateState) Task=`85b9e66c-761a-403f-acd6-d177a66e738a`::moving from state preparing -> state finished >Thread-30126::DEBUG::2012-12-02 14:50:29,232::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30126::DEBUG::2012-12-02 14:50:29,232::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30126::DEBUG::2012-12-02 14:50:29,232::task::957::TaskManager.Task::(_decref) Task=`85b9e66c-761a-403f-acd6-d177a66e738a`::ref 0 aborting False >Thread-30132::DEBUG::2012-12-02 14:50:38,904::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30132::DEBUG::2012-12-02 14:50:38,905::task::568::TaskManager.Task::(_updateState) Task=`1506c61d-b009-4706-86aa-e63c5f1c0eb2`::moving from state init -> state preparing >Thread-30132::INFO::2012-12-02 14:50:38,905::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30132::INFO::2012-12-02 14:50:38,905::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30132::DEBUG::2012-12-02 14:50:38,905::task::1151::TaskManager.Task::(prepare) Task=`1506c61d-b009-4706-86aa-e63c5f1c0eb2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30132::DEBUG::2012-12-02 14:50:38,906::task::568::TaskManager.Task::(_updateState) Task=`1506c61d-b009-4706-86aa-e63c5f1c0eb2`::moving from state preparing -> state finished >Thread-30132::DEBUG::2012-12-02 14:50:38,906::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30132::DEBUG::2012-12-02 14:50:38,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30132::DEBUG::2012-12-02 14:50:38,906::task::957::TaskManager.Task::(_decref) Task=`1506c61d-b009-4706-86aa-e63c5f1c0eb2`::ref 0 aborting False >Thread-30133::DEBUG::2012-12-02 14:50:38,909::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30133::DEBUG::2012-12-02 14:50:38,909::task::568::TaskManager.Task::(_updateState) Task=`6b931d99-ceea-4e57-adf9-cfd74f0004a4`::moving from state init -> state preparing >Thread-30133::INFO::2012-12-02 14:50:38,910::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30133::DEBUG::2012-12-02 14:50:38,910::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0c78df43-5e66-4f6c-8956-367107791e41`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30133::DEBUG::2012-12-02 14:50:38,910::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30133::DEBUG::2012-12-02 14:50:38,910::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30133::DEBUG::2012-12-02 14:50:38,910::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`0c78df43-5e66-4f6c-8956-367107791e41`::Granted request >Thread-30133::DEBUG::2012-12-02 14:50:38,911::task::794::TaskManager.Task::(resourceAcquired) Task=`6b931d99-ceea-4e57-adf9-cfd74f0004a4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30133::DEBUG::2012-12-02 14:50:38,911::task::957::TaskManager.Task::(_decref) Task=`6b931d99-ceea-4e57-adf9-cfd74f0004a4`::ref 1 aborting False >Thread-30133::INFO::2012-12-02 14:50:38,917::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30133::DEBUG::2012-12-02 14:50:38,917::task::1151::TaskManager.Task::(prepare) Task=`6b931d99-ceea-4e57-adf9-cfd74f0004a4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30133::DEBUG::2012-12-02 14:50:38,917::task::568::TaskManager.Task::(_updateState) Task=`6b931d99-ceea-4e57-adf9-cfd74f0004a4`::moving from state preparing -> state finished >Thread-30133::DEBUG::2012-12-02 14:50:38,917::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30133::DEBUG::2012-12-02 14:50:38,917::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30133::DEBUG::2012-12-02 14:50:38,917::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30133::DEBUG::2012-12-02 14:50:38,917::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30133::DEBUG::2012-12-02 14:50:38,917::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30133::DEBUG::2012-12-02 14:50:38,918::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30133::DEBUG::2012-12-02 14:50:38,918::task::957::TaskManager.Task::(_decref) Task=`6b931d99-ceea-4e57-adf9-cfd74f0004a4`::ref 0 aborting False >Thread-30134::DEBUG::2012-12-02 14:50:39,300::task::568::TaskManager.Task::(_updateState) Task=`c7a9130c-afaf-4847-8538-8e08a2b4e622`::moving from state init -> state preparing >Thread-30134::INFO::2012-12-02 14:50:39,300::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30134::INFO::2012-12-02 14:50:39,301::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00930404663086', 'lastCheck': 1354452631.417336, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00214815139771', 'lastCheck': 1354452638.038259, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00236892700195', 'lastCheck': 1354452638.0574579, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00744295120239', 'lastCheck': 1354452638.2445099, 'code': 0, 'valid': True}} >Thread-30134::DEBUG::2012-12-02 14:50:39,301::task::1151::TaskManager.Task::(prepare) Task=`c7a9130c-afaf-4847-8538-8e08a2b4e622`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00930404663086', 'lastCheck': 1354452631.417336, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00214815139771', 'lastCheck': 1354452638.038259, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00236892700195', 'lastCheck': 1354452638.0574579, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00744295120239', 'lastCheck': 1354452638.2445099, 'code': 0, 'valid': True}} >Thread-30134::DEBUG::2012-12-02 14:50:39,301::task::568::TaskManager.Task::(_updateState) Task=`c7a9130c-afaf-4847-8538-8e08a2b4e622`::moving from state preparing -> state finished >Thread-30134::DEBUG::2012-12-02 14:50:39,301::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30134::DEBUG::2012-12-02 14:50:39,301::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30134::DEBUG::2012-12-02 14:50:39,301::task::957::TaskManager.Task::(_decref) Task=`c7a9130c-afaf-4847-8538-8e08a2b4e622`::ref 0 aborting False >Thread-30140::DEBUG::2012-12-02 14:50:49,018::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30140::DEBUG::2012-12-02 14:50:49,018::task::568::TaskManager.Task::(_updateState) Task=`02970877-b1fb-4b68-9789-86dbf8bda058`::moving from state init -> state preparing >Thread-30140::INFO::2012-12-02 14:50:49,018::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30140::INFO::2012-12-02 14:50:49,018::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30140::DEBUG::2012-12-02 14:50:49,018::task::1151::TaskManager.Task::(prepare) Task=`02970877-b1fb-4b68-9789-86dbf8bda058`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30140::DEBUG::2012-12-02 14:50:49,019::task::568::TaskManager.Task::(_updateState) Task=`02970877-b1fb-4b68-9789-86dbf8bda058`::moving from state preparing -> state finished >Thread-30140::DEBUG::2012-12-02 14:50:49,019::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30140::DEBUG::2012-12-02 14:50:49,019::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30140::DEBUG::2012-12-02 14:50:49,019::task::957::TaskManager.Task::(_decref) Task=`02970877-b1fb-4b68-9789-86dbf8bda058`::ref 0 aborting False >Thread-30141::DEBUG::2012-12-02 14:50:49,021::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30141::DEBUG::2012-12-02 14:50:49,022::task::568::TaskManager.Task::(_updateState) Task=`0d2e189a-fa9d-439c-b6d9-b17836f03a09`::moving from state init -> state preparing >Thread-30141::INFO::2012-12-02 14:50:49,022::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30141::DEBUG::2012-12-02 14:50:49,022::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a3428fa1-500e-4dca-93f6-ac0c51c840b4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30141::DEBUG::2012-12-02 14:50:49,022::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30141::DEBUG::2012-12-02 14:50:49,023::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30141::DEBUG::2012-12-02 14:50:49,023::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`a3428fa1-500e-4dca-93f6-ac0c51c840b4`::Granted request >Thread-30141::DEBUG::2012-12-02 14:50:49,023::task::794::TaskManager.Task::(resourceAcquired) Task=`0d2e189a-fa9d-439c-b6d9-b17836f03a09`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30141::DEBUG::2012-12-02 14:50:49,023::task::957::TaskManager.Task::(_decref) Task=`0d2e189a-fa9d-439c-b6d9-b17836f03a09`::ref 1 aborting False >Thread-30141::INFO::2012-12-02 14:50:49,028::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30141::DEBUG::2012-12-02 14:50:49,028::task::1151::TaskManager.Task::(prepare) Task=`0d2e189a-fa9d-439c-b6d9-b17836f03a09`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30141::DEBUG::2012-12-02 14:50:49,028::task::568::TaskManager.Task::(_updateState) Task=`0d2e189a-fa9d-439c-b6d9-b17836f03a09`::moving from state preparing -> state finished >Thread-30141::DEBUG::2012-12-02 14:50:49,028::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30141::DEBUG::2012-12-02 14:50:49,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30141::DEBUG::2012-12-02 14:50:49,029::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30141::DEBUG::2012-12-02 14:50:49,029::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30141::DEBUG::2012-12-02 14:50:49,029::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30141::DEBUG::2012-12-02 14:50:49,029::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30141::DEBUG::2012-12-02 14:50:49,029::task::957::TaskManager.Task::(_decref) Task=`0d2e189a-fa9d-439c-b6d9-b17836f03a09`::ref 0 aborting False >Thread-30142::DEBUG::2012-12-02 14:50:49,408::task::568::TaskManager.Task::(_updateState) Task=`7b6f73c9-98cb-4484-bb46-2df24ebbafff`::moving from state init -> state preparing >Thread-30142::INFO::2012-12-02 14:50:49,408::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30142::INFO::2012-12-02 14:50:49,409::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00859594345093', 'lastCheck': 1354452641.439925, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00146508216858', 'lastCheck': 1354452648.0506001, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0021960735321', 'lastCheck': 1354452648.068269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0106370449066', 'lastCheck': 1354452648.2665999, 'code': 0, 'valid': True}} >Thread-30142::DEBUG::2012-12-02 14:50:49,409::task::1151::TaskManager.Task::(prepare) Task=`7b6f73c9-98cb-4484-bb46-2df24ebbafff`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00859594345093', 'lastCheck': 1354452641.439925, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00146508216858', 'lastCheck': 1354452648.0506001, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0021960735321', 'lastCheck': 1354452648.068269, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0106370449066', 'lastCheck': 1354452648.2665999, 'code': 0, 'valid': True}} >Thread-30142::DEBUG::2012-12-02 14:50:49,409::task::568::TaskManager.Task::(_updateState) Task=`7b6f73c9-98cb-4484-bb46-2df24ebbafff`::moving from state preparing -> state finished >Thread-30142::DEBUG::2012-12-02 14:50:49,409::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30142::DEBUG::2012-12-02 14:50:49,409::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30142::DEBUG::2012-12-02 14:50:49,409::task::957::TaskManager.Task::(_decref) Task=`7b6f73c9-98cb-4484-bb46-2df24ebbafff`::ref 0 aborting False >Thread-11884::DEBUG::2012-12-02 14:50:58,068::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:50:58,068::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11884::DEBUG::2012-12-02 14:50:58,069::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 718c4127-5ab7-440b-8d64-7f96ec39fce1' (cwd None) >Thread-11884::DEBUG::2012-12-02 14:50:58,276::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found\n'; <rc> = 5 >Thread-11884::WARNING::2012-12-02 14:50:58,278::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "718c4127-5ab7-440b-8d64-7f96ec39fce1" not found'] >Thread-11884::DEBUG::2012-12-02 14:50:58,278::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11884::DEBUG::2012-12-02 14:50:58,314::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1/718c4127-5ab7-440b-8d64-7f96ec39fce1 >Thread-11884::DEBUG::2012-12-02 14:50:58,314::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11884::DEBUG::2012-12-02 14:50:58,316::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export1', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'ROLE=Regular', 'SDUUID=718c4127-5ab7-440b-8d64-7f96ec39fce1', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=dbfe9e9f548161767524125e2bdcd3b003061cb9'] >Thread-11884::DEBUG::2012-12-02 14:50:58,319::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11884::WARNING::2012-12-02 14:50:58,319::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_imageNS already registered >Thread-11884::WARNING::2012-12-02 14:50:58,319::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 718c4127-5ab7-440b-8d64-7f96ec39fce1_volumeNS already registered >Thread-30209::DEBUG::2012-12-02 14:50:59,297::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30209::DEBUG::2012-12-02 14:50:59,297::task::568::TaskManager.Task::(_updateState) Task=`75dda867-4309-43e3-8108-54a56bc6c859`::moving from state init -> state preparing >Thread-30209::INFO::2012-12-02 14:50:59,297::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30209::INFO::2012-12-02 14:50:59,298::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30209::DEBUG::2012-12-02 14:50:59,298::task::1151::TaskManager.Task::(prepare) Task=`75dda867-4309-43e3-8108-54a56bc6c859`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30209::DEBUG::2012-12-02 14:50:59,298::task::568::TaskManager.Task::(_updateState) Task=`75dda867-4309-43e3-8108-54a56bc6c859`::moving from state preparing -> state finished >Thread-30209::DEBUG::2012-12-02 14:50:59,298::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30209::DEBUG::2012-12-02 14:50:59,298::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30209::DEBUG::2012-12-02 14:50:59,298::task::957::TaskManager.Task::(_decref) Task=`75dda867-4309-43e3-8108-54a56bc6c859`::ref 0 aborting False >Thread-30210::DEBUG::2012-12-02 14:50:59,301::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30210::DEBUG::2012-12-02 14:50:59,301::task::568::TaskManager.Task::(_updateState) Task=`41caca9a-1a9f-49de-8c56-07b3359cb3a3`::moving from state init -> state preparing >Thread-30210::INFO::2012-12-02 14:50:59,301::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30210::DEBUG::2012-12-02 14:50:59,302::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9cf622f2-dc9c-47d3-ae1f-7f536a585825`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30210::DEBUG::2012-12-02 14:50:59,302::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30210::DEBUG::2012-12-02 14:50:59,302::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30210::DEBUG::2012-12-02 14:50:59,302::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9cf622f2-dc9c-47d3-ae1f-7f536a585825`::Granted request >Thread-30210::DEBUG::2012-12-02 14:50:59,302::task::794::TaskManager.Task::(resourceAcquired) Task=`41caca9a-1a9f-49de-8c56-07b3359cb3a3`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30210::DEBUG::2012-12-02 14:50:59,302::task::957::TaskManager.Task::(_decref) Task=`41caca9a-1a9f-49de-8c56-07b3359cb3a3`::ref 1 aborting False >Thread-30210::INFO::2012-12-02 14:50:59,308::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30210::DEBUG::2012-12-02 14:50:59,308::task::1151::TaskManager.Task::(prepare) Task=`41caca9a-1a9f-49de-8c56-07b3359cb3a3`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30210::DEBUG::2012-12-02 14:50:59,308::task::568::TaskManager.Task::(_updateState) Task=`41caca9a-1a9f-49de-8c56-07b3359cb3a3`::moving from state preparing -> state finished >Thread-30210::DEBUG::2012-12-02 14:50:59,308::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30210::DEBUG::2012-12-02 14:50:59,308::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30210::DEBUG::2012-12-02 14:50:59,309::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30210::DEBUG::2012-12-02 14:50:59,309::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30210::DEBUG::2012-12-02 14:50:59,309::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30210::DEBUG::2012-12-02 14:50:59,309::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30210::DEBUG::2012-12-02 14:50:59,309::task::957::TaskManager.Task::(_decref) Task=`41caca9a-1a9f-49de-8c56-07b3359cb3a3`::ref 0 aborting False >Thread-30211::DEBUG::2012-12-02 14:50:59,505::task::568::TaskManager.Task::(_updateState) Task=`040f2e78-8281-439a-97b9-4b4dadebee46`::moving from state init -> state preparing >Thread-30211::INFO::2012-12-02 14:50:59,505::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30211::INFO::2012-12-02 14:50:59,506::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00904512405396', 'lastCheck': 1354452651.4583261, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00293922424316', 'lastCheck': 1354452658.065032, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253891944885', 'lastCheck': 1354452658.3397701, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00820803642273', 'lastCheck': 1354452658.344162, 'code': 0, 'valid': True}} >Thread-30211::DEBUG::2012-12-02 14:50:59,506::task::1151::TaskManager.Task::(prepare) Task=`040f2e78-8281-439a-97b9-4b4dadebee46`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00904512405396', 'lastCheck': 1354452651.4583261, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00293922424316', 'lastCheck': 1354452658.065032, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00253891944885', 'lastCheck': 1354452658.3397701, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00820803642273', 'lastCheck': 1354452658.344162, 'code': 0, 'valid': True}} >Thread-30211::DEBUG::2012-12-02 14:50:59,506::task::568::TaskManager.Task::(_updateState) Task=`040f2e78-8281-439a-97b9-4b4dadebee46`::moving from state preparing -> state finished >Thread-30211::DEBUG::2012-12-02 14:50:59,506::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30211::DEBUG::2012-12-02 14:50:59,506::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30211::DEBUG::2012-12-02 14:50:59,506::task::957::TaskManager.Task::(_decref) Task=`040f2e78-8281-439a-97b9-4b4dadebee46`::ref 0 aborting False >Thread-11821::DEBUG::2012-12-02 14:51:01,458::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:51:01,458::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11821::DEBUG::2012-12-02 14:51:01,459::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 625b62bd-21ae-4068-bc86-657056036298' (cwd None) >Thread-11821::DEBUG::2012-12-02 14:51:01,666::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found\n'; <rc> = 5 >Thread-11821::WARNING::2012-12-02 14:51:01,668::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "625b62bd-21ae-4068-bc86-657056036298" not found'] >Thread-11821::DEBUG::2012-12-02 14:51:01,668::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11821::DEBUG::2012-12-02 14:51:01,703::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4/625b62bd-21ae-4068-bc86-657056036298 >Thread-11821::DEBUG::2012-12-02 14:51:01,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11821::DEBUG::2012-12-02 14:51:01,713::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data4', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'ROLE=Regular', 'SDUUID=625b62bd-21ae-4068-bc86-657056036298', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=4904df5cc00154bbaa53751374407b2c92644d82'] >Thread-11821::DEBUG::2012-12-02 14:51:01,716::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11821::WARNING::2012-12-02 14:51:01,716::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_imageNS already registered >Thread-11821::WARNING::2012-12-02 14:51:01,717::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 625b62bd-21ae-4068-bc86-657056036298_volumeNS already registered >Thread-11822::DEBUG::2012-12-02 14:51:08,065::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:51:08,065::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11822::DEBUG::2012-12-02 14:51:08,065::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 31e5485d-d40c-4a93-baea-0ff411f7b66a' (cwd None) >Thread-11822::DEBUG::2012-12-02 14:51:08,274::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found\n'; <rc> = 5 >Thread-11822::WARNING::2012-12-02 14:51:08,276::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "31e5485d-d40c-4a93-baea-0ff411f7b66a" not found'] >Thread-11822::DEBUG::2012-12-02 14:51:08,276::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11822::DEBUG::2012-12-02 14:51:08,312::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1/31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-11822::DEBUG::2012-12-02 14:51:08,312::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11822::DEBUG::2012-12-02 14:51:08,315::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=93a7ba192c400bc7cf8723d31fa1c8b8f91381d0'] >Thread-11822::DEBUG::2012-12-02 14:51:08,318::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11822::WARNING::2012-12-02 14:51:08,318::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_imageNS already registered >Thread-11822::WARNING::2012-12-02 14:51:08,318::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 31e5485d-d40c-4a93-baea-0ff411f7b66a_volumeNS already registered >Thread-11946::DEBUG::2012-12-02 14:51:08,344::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:51:08,346::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11946::DEBUG::2012-12-02 14:51:08,347::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 95e02403-62f6-47c7-b706-e7be59adb348' (cwd None) >Thread-11946::DEBUG::2012-12-02 14:51:08,555::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found\n'; <rc> = 5 >Thread-11946::WARNING::2012-12-02 14:51:08,556::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "95e02403-62f6-47c7-b706-e7be59adb348" not found'] >Thread-11946::DEBUG::2012-12-02 14:51:08,556::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11946::DEBUG::2012-12-02 14:51:08,590::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3/95e02403-62f6-47c7-b706-e7be59adb348 >Thread-11946::DEBUG::2012-12-02 14:51:08,590::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-11946::DEBUG::2012-12-02 14:51:08,599::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'ROLE=Regular', 'SDUUID=95e02403-62f6-47c7-b706-e7be59adb348', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a389b97b78f3b49d04d22f02edfe712d0c6260d9'] >Thread-11946::DEBUG::2012-12-02 14:51:08,602::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-11946::WARNING::2012-12-02 14:51:08,602::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_imageNS already registered >Thread-11946::WARNING::2012-12-02 14:51:08,603::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 95e02403-62f6-47c7-b706-e7be59adb348_volumeNS already registered >Thread-30400::DEBUG::2012-12-02 14:51:09,459::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30400::DEBUG::2012-12-02 14:51:09,459::task::568::TaskManager.Task::(_updateState) Task=`f2674bba-0199-4720-ae35-e439e02499f4`::moving from state init -> state preparing >Thread-30400::INFO::2012-12-02 14:51:09,460::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30400::INFO::2012-12-02 14:51:09,460::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30400::DEBUG::2012-12-02 14:51:09,460::task::1151::TaskManager.Task::(prepare) Task=`f2674bba-0199-4720-ae35-e439e02499f4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30400::DEBUG::2012-12-02 14:51:09,460::task::568::TaskManager.Task::(_updateState) Task=`f2674bba-0199-4720-ae35-e439e02499f4`::moving from state preparing -> state finished >Thread-30400::DEBUG::2012-12-02 14:51:09,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30400::DEBUG::2012-12-02 14:51:09,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30400::DEBUG::2012-12-02 14:51:09,460::task::957::TaskManager.Task::(_decref) Task=`f2674bba-0199-4720-ae35-e439e02499f4`::ref 0 aborting False >Thread-30401::DEBUG::2012-12-02 14:51:09,463::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30401::DEBUG::2012-12-02 14:51:09,463::task::568::TaskManager.Task::(_updateState) Task=`ccfa3db4-f328-4c7c-aa00-10d8126100a0`::moving from state init -> state preparing >Thread-30401::INFO::2012-12-02 14:51:09,463::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30401::DEBUG::2012-12-02 14:51:09,464::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e94a9a86-9cd2-4c6f-bfa0-7c8a41ce1038`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30401::DEBUG::2012-12-02 14:51:09,464::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30401::DEBUG::2012-12-02 14:51:09,464::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30401::DEBUG::2012-12-02 14:51:09,464::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e94a9a86-9cd2-4c6f-bfa0-7c8a41ce1038`::Granted request >Thread-30401::DEBUG::2012-12-02 14:51:09,464::task::794::TaskManager.Task::(resourceAcquired) Task=`ccfa3db4-f328-4c7c-aa00-10d8126100a0`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30401::DEBUG::2012-12-02 14:51:09,465::task::957::TaskManager.Task::(_decref) Task=`ccfa3db4-f328-4c7c-aa00-10d8126100a0`::ref 1 aborting False >Thread-30401::INFO::2012-12-02 14:51:09,469::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30401::DEBUG::2012-12-02 14:51:09,469::task::1151::TaskManager.Task::(prepare) Task=`ccfa3db4-f328-4c7c-aa00-10d8126100a0`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30401::DEBUG::2012-12-02 14:51:09,470::task::568::TaskManager.Task::(_updateState) Task=`ccfa3db4-f328-4c7c-aa00-10d8126100a0`::moving from state preparing -> state finished >Thread-30401::DEBUG::2012-12-02 14:51:09,470::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30401::DEBUG::2012-12-02 14:51:09,470::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30401::DEBUG::2012-12-02 14:51:09,470::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30401::DEBUG::2012-12-02 14:51:09,470::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30401::DEBUG::2012-12-02 14:51:09,470::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30401::DEBUG::2012-12-02 14:51:09,470::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30401::DEBUG::2012-12-02 14:51:09,470::task::957::TaskManager.Task::(_decref) Task=`ccfa3db4-f328-4c7c-aa00-10d8126100a0`::ref 0 aborting False >Thread-30402::DEBUG::2012-12-02 14:51:09,575::task::568::TaskManager.Task::(_updateState) Task=`ec0e7e94-a575-45b8-8c56-0f24fb8e958b`::moving from state init -> state preparing >Thread-30402::INFO::2012-12-02 14:51:09,575::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30402::INFO::2012-12-02 14:51:09,575::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00702691078186', 'lastCheck': 1354452661.732846, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00239086151123', 'lastCheck': 1354452668.3301649, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00273585319519', 'lastCheck': 1354452668.608398, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00649905204773', 'lastCheck': 1354452668.617538, 'code': 0, 'valid': True}} >Thread-30402::DEBUG::2012-12-02 14:51:09,575::task::1151::TaskManager.Task::(prepare) Task=`ec0e7e94-a575-45b8-8c56-0f24fb8e958b`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00702691078186', 'lastCheck': 1354452661.732846, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00239086151123', 'lastCheck': 1354452668.3301649, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00273585319519', 'lastCheck': 1354452668.608398, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00649905204773', 'lastCheck': 1354452668.617538, 'code': 0, 'valid': True}} >Thread-30402::DEBUG::2012-12-02 14:51:09,576::task::568::TaskManager.Task::(_updateState) Task=`ec0e7e94-a575-45b8-8c56-0f24fb8e958b`::moving from state preparing -> state finished >Thread-30402::DEBUG::2012-12-02 14:51:09,576::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30402::DEBUG::2012-12-02 14:51:09,576::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30402::DEBUG::2012-12-02 14:51:09,576::task::957::TaskManager.Task::(_decref) Task=`ec0e7e94-a575-45b8-8c56-0f24fb8e958b`::ref 0 aborting False >Thread-30408::DEBUG::2012-12-02 14:51:19,597::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30408::DEBUG::2012-12-02 14:51:19,598::task::568::TaskManager.Task::(_updateState) Task=`be2ec66d-8edb-44a1-9fe8-e7e9e7d496fa`::moving from state init -> state preparing >Thread-30408::INFO::2012-12-02 14:51:19,598::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30408::INFO::2012-12-02 14:51:19,598::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30408::DEBUG::2012-12-02 14:51:19,598::task::1151::TaskManager.Task::(prepare) Task=`be2ec66d-8edb-44a1-9fe8-e7e9e7d496fa`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30408::DEBUG::2012-12-02 14:51:19,598::task::568::TaskManager.Task::(_updateState) Task=`be2ec66d-8edb-44a1-9fe8-e7e9e7d496fa`::moving from state preparing -> state finished >Thread-30408::DEBUG::2012-12-02 14:51:19,598::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30408::DEBUG::2012-12-02 14:51:19,598::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30408::DEBUG::2012-12-02 14:51:19,599::task::957::TaskManager.Task::(_decref) Task=`be2ec66d-8edb-44a1-9fe8-e7e9e7d496fa`::ref 0 aborting False >Thread-30409::DEBUG::2012-12-02 14:51:19,601::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30409::DEBUG::2012-12-02 14:51:19,601::task::568::TaskManager.Task::(_updateState) Task=`d5a8c787-850c-4a1d-abd4-406269b459fd`::moving from state init -> state preparing >Thread-30409::INFO::2012-12-02 14:51:19,602::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30409::DEBUG::2012-12-02 14:51:19,602::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`06fd1caf-3e66-4bde-a115-afa1d57ab12c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30409::DEBUG::2012-12-02 14:51:19,602::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30409::DEBUG::2012-12-02 14:51:19,602::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30409::DEBUG::2012-12-02 14:51:19,602::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`06fd1caf-3e66-4bde-a115-afa1d57ab12c`::Granted request >Thread-30409::DEBUG::2012-12-02 14:51:19,603::task::794::TaskManager.Task::(resourceAcquired) Task=`d5a8c787-850c-4a1d-abd4-406269b459fd`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30409::DEBUG::2012-12-02 14:51:19,603::task::957::TaskManager.Task::(_decref) Task=`d5a8c787-850c-4a1d-abd4-406269b459fd`::ref 1 aborting False >Thread-30409::INFO::2012-12-02 14:51:19,608::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30409::DEBUG::2012-12-02 14:51:19,608::task::1151::TaskManager.Task::(prepare) Task=`d5a8c787-850c-4a1d-abd4-406269b459fd`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30409::DEBUG::2012-12-02 14:51:19,608::task::568::TaskManager.Task::(_updateState) Task=`d5a8c787-850c-4a1d-abd4-406269b459fd`::moving from state preparing -> state finished >Thread-30409::DEBUG::2012-12-02 14:51:19,608::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30409::DEBUG::2012-12-02 14:51:19,608::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30409::DEBUG::2012-12-02 14:51:19,608::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30409::DEBUG::2012-12-02 14:51:19,608::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30409::DEBUG::2012-12-02 14:51:19,609::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30409::DEBUG::2012-12-02 14:51:19,609::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30409::DEBUG::2012-12-02 14:51:19,609::task::957::TaskManager.Task::(_decref) Task=`d5a8c787-850c-4a1d-abd4-406269b459fd`::ref 0 aborting False >Thread-30410::DEBUG::2012-12-02 14:51:19,690::task::568::TaskManager.Task::(_updateState) Task=`319f2973-b885-46a4-aa3e-e049a00a4c7a`::moving from state init -> state preparing >Thread-30410::INFO::2012-12-02 14:51:19,690::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30410::INFO::2012-12-02 14:51:19,690::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00851511955261', 'lastCheck': 1354452671.7514191, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00321698188782', 'lastCheck': 1354452678.346025, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00467109680176', 'lastCheck': 1354452678.6257689, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00868606567383', 'lastCheck': 1354452678.6372311, 'code': 0, 'valid': True}} >Thread-30410::DEBUG::2012-12-02 14:51:19,690::task::1151::TaskManager.Task::(prepare) Task=`319f2973-b885-46a4-aa3e-e049a00a4c7a`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00851511955261', 'lastCheck': 1354452671.7514191, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00321698188782', 'lastCheck': 1354452678.346025, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00467109680176', 'lastCheck': 1354452678.6257689, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00868606567383', 'lastCheck': 1354452678.6372311, 'code': 0, 'valid': True}} >Thread-30410::DEBUG::2012-12-02 14:51:19,690::task::568::TaskManager.Task::(_updateState) Task=`319f2973-b885-46a4-aa3e-e049a00a4c7a`::moving from state preparing -> state finished >Thread-30410::DEBUG::2012-12-02 14:51:19,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30410::DEBUG::2012-12-02 14:51:19,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30410::DEBUG::2012-12-02 14:51:19,691::task::957::TaskManager.Task::(_decref) Task=`319f2973-b885-46a4-aa3e-e049a00a4c7a`::ref 0 aborting False >Thread-30416::DEBUG::2012-12-02 14:51:29,702::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30416::DEBUG::2012-12-02 14:51:29,703::task::568::TaskManager.Task::(_updateState) Task=`f98f2f25-38cc-4b91-aeb8-79e0a25fcfc2`::moving from state init -> state preparing >Thread-30416::INFO::2012-12-02 14:51:29,703::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30416::INFO::2012-12-02 14:51:29,703::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30416::DEBUG::2012-12-02 14:51:29,703::task::1151::TaskManager.Task::(prepare) Task=`f98f2f25-38cc-4b91-aeb8-79e0a25fcfc2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30416::DEBUG::2012-12-02 14:51:29,703::task::568::TaskManager.Task::(_updateState) Task=`f98f2f25-38cc-4b91-aeb8-79e0a25fcfc2`::moving from state preparing -> state finished >Thread-30416::DEBUG::2012-12-02 14:51:29,703::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30416::DEBUG::2012-12-02 14:51:29,703::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30416::DEBUG::2012-12-02 14:51:29,704::task::957::TaskManager.Task::(_decref) Task=`f98f2f25-38cc-4b91-aeb8-79e0a25fcfc2`::ref 0 aborting False >Thread-30417::DEBUG::2012-12-02 14:51:29,706::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30417::DEBUG::2012-12-02 14:51:29,706::task::568::TaskManager.Task::(_updateState) Task=`e043a768-8b51-45a1-a0b3-898d66d774ae`::moving from state init -> state preparing >Thread-30417::INFO::2012-12-02 14:51:29,706::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30417::DEBUG::2012-12-02 14:51:29,707::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5326e438-6b42-4ff7-b3aa-380ed43d46cd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30417::DEBUG::2012-12-02 14:51:29,707::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30417::DEBUG::2012-12-02 14:51:29,707::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30417::DEBUG::2012-12-02 14:51:29,707::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5326e438-6b42-4ff7-b3aa-380ed43d46cd`::Granted request >Thread-30417::DEBUG::2012-12-02 14:51:29,707::task::794::TaskManager.Task::(resourceAcquired) Task=`e043a768-8b51-45a1-a0b3-898d66d774ae`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30417::DEBUG::2012-12-02 14:51:29,708::task::957::TaskManager.Task::(_decref) Task=`e043a768-8b51-45a1-a0b3-898d66d774ae`::ref 1 aborting False >Thread-30417::INFO::2012-12-02 14:51:29,714::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30417::DEBUG::2012-12-02 14:51:29,714::task::1151::TaskManager.Task::(prepare) Task=`e043a768-8b51-45a1-a0b3-898d66d774ae`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30417::DEBUG::2012-12-02 14:51:29,714::task::568::TaskManager.Task::(_updateState) Task=`e043a768-8b51-45a1-a0b3-898d66d774ae`::moving from state preparing -> state finished >Thread-30417::DEBUG::2012-12-02 14:51:29,714::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30417::DEBUG::2012-12-02 14:51:29,715::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30417::DEBUG::2012-12-02 14:51:29,715::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30417::DEBUG::2012-12-02 14:51:29,715::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30417::DEBUG::2012-12-02 14:51:29,715::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30417::DEBUG::2012-12-02 14:51:29,715::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30417::DEBUG::2012-12-02 14:51:29,715::task::957::TaskManager.Task::(_decref) Task=`e043a768-8b51-45a1-a0b3-898d66d774ae`::ref 0 aborting False >Thread-30418::DEBUG::2012-12-02 14:51:29,759::task::568::TaskManager.Task::(_updateState) Task=`90f6d6e5-8f61-404d-aabe-de35bf256884`::moving from state init -> state preparing >Thread-30418::INFO::2012-12-02 14:51:29,760::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30418::INFO::2012-12-02 14:51:29,760::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00779700279236', 'lastCheck': 1354452681.768548, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00263404846191', 'lastCheck': 1354452688.3631029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191879272461', 'lastCheck': 1354452688.63797, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00731706619263', 'lastCheck': 1354452688.6538219, 'code': 0, 'valid': True}} >Thread-30418::DEBUG::2012-12-02 14:51:29,760::task::1151::TaskManager.Task::(prepare) Task=`90f6d6e5-8f61-404d-aabe-de35bf256884`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00779700279236', 'lastCheck': 1354452681.768548, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00263404846191', 'lastCheck': 1354452688.3631029, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191879272461', 'lastCheck': 1354452688.63797, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00731706619263', 'lastCheck': 1354452688.6538219, 'code': 0, 'valid': True}} >Thread-30418::DEBUG::2012-12-02 14:51:29,760::task::568::TaskManager.Task::(_updateState) Task=`90f6d6e5-8f61-404d-aabe-de35bf256884`::moving from state preparing -> state finished >Thread-30418::DEBUG::2012-12-02 14:51:29,760::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30418::DEBUG::2012-12-02 14:51:29,760::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30418::DEBUG::2012-12-02 14:51:29,761::task::957::TaskManager.Task::(_decref) Task=`90f6d6e5-8f61-404d-aabe-de35bf256884`::ref 0 aborting False >Thread-30424::DEBUG::2012-12-02 14:51:39,840::task::568::TaskManager.Task::(_updateState) Task=`8c3ef35a-144b-4388-90cc-47aeed28d391`::moving from state init -> state preparing >Thread-30424::INFO::2012-12-02 14:51:39,840::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30424::INFO::2012-12-02 14:51:39,840::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00873303413391', 'lastCheck': 1354452691.7867429, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0027871131897', 'lastCheck': 1354452698.374265, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00311708450317', 'lastCheck': 1354452698.6518099, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00713610649109', 'lastCheck': 1354452698.6685741, 'code': 0, 'valid': True}} >Thread-30424::DEBUG::2012-12-02 14:51:39,840::task::1151::TaskManager.Task::(prepare) Task=`8c3ef35a-144b-4388-90cc-47aeed28d391`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00873303413391', 'lastCheck': 1354452691.7867429, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0027871131897', 'lastCheck': 1354452698.374265, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00311708450317', 'lastCheck': 1354452698.6518099, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00713610649109', 'lastCheck': 1354452698.6685741, 'code': 0, 'valid': True}} >Thread-30424::DEBUG::2012-12-02 14:51:39,840::task::568::TaskManager.Task::(_updateState) Task=`8c3ef35a-144b-4388-90cc-47aeed28d391`::moving from state preparing -> state finished >Thread-30424::DEBUG::2012-12-02 14:51:39,841::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30424::DEBUG::2012-12-02 14:51:39,841::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30424::DEBUG::2012-12-02 14:51:39,841::task::957::TaskManager.Task::(_decref) Task=`8c3ef35a-144b-4388-90cc-47aeed28d391`::ref 0 aborting False >Thread-30426::DEBUG::2012-12-02 14:51:39,874::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30426::DEBUG::2012-12-02 14:51:39,874::task::568::TaskManager.Task::(_updateState) Task=`9ba44961-e7c3-43d6-85d0-47756aa41d9c`::moving from state init -> state preparing >Thread-30426::INFO::2012-12-02 14:51:39,874::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30426::INFO::2012-12-02 14:51:39,874::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30426::DEBUG::2012-12-02 14:51:39,874::task::1151::TaskManager.Task::(prepare) Task=`9ba44961-e7c3-43d6-85d0-47756aa41d9c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30426::DEBUG::2012-12-02 14:51:39,875::task::568::TaskManager.Task::(_updateState) Task=`9ba44961-e7c3-43d6-85d0-47756aa41d9c`::moving from state preparing -> state finished >Thread-30426::DEBUG::2012-12-02 14:51:39,875::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30426::DEBUG::2012-12-02 14:51:39,875::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30426::DEBUG::2012-12-02 14:51:39,875::task::957::TaskManager.Task::(_decref) Task=`9ba44961-e7c3-43d6-85d0-47756aa41d9c`::ref 0 aborting False >Thread-30427::DEBUG::2012-12-02 14:51:39,877::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30427::DEBUG::2012-12-02 14:51:39,877::task::568::TaskManager.Task::(_updateState) Task=`e3fbc839-54b2-42fe-b1b4-2e73becc8a41`::moving from state init -> state preparing >Thread-30427::INFO::2012-12-02 14:51:39,878::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30427::DEBUG::2012-12-02 14:51:39,878::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`44be682c-8e7c-4a44-86f5-01793af9e9a5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30427::DEBUG::2012-12-02 14:51:39,878::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30427::DEBUG::2012-12-02 14:51:39,878::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30427::DEBUG::2012-12-02 14:51:39,878::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`44be682c-8e7c-4a44-86f5-01793af9e9a5`::Granted request >Thread-30427::DEBUG::2012-12-02 14:51:39,878::task::794::TaskManager.Task::(resourceAcquired) Task=`e3fbc839-54b2-42fe-b1b4-2e73becc8a41`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30427::DEBUG::2012-12-02 14:51:39,878::task::957::TaskManager.Task::(_decref) Task=`e3fbc839-54b2-42fe-b1b4-2e73becc8a41`::ref 1 aborting False >Thread-30427::INFO::2012-12-02 14:51:39,883::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30427::DEBUG::2012-12-02 14:51:39,883::task::1151::TaskManager.Task::(prepare) Task=`e3fbc839-54b2-42fe-b1b4-2e73becc8a41`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30427::DEBUG::2012-12-02 14:51:39,883::task::568::TaskManager.Task::(_updateState) Task=`e3fbc839-54b2-42fe-b1b4-2e73becc8a41`::moving from state preparing -> state finished >Thread-30427::DEBUG::2012-12-02 14:51:39,883::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30427::DEBUG::2012-12-02 14:51:39,883::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30427::DEBUG::2012-12-02 14:51:39,883::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30427::DEBUG::2012-12-02 14:51:39,883::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30427::DEBUG::2012-12-02 14:51:39,883::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30427::DEBUG::2012-12-02 14:51:39,883::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30427::DEBUG::2012-12-02 14:51:39,883::task::957::TaskManager.Task::(_decref) Task=`e3fbc839-54b2-42fe-b1b4-2e73becc8a41`::ref 0 aborting False >Thread-30432::DEBUG::2012-12-02 14:51:49,935::task::568::TaskManager.Task::(_updateState) Task=`886879a8-b3ef-49ee-83fd-5101fa297218`::moving from state init -> state preparing >Thread-30432::INFO::2012-12-02 14:51:49,935::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30432::INFO::2012-12-02 14:51:49,936::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00781106948853', 'lastCheck': 1354452701.8065579, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00173282623291', 'lastCheck': 1354452708.3873889, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00209784507751', 'lastCheck': 1354452708.6627319, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00799989700317', 'lastCheck': 1354452708.6874261, 'code': 0, 'valid': True}} >Thread-30432::DEBUG::2012-12-02 14:51:49,936::task::1151::TaskManager.Task::(prepare) Task=`886879a8-b3ef-49ee-83fd-5101fa297218`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00781106948853', 'lastCheck': 1354452701.8065579, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00173282623291', 'lastCheck': 1354452708.3873889, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00209784507751', 'lastCheck': 1354452708.6627319, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00799989700317', 'lastCheck': 1354452708.6874261, 'code': 0, 'valid': True}} >Thread-30432::DEBUG::2012-12-02 14:51:49,936::task::568::TaskManager.Task::(_updateState) Task=`886879a8-b3ef-49ee-83fd-5101fa297218`::moving from state preparing -> state finished >Thread-30432::DEBUG::2012-12-02 14:51:49,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30432::DEBUG::2012-12-02 14:51:49,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30432::DEBUG::2012-12-02 14:51:49,936::task::957::TaskManager.Task::(_decref) Task=`886879a8-b3ef-49ee-83fd-5101fa297218`::ref 0 aborting False >Thread-30434::DEBUG::2012-12-02 14:51:49,994::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30434::DEBUG::2012-12-02 14:51:49,994::task::568::TaskManager.Task::(_updateState) Task=`344ea774-62b5-4234-b0d0-17e2059e9d3f`::moving from state init -> state preparing >Thread-30434::INFO::2012-12-02 14:51:49,994::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30434::INFO::2012-12-02 14:51:49,995::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30434::DEBUG::2012-12-02 14:51:49,995::task::1151::TaskManager.Task::(prepare) Task=`344ea774-62b5-4234-b0d0-17e2059e9d3f`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30434::DEBUG::2012-12-02 14:51:49,995::task::568::TaskManager.Task::(_updateState) Task=`344ea774-62b5-4234-b0d0-17e2059e9d3f`::moving from state preparing -> state finished >Thread-30434::DEBUG::2012-12-02 14:51:49,995::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30434::DEBUG::2012-12-02 14:51:49,995::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30434::DEBUG::2012-12-02 14:51:49,995::task::957::TaskManager.Task::(_decref) Task=`344ea774-62b5-4234-b0d0-17e2059e9d3f`::ref 0 aborting False >Thread-30435::DEBUG::2012-12-02 14:51:49,998::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30435::DEBUG::2012-12-02 14:51:49,998::task::568::TaskManager.Task::(_updateState) Task=`99306380-f623-4218-afa5-4ee665bd1457`::moving from state init -> state preparing >Thread-30435::INFO::2012-12-02 14:51:49,998::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30435::DEBUG::2012-12-02 14:51:49,998::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e284ad61-41e6-4666-80dd-5f5bfe1f82c0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30435::DEBUG::2012-12-02 14:51:49,998::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30435::DEBUG::2012-12-02 14:51:49,998::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30435::DEBUG::2012-12-02 14:51:49,998::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e284ad61-41e6-4666-80dd-5f5bfe1f82c0`::Granted request >Thread-30435::DEBUG::2012-12-02 14:51:49,999::task::794::TaskManager.Task::(resourceAcquired) Task=`99306380-f623-4218-afa5-4ee665bd1457`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30435::DEBUG::2012-12-02 14:51:49,999::task::957::TaskManager.Task::(_decref) Task=`99306380-f623-4218-afa5-4ee665bd1457`::ref 1 aborting False >Thread-30435::INFO::2012-12-02 14:51:50,004::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30435::DEBUG::2012-12-02 14:51:50,004::task::1151::TaskManager.Task::(prepare) Task=`99306380-f623-4218-afa5-4ee665bd1457`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30435::DEBUG::2012-12-02 14:51:50,004::task::568::TaskManager.Task::(_updateState) Task=`99306380-f623-4218-afa5-4ee665bd1457`::moving from state preparing -> state finished >Thread-30435::DEBUG::2012-12-02 14:51:50,004::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30435::DEBUG::2012-12-02 14:51:50,004::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30435::DEBUG::2012-12-02 14:51:50,004::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30435::DEBUG::2012-12-02 14:51:50,004::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30435::DEBUG::2012-12-02 14:51:50,004::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30435::DEBUG::2012-12-02 14:51:50,004::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30435::DEBUG::2012-12-02 14:51:50,005::task::957::TaskManager.Task::(_decref) Task=`99306380-f623-4218-afa5-4ee665bd1457`::ref 0 aborting False >Thread-30440::DEBUG::2012-12-02 14:52:00,032::task::568::TaskManager.Task::(_updateState) Task=`dd36131e-175a-42d0-91f5-d7b29ef6626c`::moving from state init -> state preparing >Thread-30440::INFO::2012-12-02 14:52:00,032::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30440::INFO::2012-12-02 14:52:00,033::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00851202011108', 'lastCheck': 1354452711.8253019, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00181698799133', 'lastCheck': 1354452718.4014111, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177097320557', 'lastCheck': 1354452718.6735549, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809001922607', 'lastCheck': 1354452718.7045219, 'code': 0, 'valid': True}} >Thread-30440::DEBUG::2012-12-02 14:52:00,033::task::1151::TaskManager.Task::(prepare) Task=`dd36131e-175a-42d0-91f5-d7b29ef6626c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00851202011108', 'lastCheck': 1354452711.8253019, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00181698799133', 'lastCheck': 1354452718.4014111, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00177097320557', 'lastCheck': 1354452718.6735549, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809001922607', 'lastCheck': 1354452718.7045219, 'code': 0, 'valid': True}} >Thread-30440::DEBUG::2012-12-02 14:52:00,033::task::568::TaskManager.Task::(_updateState) Task=`dd36131e-175a-42d0-91f5-d7b29ef6626c`::moving from state preparing -> state finished >Thread-30440::DEBUG::2012-12-02 14:52:00,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30440::DEBUG::2012-12-02 14:52:00,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30440::DEBUG::2012-12-02 14:52:00,033::task::957::TaskManager.Task::(_decref) Task=`dd36131e-175a-42d0-91f5-d7b29ef6626c`::ref 0 aborting False >Thread-30442::DEBUG::2012-12-02 14:52:00,124::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30442::DEBUG::2012-12-02 14:52:00,124::task::568::TaskManager.Task::(_updateState) Task=`e161b0f2-8695-4426-b2bc-3f77f511f600`::moving from state init -> state preparing >Thread-30442::INFO::2012-12-02 14:52:00,124::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30442::INFO::2012-12-02 14:52:00,125::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30442::DEBUG::2012-12-02 14:52:00,125::task::1151::TaskManager.Task::(prepare) Task=`e161b0f2-8695-4426-b2bc-3f77f511f600`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30442::DEBUG::2012-12-02 14:52:00,125::task::568::TaskManager.Task::(_updateState) Task=`e161b0f2-8695-4426-b2bc-3f77f511f600`::moving from state preparing -> state finished >Thread-30442::DEBUG::2012-12-02 14:52:00,125::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30442::DEBUG::2012-12-02 14:52:00,125::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30442::DEBUG::2012-12-02 14:52:00,125::task::957::TaskManager.Task::(_decref) Task=`e161b0f2-8695-4426-b2bc-3f77f511f600`::ref 0 aborting False >Thread-30443::DEBUG::2012-12-02 14:52:00,128::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30443::DEBUG::2012-12-02 14:52:00,128::task::568::TaskManager.Task::(_updateState) Task=`b0760b46-2135-4eb9-b928-d98af296069e`::moving from state init -> state preparing >Thread-30443::INFO::2012-12-02 14:52:00,128::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30443::DEBUG::2012-12-02 14:52:00,128::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`539fc594-3281-4540-89c5-811c2cff01ef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30443::DEBUG::2012-12-02 14:52:00,128::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30443::DEBUG::2012-12-02 14:52:00,129::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30443::DEBUG::2012-12-02 14:52:00,129::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`539fc594-3281-4540-89c5-811c2cff01ef`::Granted request >Thread-30443::DEBUG::2012-12-02 14:52:00,129::task::794::TaskManager.Task::(resourceAcquired) Task=`b0760b46-2135-4eb9-b928-d98af296069e`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30443::DEBUG::2012-12-02 14:52:00,129::task::957::TaskManager.Task::(_decref) Task=`b0760b46-2135-4eb9-b928-d98af296069e`::ref 1 aborting False >Thread-30443::INFO::2012-12-02 14:52:00,135::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30443::DEBUG::2012-12-02 14:52:00,135::task::1151::TaskManager.Task::(prepare) Task=`b0760b46-2135-4eb9-b928-d98af296069e`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30443::DEBUG::2012-12-02 14:52:00,136::task::568::TaskManager.Task::(_updateState) Task=`b0760b46-2135-4eb9-b928-d98af296069e`::moving from state preparing -> state finished >Thread-30443::DEBUG::2012-12-02 14:52:00,136::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30443::DEBUG::2012-12-02 14:52:00,136::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30443::DEBUG::2012-12-02 14:52:00,136::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30443::DEBUG::2012-12-02 14:52:00,136::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30443::DEBUG::2012-12-02 14:52:00,136::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30443::DEBUG::2012-12-02 14:52:00,136::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30443::DEBUG::2012-12-02 14:52:00,136::task::957::TaskManager.Task::(_decref) Task=`b0760b46-2135-4eb9-b928-d98af296069e`::ref 0 aborting False >Thread-30448::DEBUG::2012-12-02 14:52:10,164::task::568::TaskManager.Task::(_updateState) Task=`9c11f43b-3ddb-4757-bf9d-b9b5507f0715`::moving from state init -> state preparing >Thread-30448::INFO::2012-12-02 14:52:10,164::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30448::INFO::2012-12-02 14:52:10,165::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00402617454529', 'lastCheck': 1354452721.838959, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00176596641541', 'lastCheck': 1354452728.4131501, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00151205062866', 'lastCheck': 1354452728.6835141, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812196731567', 'lastCheck': 1354452728.7213299, 'code': 0, 'valid': True}} >Thread-30448::DEBUG::2012-12-02 14:52:10,165::task::1151::TaskManager.Task::(prepare) Task=`9c11f43b-3ddb-4757-bf9d-b9b5507f0715`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00402617454529', 'lastCheck': 1354452721.838959, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00176596641541', 'lastCheck': 1354452728.4131501, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00151205062866', 'lastCheck': 1354452728.6835141, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812196731567', 'lastCheck': 1354452728.7213299, 'code': 0, 'valid': True}} >Thread-30448::DEBUG::2012-12-02 14:52:10,165::task::568::TaskManager.Task::(_updateState) Task=`9c11f43b-3ddb-4757-bf9d-b9b5507f0715`::moving from state preparing -> state finished >Thread-30448::DEBUG::2012-12-02 14:52:10,165::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30448::DEBUG::2012-12-02 14:52:10,165::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30448::DEBUG::2012-12-02 14:52:10,165::task::957::TaskManager.Task::(_decref) Task=`9c11f43b-3ddb-4757-bf9d-b9b5507f0715`::ref 0 aborting False >Thread-30450::DEBUG::2012-12-02 14:52:10,229::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30450::DEBUG::2012-12-02 14:52:10,230::task::568::TaskManager.Task::(_updateState) Task=`31b3e860-0667-421d-8445-02e230419969`::moving from state init -> state preparing >Thread-30450::INFO::2012-12-02 14:52:10,230::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30450::INFO::2012-12-02 14:52:10,230::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30450::DEBUG::2012-12-02 14:52:10,230::task::1151::TaskManager.Task::(prepare) Task=`31b3e860-0667-421d-8445-02e230419969`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30450::DEBUG::2012-12-02 14:52:10,230::task::568::TaskManager.Task::(_updateState) Task=`31b3e860-0667-421d-8445-02e230419969`::moving from state preparing -> state finished >Thread-30450::DEBUG::2012-12-02 14:52:10,230::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30450::DEBUG::2012-12-02 14:52:10,230::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30450::DEBUG::2012-12-02 14:52:10,231::task::957::TaskManager.Task::(_decref) Task=`31b3e860-0667-421d-8445-02e230419969`::ref 0 aborting False >Thread-30451::DEBUG::2012-12-02 14:52:10,233::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30451::DEBUG::2012-12-02 14:52:10,233::task::568::TaskManager.Task::(_updateState) Task=`2d09696d-4db0-4bdd-8ec4-f0eb7f6b65ad`::moving from state init -> state preparing >Thread-30451::INFO::2012-12-02 14:52:10,233::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30451::DEBUG::2012-12-02 14:52:10,234::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e2da5993-092a-4ea8-b335-41a85c3c3468`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30451::DEBUG::2012-12-02 14:52:10,234::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30451::DEBUG::2012-12-02 14:52:10,234::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30451::DEBUG::2012-12-02 14:52:10,234::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`e2da5993-092a-4ea8-b335-41a85c3c3468`::Granted request >Thread-30451::DEBUG::2012-12-02 14:52:10,234::task::794::TaskManager.Task::(resourceAcquired) Task=`2d09696d-4db0-4bdd-8ec4-f0eb7f6b65ad`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30451::DEBUG::2012-12-02 14:52:10,235::task::957::TaskManager.Task::(_decref) Task=`2d09696d-4db0-4bdd-8ec4-f0eb7f6b65ad`::ref 1 aborting False >Thread-30451::INFO::2012-12-02 14:52:10,241::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30451::DEBUG::2012-12-02 14:52:10,241::task::1151::TaskManager.Task::(prepare) Task=`2d09696d-4db0-4bdd-8ec4-f0eb7f6b65ad`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30451::DEBUG::2012-12-02 14:52:10,241::task::568::TaskManager.Task::(_updateState) Task=`2d09696d-4db0-4bdd-8ec4-f0eb7f6b65ad`::moving from state preparing -> state finished >Thread-30451::DEBUG::2012-12-02 14:52:10,241::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30451::DEBUG::2012-12-02 14:52:10,242::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30451::DEBUG::2012-12-02 14:52:10,242::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30451::DEBUG::2012-12-02 14:52:10,242::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30451::DEBUG::2012-12-02 14:52:10,242::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30451::DEBUG::2012-12-02 14:52:10,242::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30451::DEBUG::2012-12-02 14:52:10,242::task::957::TaskManager.Task::(_decref) Task=`2d09696d-4db0-4bdd-8ec4-f0eb7f6b65ad`::ref 0 aborting False >Thread-30456::DEBUG::2012-12-02 14:52:20,234::task::568::TaskManager.Task::(_updateState) Task=`f526e7ca-9505-4c0a-b5fe-4942e3a93f19`::moving from state init -> state preparing >Thread-30456::INFO::2012-12-02 14:52:20,234::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30456::INFO::2012-12-02 14:52:20,234::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00835204124451', 'lastCheck': 1354452731.85724, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00390100479126', 'lastCheck': 1354452738.4280369, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00279593467712', 'lastCheck': 1354452738.6966231, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00704002380371', 'lastCheck': 1354452738.7365861, 'code': 0, 'valid': True}} >Thread-30456::DEBUG::2012-12-02 14:52:20,235::task::1151::TaskManager.Task::(prepare) Task=`f526e7ca-9505-4c0a-b5fe-4942e3a93f19`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00835204124451', 'lastCheck': 1354452731.85724, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00390100479126', 'lastCheck': 1354452738.4280369, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00279593467712', 'lastCheck': 1354452738.6966231, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00704002380371', 'lastCheck': 1354452738.7365861, 'code': 0, 'valid': True}} >Thread-30456::DEBUG::2012-12-02 14:52:20,235::task::568::TaskManager.Task::(_updateState) Task=`f526e7ca-9505-4c0a-b5fe-4942e3a93f19`::moving from state preparing -> state finished >Thread-30456::DEBUG::2012-12-02 14:52:20,235::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30456::DEBUG::2012-12-02 14:52:20,235::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30456::DEBUG::2012-12-02 14:52:20,235::task::957::TaskManager.Task::(_decref) Task=`f526e7ca-9505-4c0a-b5fe-4942e3a93f19`::ref 0 aborting False >Thread-30458::DEBUG::2012-12-02 14:52:20,334::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30458::DEBUG::2012-12-02 14:52:20,334::task::568::TaskManager.Task::(_updateState) Task=`9354c8cf-97ce-4783-9fd8-ff03a285898d`::moving from state init -> state preparing >Thread-30458::INFO::2012-12-02 14:52:20,334::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30458::INFO::2012-12-02 14:52:20,334::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30458::DEBUG::2012-12-02 14:52:20,334::task::1151::TaskManager.Task::(prepare) Task=`9354c8cf-97ce-4783-9fd8-ff03a285898d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30458::DEBUG::2012-12-02 14:52:20,335::task::568::TaskManager.Task::(_updateState) Task=`9354c8cf-97ce-4783-9fd8-ff03a285898d`::moving from state preparing -> state finished >Thread-30458::DEBUG::2012-12-02 14:52:20,335::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30458::DEBUG::2012-12-02 14:52:20,335::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30458::DEBUG::2012-12-02 14:52:20,335::task::957::TaskManager.Task::(_decref) Task=`9354c8cf-97ce-4783-9fd8-ff03a285898d`::ref 0 aborting False >Thread-30459::DEBUG::2012-12-02 14:52:20,337::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30459::DEBUG::2012-12-02 14:52:20,338::task::568::TaskManager.Task::(_updateState) Task=`a58d1ba9-4c4b-4bf4-bd2f-e2d9960fecd4`::moving from state init -> state preparing >Thread-30459::INFO::2012-12-02 14:52:20,338::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30459::DEBUG::2012-12-02 14:52:20,338::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4cf168fb-94cf-44e1-a7ce-05835ba1f853`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30459::DEBUG::2012-12-02 14:52:20,338::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30459::DEBUG::2012-12-02 14:52:20,338::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30459::DEBUG::2012-12-02 14:52:20,339::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4cf168fb-94cf-44e1-a7ce-05835ba1f853`::Granted request >Thread-30459::DEBUG::2012-12-02 14:52:20,339::task::794::TaskManager.Task::(resourceAcquired) Task=`a58d1ba9-4c4b-4bf4-bd2f-e2d9960fecd4`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30459::DEBUG::2012-12-02 14:52:20,339::task::957::TaskManager.Task::(_decref) Task=`a58d1ba9-4c4b-4bf4-bd2f-e2d9960fecd4`::ref 1 aborting False >Thread-30459::INFO::2012-12-02 14:52:20,345::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30459::DEBUG::2012-12-02 14:52:20,345::task::1151::TaskManager.Task::(prepare) Task=`a58d1ba9-4c4b-4bf4-bd2f-e2d9960fecd4`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30459::DEBUG::2012-12-02 14:52:20,345::task::568::TaskManager.Task::(_updateState) Task=`a58d1ba9-4c4b-4bf4-bd2f-e2d9960fecd4`::moving from state preparing -> state finished >Thread-30459::DEBUG::2012-12-02 14:52:20,346::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30459::DEBUG::2012-12-02 14:52:20,346::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30459::DEBUG::2012-12-02 14:52:20,346::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30459::DEBUG::2012-12-02 14:52:20,346::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30459::DEBUG::2012-12-02 14:52:20,346::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30459::DEBUG::2012-12-02 14:52:20,346::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30459::DEBUG::2012-12-02 14:52:20,346::task::957::TaskManager.Task::(_decref) Task=`a58d1ba9-4c4b-4bf4-bd2f-e2d9960fecd4`::ref 0 aborting False >Thread-30464::DEBUG::2012-12-02 14:52:30,323::task::568::TaskManager.Task::(_updateState) Task=`39850b63-2eb7-48e2-a508-b4fe285c063e`::moving from state init -> state preparing >Thread-30464::INFO::2012-12-02 14:52:30,323::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30464::INFO::2012-12-02 14:52:30,323::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00742101669312', 'lastCheck': 1354452741.873935, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191402435303', 'lastCheck': 1354452748.44472, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00228190422058', 'lastCheck': 1354452748.7078869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00695896148682', 'lastCheck': 1354452748.752548, 'code': 0, 'valid': True}} >Thread-30464::DEBUG::2012-12-02 14:52:30,323::task::1151::TaskManager.Task::(prepare) Task=`39850b63-2eb7-48e2-a508-b4fe285c063e`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00742101669312', 'lastCheck': 1354452741.873935, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00191402435303', 'lastCheck': 1354452748.44472, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00228190422058', 'lastCheck': 1354452748.7078869, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00695896148682', 'lastCheck': 1354452748.752548, 'code': 0, 'valid': True}} >Thread-30464::DEBUG::2012-12-02 14:52:30,324::task::568::TaskManager.Task::(_updateState) Task=`39850b63-2eb7-48e2-a508-b4fe285c063e`::moving from state preparing -> state finished >Thread-30464::DEBUG::2012-12-02 14:52:30,324::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30464::DEBUG::2012-12-02 14:52:30,324::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30464::DEBUG::2012-12-02 14:52:30,324::task::957::TaskManager.Task::(_decref) Task=`39850b63-2eb7-48e2-a508-b4fe285c063e`::ref 0 aborting False >Thread-30466::DEBUG::2012-12-02 14:52:30,439::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30466::DEBUG::2012-12-02 14:52:30,440::task::568::TaskManager.Task::(_updateState) Task=`4cc32513-3bc9-4529-831a-88a5dd435c59`::moving from state init -> state preparing >Thread-30466::INFO::2012-12-02 14:52:30,440::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30466::INFO::2012-12-02 14:52:30,440::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30466::DEBUG::2012-12-02 14:52:30,440::task::1151::TaskManager.Task::(prepare) Task=`4cc32513-3bc9-4529-831a-88a5dd435c59`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30466::DEBUG::2012-12-02 14:52:30,440::task::568::TaskManager.Task::(_updateState) Task=`4cc32513-3bc9-4529-831a-88a5dd435c59`::moving from state preparing -> state finished >Thread-30466::DEBUG::2012-12-02 14:52:30,440::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30466::DEBUG::2012-12-02 14:52:30,441::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30466::DEBUG::2012-12-02 14:52:30,441::task::957::TaskManager.Task::(_decref) Task=`4cc32513-3bc9-4529-831a-88a5dd435c59`::ref 0 aborting False >Thread-30467::DEBUG::2012-12-02 14:52:30,443::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30467::DEBUG::2012-12-02 14:52:30,444::task::568::TaskManager.Task::(_updateState) Task=`119d27fa-25ff-43ac-ab5f-fdccd341b289`::moving from state init -> state preparing >Thread-30467::INFO::2012-12-02 14:52:30,444::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30467::DEBUG::2012-12-02 14:52:30,444::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cf26bef1-f770-4a38-af2f-b19dd8c55b5f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30467::DEBUG::2012-12-02 14:52:30,444::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30467::DEBUG::2012-12-02 14:52:30,444::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30467::DEBUG::2012-12-02 14:52:30,444::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`cf26bef1-f770-4a38-af2f-b19dd8c55b5f`::Granted request >Thread-30467::DEBUG::2012-12-02 14:52:30,445::task::794::TaskManager.Task::(resourceAcquired) Task=`119d27fa-25ff-43ac-ab5f-fdccd341b289`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30467::DEBUG::2012-12-02 14:52:30,445::task::957::TaskManager.Task::(_decref) Task=`119d27fa-25ff-43ac-ab5f-fdccd341b289`::ref 1 aborting False >Thread-30467::INFO::2012-12-02 14:52:30,449::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30467::DEBUG::2012-12-02 14:52:30,449::task::1151::TaskManager.Task::(prepare) Task=`119d27fa-25ff-43ac-ab5f-fdccd341b289`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30467::DEBUG::2012-12-02 14:52:30,449::task::568::TaskManager.Task::(_updateState) Task=`119d27fa-25ff-43ac-ab5f-fdccd341b289`::moving from state preparing -> state finished >Thread-30467::DEBUG::2012-12-02 14:52:30,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30467::DEBUG::2012-12-02 14:52:30,450::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30467::DEBUG::2012-12-02 14:52:30,450::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30467::DEBUG::2012-12-02 14:52:30,450::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30467::DEBUG::2012-12-02 14:52:30,450::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30467::DEBUG::2012-12-02 14:52:30,450::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30467::DEBUG::2012-12-02 14:52:30,450::task::957::TaskManager.Task::(_decref) Task=`119d27fa-25ff-43ac-ab5f-fdccd341b289`::ref 0 aborting False >Thread-30472::DEBUG::2012-12-02 14:52:40,386::task::568::TaskManager.Task::(_updateState) Task=`956fe0a1-2b0e-49c1-b7b5-115f25c2252d`::moving from state init -> state preparing >Thread-30472::INFO::2012-12-02 14:52:40,386::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30472::INFO::2012-12-02 14:52:40,386::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00725293159485', 'lastCheck': 1354452751.8914249, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00482892990112', 'lastCheck': 1354452758.4597111, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00336003303528', 'lastCheck': 1354452758.7219651, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00734305381775', 'lastCheck': 1354452758.7677381, 'code': 0, 'valid': True}} >Thread-30472::DEBUG::2012-12-02 14:52:40,386::task::1151::TaskManager.Task::(prepare) Task=`956fe0a1-2b0e-49c1-b7b5-115f25c2252d`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00725293159485', 'lastCheck': 1354452751.8914249, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00482892990112', 'lastCheck': 1354452758.4597111, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00336003303528', 'lastCheck': 1354452758.7219651, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00734305381775', 'lastCheck': 1354452758.7677381, 'code': 0, 'valid': True}} >Thread-30472::DEBUG::2012-12-02 14:52:40,387::task::568::TaskManager.Task::(_updateState) Task=`956fe0a1-2b0e-49c1-b7b5-115f25c2252d`::moving from state preparing -> state finished >Thread-30472::DEBUG::2012-12-02 14:52:40,387::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30472::DEBUG::2012-12-02 14:52:40,387::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30472::DEBUG::2012-12-02 14:52:40,387::task::957::TaskManager.Task::(_decref) Task=`956fe0a1-2b0e-49c1-b7b5-115f25c2252d`::ref 0 aborting False >Thread-30474::DEBUG::2012-12-02 14:52:40,544::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30474::DEBUG::2012-12-02 14:52:40,544::task::568::TaskManager.Task::(_updateState) Task=`e6f00dc8-52da-472f-a802-fbfdce121bfd`::moving from state init -> state preparing >Thread-30474::INFO::2012-12-02 14:52:40,545::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30474::INFO::2012-12-02 14:52:40,545::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30474::DEBUG::2012-12-02 14:52:40,545::task::1151::TaskManager.Task::(prepare) Task=`e6f00dc8-52da-472f-a802-fbfdce121bfd`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30474::DEBUG::2012-12-02 14:52:40,545::task::568::TaskManager.Task::(_updateState) Task=`e6f00dc8-52da-472f-a802-fbfdce121bfd`::moving from state preparing -> state finished >Thread-30474::DEBUG::2012-12-02 14:52:40,545::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30474::DEBUG::2012-12-02 14:52:40,545::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30474::DEBUG::2012-12-02 14:52:40,545::task::957::TaskManager.Task::(_decref) Task=`e6f00dc8-52da-472f-a802-fbfdce121bfd`::ref 0 aborting False >Thread-30475::DEBUG::2012-12-02 14:52:40,548::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30475::DEBUG::2012-12-02 14:52:40,548::task::568::TaskManager.Task::(_updateState) Task=`51a22391-5f8f-4909-8b3d-d0c2e4afd617`::moving from state init -> state preparing >Thread-30475::INFO::2012-12-02 14:52:40,548::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30475::DEBUG::2012-12-02 14:52:40,548::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`780448fd-7c79-4f07-a1a4-05984a134cb8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30475::DEBUG::2012-12-02 14:52:40,548::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30475::DEBUG::2012-12-02 14:52:40,548::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30475::DEBUG::2012-12-02 14:52:40,549::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`780448fd-7c79-4f07-a1a4-05984a134cb8`::Granted request >Thread-30475::DEBUG::2012-12-02 14:52:40,549::task::794::TaskManager.Task::(resourceAcquired) Task=`51a22391-5f8f-4909-8b3d-d0c2e4afd617`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30475::DEBUG::2012-12-02 14:52:40,549::task::957::TaskManager.Task::(_decref) Task=`51a22391-5f8f-4909-8b3d-d0c2e4afd617`::ref 1 aborting False >Thread-30475::INFO::2012-12-02 14:52:40,553::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30475::DEBUG::2012-12-02 14:52:40,553::task::1151::TaskManager.Task::(prepare) Task=`51a22391-5f8f-4909-8b3d-d0c2e4afd617`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30475::DEBUG::2012-12-02 14:52:40,553::task::568::TaskManager.Task::(_updateState) Task=`51a22391-5f8f-4909-8b3d-d0c2e4afd617`::moving from state preparing -> state finished >Thread-30475::DEBUG::2012-12-02 14:52:40,553::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30475::DEBUG::2012-12-02 14:52:40,554::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30475::DEBUG::2012-12-02 14:52:40,554::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30475::DEBUG::2012-12-02 14:52:40,554::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30475::DEBUG::2012-12-02 14:52:40,554::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30475::DEBUG::2012-12-02 14:52:40,554::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30475::DEBUG::2012-12-02 14:52:40,554::task::957::TaskManager.Task::(_decref) Task=`51a22391-5f8f-4909-8b3d-d0c2e4afd617`::ref 0 aborting False >Thread-30480::DEBUG::2012-12-02 14:52:50,492::task::568::TaskManager.Task::(_updateState) Task=`243fdba7-a446-4eb5-9a62-9de4f0f4e63c`::moving from state init -> state preparing >Thread-30480::INFO::2012-12-02 14:52:50,492::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30480::INFO::2012-12-02 14:52:50,492::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00949001312256', 'lastCheck': 1354452761.9133511, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00201392173767', 'lastCheck': 1354452768.4747379, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00222396850586', 'lastCheck': 1354452768.732209, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00799512863159', 'lastCheck': 1354452768.786679, 'code': 0, 'valid': True}} >Thread-30480::DEBUG::2012-12-02 14:52:50,492::task::1151::TaskManager.Task::(prepare) Task=`243fdba7-a446-4eb5-9a62-9de4f0f4e63c`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00949001312256', 'lastCheck': 1354452761.9133511, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00201392173767', 'lastCheck': 1354452768.4747379, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00222396850586', 'lastCheck': 1354452768.732209, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00799512863159', 'lastCheck': 1354452768.786679, 'code': 0, 'valid': True}} >Thread-30480::DEBUG::2012-12-02 14:52:50,493::task::568::TaskManager.Task::(_updateState) Task=`243fdba7-a446-4eb5-9a62-9de4f0f4e63c`::moving from state preparing -> state finished >Thread-30480::DEBUG::2012-12-02 14:52:50,493::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30480::DEBUG::2012-12-02 14:52:50,493::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30480::DEBUG::2012-12-02 14:52:50,493::task::957::TaskManager.Task::(_decref) Task=`243fdba7-a446-4eb5-9a62-9de4f0f4e63c`::ref 0 aborting False >Thread-30482::DEBUG::2012-12-02 14:52:50,642::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30482::DEBUG::2012-12-02 14:52:50,642::task::568::TaskManager.Task::(_updateState) Task=`b2cea693-f5aa-4dd3-9f5b-37f51661afb2`::moving from state init -> state preparing >Thread-30482::INFO::2012-12-02 14:52:50,642::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30482::INFO::2012-12-02 14:52:50,642::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30482::DEBUG::2012-12-02 14:52:50,642::task::1151::TaskManager.Task::(prepare) Task=`b2cea693-f5aa-4dd3-9f5b-37f51661afb2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30482::DEBUG::2012-12-02 14:52:50,643::task::568::TaskManager.Task::(_updateState) Task=`b2cea693-f5aa-4dd3-9f5b-37f51661afb2`::moving from state preparing -> state finished >Thread-30482::DEBUG::2012-12-02 14:52:50,643::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30482::DEBUG::2012-12-02 14:52:50,643::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30482::DEBUG::2012-12-02 14:52:50,643::task::957::TaskManager.Task::(_decref) Task=`b2cea693-f5aa-4dd3-9f5b-37f51661afb2`::ref 0 aborting False >Thread-30483::DEBUG::2012-12-02 14:52:50,645::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30483::DEBUG::2012-12-02 14:52:50,646::task::568::TaskManager.Task::(_updateState) Task=`de29ffa9-6b21-440f-b590-d26d84a2bd74`::moving from state init -> state preparing >Thread-30483::INFO::2012-12-02 14:52:50,646::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30483::DEBUG::2012-12-02 14:52:50,646::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`97db923e-d6ee-45c5-aa7c-b31e93c9b1c7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30483::DEBUG::2012-12-02 14:52:50,646::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30483::DEBUG::2012-12-02 14:52:50,646::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30483::DEBUG::2012-12-02 14:52:50,646::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`97db923e-d6ee-45c5-aa7c-b31e93c9b1c7`::Granted request >Thread-30483::DEBUG::2012-12-02 14:52:50,647::task::794::TaskManager.Task::(resourceAcquired) Task=`de29ffa9-6b21-440f-b590-d26d84a2bd74`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30483::DEBUG::2012-12-02 14:52:50,647::task::957::TaskManager.Task::(_decref) Task=`de29ffa9-6b21-440f-b590-d26d84a2bd74`::ref 1 aborting False >Thread-30483::INFO::2012-12-02 14:52:50,653::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30483::DEBUG::2012-12-02 14:52:50,653::task::1151::TaskManager.Task::(prepare) Task=`de29ffa9-6b21-440f-b590-d26d84a2bd74`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30483::DEBUG::2012-12-02 14:52:50,654::task::568::TaskManager.Task::(_updateState) Task=`de29ffa9-6b21-440f-b590-d26d84a2bd74`::moving from state preparing -> state finished >Thread-30483::DEBUG::2012-12-02 14:52:50,654::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30483::DEBUG::2012-12-02 14:52:50,654::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30483::DEBUG::2012-12-02 14:52:50,654::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30483::DEBUG::2012-12-02 14:52:50,654::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30483::DEBUG::2012-12-02 14:52:50,654::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30483::DEBUG::2012-12-02 14:52:50,654::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30483::DEBUG::2012-12-02 14:52:50,655::task::957::TaskManager.Task::(_decref) Task=`de29ffa9-6b21-440f-b590-d26d84a2bd74`::ref 0 aborting False >Thread-30488::DEBUG::2012-12-02 14:53:00,577::task::568::TaskManager.Task::(_updateState) Task=`ebf99493-8a08-47c6-9e21-1f5eb3ce9163`::moving from state init -> state preparing >Thread-30488::INFO::2012-12-02 14:53:00,578::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30488::INFO::2012-12-02 14:53:00,578::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00958490371704', 'lastCheck': 1354452771.932471, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00146007537842', 'lastCheck': 1354452778.4852459, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00202107429504', 'lastCheck': 1354452778.743952, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00746393203735', 'lastCheck': 1354452778.803164, 'code': 0, 'valid': True}} >Thread-30488::DEBUG::2012-12-02 14:53:00,578::task::1151::TaskManager.Task::(prepare) Task=`ebf99493-8a08-47c6-9e21-1f5eb3ce9163`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00958490371704', 'lastCheck': 1354452771.932471, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00146007537842', 'lastCheck': 1354452778.4852459, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00202107429504', 'lastCheck': 1354452778.743952, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00746393203735', 'lastCheck': 1354452778.803164, 'code': 0, 'valid': True}} >Thread-30488::DEBUG::2012-12-02 14:53:00,578::task::568::TaskManager.Task::(_updateState) Task=`ebf99493-8a08-47c6-9e21-1f5eb3ce9163`::moving from state preparing -> state finished >Thread-30488::DEBUG::2012-12-02 14:53:00,578::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30488::DEBUG::2012-12-02 14:53:00,578::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30488::DEBUG::2012-12-02 14:53:00,579::task::957::TaskManager.Task::(_decref) Task=`ebf99493-8a08-47c6-9e21-1f5eb3ce9163`::ref 0 aborting False >Thread-30490::DEBUG::2012-12-02 14:53:00,746::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30490::DEBUG::2012-12-02 14:53:00,746::task::568::TaskManager.Task::(_updateState) Task=`571fc1b7-210a-4586-a008-8538710d5736`::moving from state init -> state preparing >Thread-30490::INFO::2012-12-02 14:53:00,746::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30490::INFO::2012-12-02 14:53:00,746::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30490::DEBUG::2012-12-02 14:53:00,746::task::1151::TaskManager.Task::(prepare) Task=`571fc1b7-210a-4586-a008-8538710d5736`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30490::DEBUG::2012-12-02 14:53:00,747::task::568::TaskManager.Task::(_updateState) Task=`571fc1b7-210a-4586-a008-8538710d5736`::moving from state preparing -> state finished >Thread-30490::DEBUG::2012-12-02 14:53:00,747::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30490::DEBUG::2012-12-02 14:53:00,747::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30490::DEBUG::2012-12-02 14:53:00,747::task::957::TaskManager.Task::(_decref) Task=`571fc1b7-210a-4586-a008-8538710d5736`::ref 0 aborting False >Thread-30491::DEBUG::2012-12-02 14:53:00,749::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30491::DEBUG::2012-12-02 14:53:00,750::task::568::TaskManager.Task::(_updateState) Task=`c070348a-ab56-4db6-8f0f-557bc9053b2a`::moving from state init -> state preparing >Thread-30491::INFO::2012-12-02 14:53:00,750::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30491::DEBUG::2012-12-02 14:53:00,750::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7ef6c47a-98cd-484f-8590-25ebc544ceaf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30491::DEBUG::2012-12-02 14:53:00,750::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30491::DEBUG::2012-12-02 14:53:00,751::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30491::DEBUG::2012-12-02 14:53:00,751::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7ef6c47a-98cd-484f-8590-25ebc544ceaf`::Granted request >Thread-30491::DEBUG::2012-12-02 14:53:00,752::task::794::TaskManager.Task::(resourceAcquired) Task=`c070348a-ab56-4db6-8f0f-557bc9053b2a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30491::DEBUG::2012-12-02 14:53:00,752::task::957::TaskManager.Task::(_decref) Task=`c070348a-ab56-4db6-8f0f-557bc9053b2a`::ref 1 aborting False >Thread-30491::INFO::2012-12-02 14:53:00,757::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30491::DEBUG::2012-12-02 14:53:00,757::task::1151::TaskManager.Task::(prepare) Task=`c070348a-ab56-4db6-8f0f-557bc9053b2a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30491::DEBUG::2012-12-02 14:53:00,757::task::568::TaskManager.Task::(_updateState) Task=`c070348a-ab56-4db6-8f0f-557bc9053b2a`::moving from state preparing -> state finished >Thread-30491::DEBUG::2012-12-02 14:53:00,758::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30491::DEBUG::2012-12-02 14:53:00,758::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30491::DEBUG::2012-12-02 14:53:00,758::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30491::DEBUG::2012-12-02 14:53:00,758::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30491::DEBUG::2012-12-02 14:53:00,758::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30491::DEBUG::2012-12-02 14:53:00,758::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30491::DEBUG::2012-12-02 14:53:00,758::task::957::TaskManager.Task::(_decref) Task=`c070348a-ab56-4db6-8f0f-557bc9053b2a`::ref 0 aborting False >Thread-30496::DEBUG::2012-12-02 14:53:10,675::task::568::TaskManager.Task::(_updateState) Task=`8c98c773-a4f5-42db-9e20-10524f2c4166`::moving from state init -> state preparing >Thread-30496::INFO::2012-12-02 14:53:10,675::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30496::INFO::2012-12-02 14:53:10,675::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00869989395142', 'lastCheck': 1354452781.9501359, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00253510475159', 'lastCheck': 1354452788.4986849, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191903114319', 'lastCheck': 1354452788.7550311, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00877785682678', 'lastCheck': 1354452788.8208251, 'code': 0, 'valid': True}} >Thread-30496::DEBUG::2012-12-02 14:53:10,675::task::1151::TaskManager.Task::(prepare) Task=`8c98c773-a4f5-42db-9e20-10524f2c4166`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00869989395142', 'lastCheck': 1354452781.9501359, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00253510475159', 'lastCheck': 1354452788.4986849, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191903114319', 'lastCheck': 1354452788.7550311, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00877785682678', 'lastCheck': 1354452788.8208251, 'code': 0, 'valid': True}} >Thread-30496::DEBUG::2012-12-02 14:53:10,675::task::568::TaskManager.Task::(_updateState) Task=`8c98c773-a4f5-42db-9e20-10524f2c4166`::moving from state preparing -> state finished >Thread-30496::DEBUG::2012-12-02 14:53:10,675::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30496::DEBUG::2012-12-02 14:53:10,675::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30496::DEBUG::2012-12-02 14:53:10,676::task::957::TaskManager.Task::(_decref) Task=`8c98c773-a4f5-42db-9e20-10524f2c4166`::ref 0 aborting False >Thread-30498::DEBUG::2012-12-02 14:53:10,859::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30498::DEBUG::2012-12-02 14:53:10,859::task::568::TaskManager.Task::(_updateState) Task=`56e64ce0-d51d-487a-b8c9-bfd71252e821`::moving from state init -> state preparing >Thread-30498::INFO::2012-12-02 14:53:10,859::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30498::INFO::2012-12-02 14:53:10,860::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30498::DEBUG::2012-12-02 14:53:10,860::task::1151::TaskManager.Task::(prepare) Task=`56e64ce0-d51d-487a-b8c9-bfd71252e821`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30498::DEBUG::2012-12-02 14:53:10,860::task::568::TaskManager.Task::(_updateState) Task=`56e64ce0-d51d-487a-b8c9-bfd71252e821`::moving from state preparing -> state finished >Thread-30498::DEBUG::2012-12-02 14:53:10,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30498::DEBUG::2012-12-02 14:53:10,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30498::DEBUG::2012-12-02 14:53:10,860::task::957::TaskManager.Task::(_decref) Task=`56e64ce0-d51d-487a-b8c9-bfd71252e821`::ref 0 aborting False >Thread-30499::DEBUG::2012-12-02 14:53:10,863::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30499::DEBUG::2012-12-02 14:53:10,863::task::568::TaskManager.Task::(_updateState) Task=`8bfcbb7d-8820-4dc7-ba55-dc02b1b5b5c9`::moving from state init -> state preparing >Thread-30499::INFO::2012-12-02 14:53:10,863::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30499::DEBUG::2012-12-02 14:53:10,863::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6347b816-b029-4952-a050-5d420e2a6e83`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30499::DEBUG::2012-12-02 14:53:10,864::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30499::DEBUG::2012-12-02 14:53:10,864::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30499::DEBUG::2012-12-02 14:53:10,864::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6347b816-b029-4952-a050-5d420e2a6e83`::Granted request >Thread-30499::DEBUG::2012-12-02 14:53:10,864::task::794::TaskManager.Task::(resourceAcquired) Task=`8bfcbb7d-8820-4dc7-ba55-dc02b1b5b5c9`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30499::DEBUG::2012-12-02 14:53:10,864::task::957::TaskManager.Task::(_decref) Task=`8bfcbb7d-8820-4dc7-ba55-dc02b1b5b5c9`::ref 1 aborting False >Thread-30499::INFO::2012-12-02 14:53:10,869::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30499::DEBUG::2012-12-02 14:53:10,869::task::1151::TaskManager.Task::(prepare) Task=`8bfcbb7d-8820-4dc7-ba55-dc02b1b5b5c9`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30499::DEBUG::2012-12-02 14:53:10,869::task::568::TaskManager.Task::(_updateState) Task=`8bfcbb7d-8820-4dc7-ba55-dc02b1b5b5c9`::moving from state preparing -> state finished >Thread-30499::DEBUG::2012-12-02 14:53:10,869::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30499::DEBUG::2012-12-02 14:53:10,870::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30499::DEBUG::2012-12-02 14:53:10,870::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30499::DEBUG::2012-12-02 14:53:10,870::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30499::DEBUG::2012-12-02 14:53:10,870::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30499::DEBUG::2012-12-02 14:53:10,870::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30499::DEBUG::2012-12-02 14:53:10,870::task::957::TaskManager.Task::(_decref) Task=`8bfcbb7d-8820-4dc7-ba55-dc02b1b5b5c9`::ref 0 aborting False >Thread-30504::DEBUG::2012-12-02 14:53:20,780::task::568::TaskManager.Task::(_updateState) Task=`eaf7020a-ae32-4e09-a4eb-d83e5418ce50`::moving from state init -> state preparing >Thread-30504::INFO::2012-12-02 14:53:20,780::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30504::INFO::2012-12-02 14:53:20,780::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00855994224548', 'lastCheck': 1354452791.968353, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00236511230469', 'lastCheck': 1354452798.51386, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00338506698608', 'lastCheck': 1354452798.768389, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00701117515564', 'lastCheck': 1354452798.836751, 'code': 0, 'valid': True}} >Thread-30504::DEBUG::2012-12-02 14:53:20,780::task::1151::TaskManager.Task::(prepare) Task=`eaf7020a-ae32-4e09-a4eb-d83e5418ce50`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00855994224548', 'lastCheck': 1354452791.968353, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00236511230469', 'lastCheck': 1354452798.51386, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00338506698608', 'lastCheck': 1354452798.768389, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00701117515564', 'lastCheck': 1354452798.836751, 'code': 0, 'valid': True}} >Thread-30504::DEBUG::2012-12-02 14:53:20,781::task::568::TaskManager.Task::(_updateState) Task=`eaf7020a-ae32-4e09-a4eb-d83e5418ce50`::moving from state preparing -> state finished >Thread-30504::DEBUG::2012-12-02 14:53:20,781::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30504::DEBUG::2012-12-02 14:53:20,781::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30504::DEBUG::2012-12-02 14:53:20,781::task::957::TaskManager.Task::(_decref) Task=`eaf7020a-ae32-4e09-a4eb-d83e5418ce50`::ref 0 aborting False >Thread-30506::DEBUG::2012-12-02 14:53:20,971::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30506::DEBUG::2012-12-02 14:53:20,971::task::568::TaskManager.Task::(_updateState) Task=`ceb6a1a1-302a-4c18-95e0-04cec7f9ce4a`::moving from state init -> state preparing >Thread-30506::INFO::2012-12-02 14:53:20,971::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30506::INFO::2012-12-02 14:53:20,972::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30506::DEBUG::2012-12-02 14:53:20,972::task::1151::TaskManager.Task::(prepare) Task=`ceb6a1a1-302a-4c18-95e0-04cec7f9ce4a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30506::DEBUG::2012-12-02 14:53:20,972::task::568::TaskManager.Task::(_updateState) Task=`ceb6a1a1-302a-4c18-95e0-04cec7f9ce4a`::moving from state preparing -> state finished >Thread-30506::DEBUG::2012-12-02 14:53:20,972::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30506::DEBUG::2012-12-02 14:53:20,972::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30506::DEBUG::2012-12-02 14:53:20,972::task::957::TaskManager.Task::(_decref) Task=`ceb6a1a1-302a-4c18-95e0-04cec7f9ce4a`::ref 0 aborting False >Thread-30507::DEBUG::2012-12-02 14:53:20,975::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30507::DEBUG::2012-12-02 14:53:20,975::task::568::TaskManager.Task::(_updateState) Task=`2a0d7760-f758-4414-b7c7-502b6d52c243`::moving from state init -> state preparing >Thread-30507::INFO::2012-12-02 14:53:20,975::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30507::DEBUG::2012-12-02 14:53:20,975::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`24a911fe-85fc-42de-9277-b3bef6fb838f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30507::DEBUG::2012-12-02 14:53:20,975::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30507::DEBUG::2012-12-02 14:53:20,975::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30507::DEBUG::2012-12-02 14:53:20,976::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`24a911fe-85fc-42de-9277-b3bef6fb838f`::Granted request >Thread-30507::DEBUG::2012-12-02 14:53:20,976::task::794::TaskManager.Task::(resourceAcquired) Task=`2a0d7760-f758-4414-b7c7-502b6d52c243`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30507::DEBUG::2012-12-02 14:53:20,976::task::957::TaskManager.Task::(_decref) Task=`2a0d7760-f758-4414-b7c7-502b6d52c243`::ref 1 aborting False >Thread-30507::INFO::2012-12-02 14:53:20,980::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30507::DEBUG::2012-12-02 14:53:20,980::task::1151::TaskManager.Task::(prepare) Task=`2a0d7760-f758-4414-b7c7-502b6d52c243`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30507::DEBUG::2012-12-02 14:53:20,980::task::568::TaskManager.Task::(_updateState) Task=`2a0d7760-f758-4414-b7c7-502b6d52c243`::moving from state preparing -> state finished >Thread-30507::DEBUG::2012-12-02 14:53:20,980::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30507::DEBUG::2012-12-02 14:53:20,980::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30507::DEBUG::2012-12-02 14:53:20,981::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30507::DEBUG::2012-12-02 14:53:20,981::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30507::DEBUG::2012-12-02 14:53:20,981::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30507::DEBUG::2012-12-02 14:53:20,981::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30507::DEBUG::2012-12-02 14:53:20,981::task::957::TaskManager.Task::(_decref) Task=`2a0d7760-f758-4414-b7c7-502b6d52c243`::ref 0 aborting False >Thread-30512::DEBUG::2012-12-02 14:53:30,903::task::568::TaskManager.Task::(_updateState) Task=`63c46be7-c948-441a-9143-31d7884b5842`::moving from state init -> state preparing >Thread-30512::INFO::2012-12-02 14:53:30,903::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30512::INFO::2012-12-02 14:53:30,903::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0080509185791', 'lastCheck': 1354452801.9860139, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00195217132568', 'lastCheck': 1354452808.528909, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00210404396057', 'lastCheck': 1354452808.780489, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00768494606018', 'lastCheck': 1354452808.8533931, 'code': 0, 'valid': True}} >Thread-30512::DEBUG::2012-12-02 14:53:30,904::task::1151::TaskManager.Task::(prepare) Task=`63c46be7-c948-441a-9143-31d7884b5842`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0080509185791', 'lastCheck': 1354452801.9860139, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00195217132568', 'lastCheck': 1354452808.528909, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00210404396057', 'lastCheck': 1354452808.780489, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00768494606018', 'lastCheck': 1354452808.8533931, 'code': 0, 'valid': True}} >Thread-30512::DEBUG::2012-12-02 14:53:30,904::task::568::TaskManager.Task::(_updateState) Task=`63c46be7-c948-441a-9143-31d7884b5842`::moving from state preparing -> state finished >Thread-30512::DEBUG::2012-12-02 14:53:30,904::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30512::DEBUG::2012-12-02 14:53:30,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30512::DEBUG::2012-12-02 14:53:30,904::task::957::TaskManager.Task::(_decref) Task=`63c46be7-c948-441a-9143-31d7884b5842`::ref 0 aborting False >Thread-30514::DEBUG::2012-12-02 14:53:31,133::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30514::DEBUG::2012-12-02 14:53:31,134::task::568::TaskManager.Task::(_updateState) Task=`84741a31-5ba8-4884-ab25-afd7e7f4d8b7`::moving from state init -> state preparing >Thread-30514::INFO::2012-12-02 14:53:31,134::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30514::INFO::2012-12-02 14:53:31,134::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30514::DEBUG::2012-12-02 14:53:31,134::task::1151::TaskManager.Task::(prepare) Task=`84741a31-5ba8-4884-ab25-afd7e7f4d8b7`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30514::DEBUG::2012-12-02 14:53:31,134::task::568::TaskManager.Task::(_updateState) Task=`84741a31-5ba8-4884-ab25-afd7e7f4d8b7`::moving from state preparing -> state finished >Thread-30514::DEBUG::2012-12-02 14:53:31,134::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30514::DEBUG::2012-12-02 14:53:31,134::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30514::DEBUG::2012-12-02 14:53:31,135::task::957::TaskManager.Task::(_decref) Task=`84741a31-5ba8-4884-ab25-afd7e7f4d8b7`::ref 0 aborting False >Thread-30515::DEBUG::2012-12-02 14:53:31,137::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30515::DEBUG::2012-12-02 14:53:31,137::task::568::TaskManager.Task::(_updateState) Task=`61c5fc20-9137-4f98-9911-a4e5d640575b`::moving from state init -> state preparing >Thread-30515::INFO::2012-12-02 14:53:31,137::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30515::DEBUG::2012-12-02 14:53:31,138::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`77ce2fbd-8d64-4893-b72a-1279fdb53e14`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30515::DEBUG::2012-12-02 14:53:31,138::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30515::DEBUG::2012-12-02 14:53:31,138::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30515::DEBUG::2012-12-02 14:53:31,138::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`77ce2fbd-8d64-4893-b72a-1279fdb53e14`::Granted request >Thread-30515::DEBUG::2012-12-02 14:53:31,138::task::794::TaskManager.Task::(resourceAcquired) Task=`61c5fc20-9137-4f98-9911-a4e5d640575b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30515::DEBUG::2012-12-02 14:53:31,138::task::957::TaskManager.Task::(_decref) Task=`61c5fc20-9137-4f98-9911-a4e5d640575b`::ref 1 aborting False >Thread-30515::INFO::2012-12-02 14:53:31,142::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30515::DEBUG::2012-12-02 14:53:31,142::task::1151::TaskManager.Task::(prepare) Task=`61c5fc20-9137-4f98-9911-a4e5d640575b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30515::DEBUG::2012-12-02 14:53:31,143::task::568::TaskManager.Task::(_updateState) Task=`61c5fc20-9137-4f98-9911-a4e5d640575b`::moving from state preparing -> state finished >Thread-30515::DEBUG::2012-12-02 14:53:31,143::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30515::DEBUG::2012-12-02 14:53:31,143::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30515::DEBUG::2012-12-02 14:53:31,143::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30515::DEBUG::2012-12-02 14:53:31,143::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30515::DEBUG::2012-12-02 14:53:31,143::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30515::DEBUG::2012-12-02 14:53:31,143::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30515::DEBUG::2012-12-02 14:53:31,143::task::957::TaskManager.Task::(_decref) Task=`61c5fc20-9137-4f98-9911-a4e5d640575b`::ref 0 aborting False >Thread-30520::DEBUG::2012-12-02 14:53:41,012::task::568::TaskManager.Task::(_updateState) Task=`91626b8c-52fa-48d8-9e7e-7cc8b86bc9f3`::moving from state init -> state preparing >Thread-30520::INFO::2012-12-02 14:53:41,012::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30520::INFO::2012-12-02 14:53:41,012::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830221176147', 'lastCheck': 1354452812.0045381, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00352597236633', 'lastCheck': 1354452818.5429299, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00309205055237', 'lastCheck': 1354452818.7933991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00681710243225', 'lastCheck': 1354452818.867821, 'code': 0, 'valid': True}} >Thread-30520::DEBUG::2012-12-02 14:53:41,013::task::1151::TaskManager.Task::(prepare) Task=`91626b8c-52fa-48d8-9e7e-7cc8b86bc9f3`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00830221176147', 'lastCheck': 1354452812.0045381, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00352597236633', 'lastCheck': 1354452818.5429299, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00309205055237', 'lastCheck': 1354452818.7933991, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00681710243225', 'lastCheck': 1354452818.867821, 'code': 0, 'valid': True}} >Thread-30520::DEBUG::2012-12-02 14:53:41,013::task::568::TaskManager.Task::(_updateState) Task=`91626b8c-52fa-48d8-9e7e-7cc8b86bc9f3`::moving from state preparing -> state finished >Thread-30520::DEBUG::2012-12-02 14:53:41,013::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30520::DEBUG::2012-12-02 14:53:41,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30520::DEBUG::2012-12-02 14:53:41,013::task::957::TaskManager.Task::(_decref) Task=`91626b8c-52fa-48d8-9e7e-7cc8b86bc9f3`::ref 0 aborting False >Thread-30522::DEBUG::2012-12-02 14:53:41,247::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30522::DEBUG::2012-12-02 14:53:41,247::task::568::TaskManager.Task::(_updateState) Task=`2a380dc8-53ea-4819-9a0e-fa067757e148`::moving from state init -> state preparing >Thread-30522::INFO::2012-12-02 14:53:41,247::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30522::INFO::2012-12-02 14:53:41,247::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30522::DEBUG::2012-12-02 14:53:41,248::task::1151::TaskManager.Task::(prepare) Task=`2a380dc8-53ea-4819-9a0e-fa067757e148`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30522::DEBUG::2012-12-02 14:53:41,248::task::568::TaskManager.Task::(_updateState) Task=`2a380dc8-53ea-4819-9a0e-fa067757e148`::moving from state preparing -> state finished >Thread-30522::DEBUG::2012-12-02 14:53:41,248::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30522::DEBUG::2012-12-02 14:53:41,248::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30522::DEBUG::2012-12-02 14:53:41,248::task::957::TaskManager.Task::(_decref) Task=`2a380dc8-53ea-4819-9a0e-fa067757e148`::ref 0 aborting False >Thread-30523::DEBUG::2012-12-02 14:53:41,251::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30523::DEBUG::2012-12-02 14:53:41,251::task::568::TaskManager.Task::(_updateState) Task=`dd44cef4-7edd-41f8-9b7d-08e0224ab42b`::moving from state init -> state preparing >Thread-30523::INFO::2012-12-02 14:53:41,251::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30523::DEBUG::2012-12-02 14:53:41,251::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d088c7db-6fc9-48a1-879a-1943c5d52b10`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30523::DEBUG::2012-12-02 14:53:41,251::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30523::DEBUG::2012-12-02 14:53:41,252::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30523::DEBUG::2012-12-02 14:53:41,252::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`d088c7db-6fc9-48a1-879a-1943c5d52b10`::Granted request >Thread-30523::DEBUG::2012-12-02 14:53:41,252::task::794::TaskManager.Task::(resourceAcquired) Task=`dd44cef4-7edd-41f8-9b7d-08e0224ab42b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30523::DEBUG::2012-12-02 14:53:41,252::task::957::TaskManager.Task::(_decref) Task=`dd44cef4-7edd-41f8-9b7d-08e0224ab42b`::ref 1 aborting False >Thread-30523::INFO::2012-12-02 14:53:41,257::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30523::DEBUG::2012-12-02 14:53:41,258::task::1151::TaskManager.Task::(prepare) Task=`dd44cef4-7edd-41f8-9b7d-08e0224ab42b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873312813056', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30523::DEBUG::2012-12-02 14:53:41,258::task::568::TaskManager.Task::(_updateState) Task=`dd44cef4-7edd-41f8-9b7d-08e0224ab42b`::moving from state preparing -> state finished >Thread-30523::DEBUG::2012-12-02 14:53:41,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30523::DEBUG::2012-12-02 14:53:41,258::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30523::DEBUG::2012-12-02 14:53:41,258::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30523::DEBUG::2012-12-02 14:53:41,258::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30523::DEBUG::2012-12-02 14:53:41,258::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30523::DEBUG::2012-12-02 14:53:41,258::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30523::DEBUG::2012-12-02 14:53:41,258::task::957::TaskManager.Task::(_decref) Task=`dd44cef4-7edd-41f8-9b7d-08e0224ab42b`::ref 0 aborting False >Thread-30528::DEBUG::2012-12-02 14:53:51,150::task::568::TaskManager.Task::(_updateState) Task=`eb95b97e-077c-4aa1-9cfc-3e586fca3cb9`::moving from state init -> state preparing >Thread-30528::INFO::2012-12-02 14:53:51,150::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30528::INFO::2012-12-02 14:53:51,150::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0148360729218', 'lastCheck': 1354452822.036056, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206303596497', 'lastCheck': 1354452828.556103, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191593170166', 'lastCheck': 1354452828.804559, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809407234192', 'lastCheck': 1354452828.8870139, 'code': 0, 'valid': True}} >Thread-30528::DEBUG::2012-12-02 14:53:51,151::task::1151::TaskManager.Task::(prepare) Task=`eb95b97e-077c-4aa1-9cfc-3e586fca3cb9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0148360729218', 'lastCheck': 1354452822.036056, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206303596497', 'lastCheck': 1354452828.556103, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00191593170166', 'lastCheck': 1354452828.804559, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00809407234192', 'lastCheck': 1354452828.8870139, 'code': 0, 'valid': True}} >Thread-30528::DEBUG::2012-12-02 14:53:51,151::task::568::TaskManager.Task::(_updateState) Task=`eb95b97e-077c-4aa1-9cfc-3e586fca3cb9`::moving from state preparing -> state finished >Thread-30528::DEBUG::2012-12-02 14:53:51,151::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30528::DEBUG::2012-12-02 14:53:51,151::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30528::DEBUG::2012-12-02 14:53:51,151::task::957::TaskManager.Task::(_decref) Task=`eb95b97e-077c-4aa1-9cfc-3e586fca3cb9`::ref 0 aborting False >Thread-30530::DEBUG::2012-12-02 14:53:51,352::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30530::DEBUG::2012-12-02 14:53:51,352::task::568::TaskManager.Task::(_updateState) Task=`fc746b5c-fd88-4b25-b4fd-bc77f6baecb8`::moving from state init -> state preparing >Thread-30530::INFO::2012-12-02 14:53:51,352::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30530::INFO::2012-12-02 14:53:51,353::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30530::DEBUG::2012-12-02 14:53:51,353::task::1151::TaskManager.Task::(prepare) Task=`fc746b5c-fd88-4b25-b4fd-bc77f6baecb8`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30530::DEBUG::2012-12-02 14:53:51,353::task::568::TaskManager.Task::(_updateState) Task=`fc746b5c-fd88-4b25-b4fd-bc77f6baecb8`::moving from state preparing -> state finished >Thread-30530::DEBUG::2012-12-02 14:53:51,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30530::DEBUG::2012-12-02 14:53:51,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30530::DEBUG::2012-12-02 14:53:51,353::task::957::TaskManager.Task::(_decref) Task=`fc746b5c-fd88-4b25-b4fd-bc77f6baecb8`::ref 0 aborting False >Thread-30531::DEBUG::2012-12-02 14:53:51,356::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30531::DEBUG::2012-12-02 14:53:51,356::task::568::TaskManager.Task::(_updateState) Task=`a250f8fe-b728-4396-82e0-f97d81a6615b`::moving from state init -> state preparing >Thread-30531::INFO::2012-12-02 14:53:51,356::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30531::DEBUG::2012-12-02 14:53:51,356::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4710ba3c-6bfe-4877-98ca-0638411b4371`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30531::DEBUG::2012-12-02 14:53:51,357::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30531::DEBUG::2012-12-02 14:53:51,357::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30531::DEBUG::2012-12-02 14:53:51,357::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`4710ba3c-6bfe-4877-98ca-0638411b4371`::Granted request >Thread-30531::DEBUG::2012-12-02 14:53:51,357::task::794::TaskManager.Task::(resourceAcquired) Task=`a250f8fe-b728-4396-82e0-f97d81a6615b`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30531::DEBUG::2012-12-02 14:53:51,357::task::957::TaskManager.Task::(_decref) Task=`a250f8fe-b728-4396-82e0-f97d81a6615b`::ref 1 aborting False >Thread-30531::INFO::2012-12-02 14:53:51,364::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30531::DEBUG::2012-12-02 14:53:51,364::task::1151::TaskManager.Task::(prepare) Task=`a250f8fe-b728-4396-82e0-f97d81a6615b`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30531::DEBUG::2012-12-02 14:53:51,364::task::568::TaskManager.Task::(_updateState) Task=`a250f8fe-b728-4396-82e0-f97d81a6615b`::moving from state preparing -> state finished >Thread-30531::DEBUG::2012-12-02 14:53:51,364::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30531::DEBUG::2012-12-02 14:53:51,364::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30531::DEBUG::2012-12-02 14:53:51,365::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30531::DEBUG::2012-12-02 14:53:51,365::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30531::DEBUG::2012-12-02 14:53:51,365::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30531::DEBUG::2012-12-02 14:53:51,365::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30531::DEBUG::2012-12-02 14:53:51,365::task::957::TaskManager.Task::(_decref) Task=`a250f8fe-b728-4396-82e0-f97d81a6615b`::ref 0 aborting False >Thread-30536::DEBUG::2012-12-02 14:54:01,237::task::568::TaskManager.Task::(_updateState) Task=`a082f7e1-d6b8-4075-b7e7-82e9a0e7f3c7`::moving from state init -> state preparing >Thread-30536::INFO::2012-12-02 14:54:01,238::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30536::INFO::2012-12-02 14:54:01,238::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0121350288391', 'lastCheck': 1354452832.0585179, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00203704833984', 'lastCheck': 1354452838.56865, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022919178009', 'lastCheck': 1354452838.814682, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00796294212341', 'lastCheck': 1354452838.904567, 'code': 0, 'valid': True}} >Thread-30536::DEBUG::2012-12-02 14:54:01,238::task::1151::TaskManager.Task::(prepare) Task=`a082f7e1-d6b8-4075-b7e7-82e9a0e7f3c7`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.0121350288391', 'lastCheck': 1354452832.0585179, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00203704833984', 'lastCheck': 1354452838.56865, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.0022919178009', 'lastCheck': 1354452838.814682, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00796294212341', 'lastCheck': 1354452838.904567, 'code': 0, 'valid': True}} >Thread-30536::DEBUG::2012-12-02 14:54:01,238::task::568::TaskManager.Task::(_updateState) Task=`a082f7e1-d6b8-4075-b7e7-82e9a0e7f3c7`::moving from state preparing -> state finished >Thread-30536::DEBUG::2012-12-02 14:54:01,238::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30536::DEBUG::2012-12-02 14:54:01,238::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30536::DEBUG::2012-12-02 14:54:01,239::task::957::TaskManager.Task::(_decref) Task=`a082f7e1-d6b8-4075-b7e7-82e9a0e7f3c7`::ref 0 aborting False >Thread-30538::DEBUG::2012-12-02 14:54:01,455::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30538::DEBUG::2012-12-02 14:54:01,455::task::568::TaskManager.Task::(_updateState) Task=`e8aa39e1-e783-4ad6-ac9d-9b3b9d3f3a8b`::moving from state init -> state preparing >Thread-30538::INFO::2012-12-02 14:54:01,456::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30538::INFO::2012-12-02 14:54:01,456::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30538::DEBUG::2012-12-02 14:54:01,456::task::1151::TaskManager.Task::(prepare) Task=`e8aa39e1-e783-4ad6-ac9d-9b3b9d3f3a8b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30538::DEBUG::2012-12-02 14:54:01,456::task::568::TaskManager.Task::(_updateState) Task=`e8aa39e1-e783-4ad6-ac9d-9b3b9d3f3a8b`::moving from state preparing -> state finished >Thread-30538::DEBUG::2012-12-02 14:54:01,456::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30538::DEBUG::2012-12-02 14:54:01,456::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30538::DEBUG::2012-12-02 14:54:01,457::task::957::TaskManager.Task::(_decref) Task=`e8aa39e1-e783-4ad6-ac9d-9b3b9d3f3a8b`::ref 0 aborting False >Thread-30539::DEBUG::2012-12-02 14:54:01,459::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30539::DEBUG::2012-12-02 14:54:01,459::task::568::TaskManager.Task::(_updateState) Task=`b27d7219-8fba-4a45-bfc6-e7a241c203c7`::moving from state init -> state preparing >Thread-30539::INFO::2012-12-02 14:54:01,459::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30539::DEBUG::2012-12-02 14:54:01,460::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8929c68f-7d73-42c9-b7e7-c34a63feec6d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30539::DEBUG::2012-12-02 14:54:01,460::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30539::DEBUG::2012-12-02 14:54:01,460::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30539::DEBUG::2012-12-02 14:54:01,460::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`8929c68f-7d73-42c9-b7e7-c34a63feec6d`::Granted request >Thread-30539::DEBUG::2012-12-02 14:54:01,460::task::794::TaskManager.Task::(resourceAcquired) Task=`b27d7219-8fba-4a45-bfc6-e7a241c203c7`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30539::DEBUG::2012-12-02 14:54:01,460::task::957::TaskManager.Task::(_decref) Task=`b27d7219-8fba-4a45-bfc6-e7a241c203c7`::ref 1 aborting False >Thread-30539::INFO::2012-12-02 14:54:01,466::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30539::DEBUG::2012-12-02 14:54:01,466::task::1151::TaskManager.Task::(prepare) Task=`b27d7219-8fba-4a45-bfc6-e7a241c203c7`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30539::DEBUG::2012-12-02 14:54:01,466::task::568::TaskManager.Task::(_updateState) Task=`b27d7219-8fba-4a45-bfc6-e7a241c203c7`::moving from state preparing -> state finished >Thread-30539::DEBUG::2012-12-02 14:54:01,467::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30539::DEBUG::2012-12-02 14:54:01,467::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30539::DEBUG::2012-12-02 14:54:01,467::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30539::DEBUG::2012-12-02 14:54:01,467::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30539::DEBUG::2012-12-02 14:54:01,467::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30539::DEBUG::2012-12-02 14:54:01,467::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30539::DEBUG::2012-12-02 14:54:01,467::task::957::TaskManager.Task::(_decref) Task=`b27d7219-8fba-4a45-bfc6-e7a241c203c7`::ref 0 aborting False >Thread-30544::DEBUG::2012-12-02 14:54:11,302::task::568::TaskManager.Task::(_updateState) Task=`2b7d9653-f964-44ad-a773-3761013d4056`::moving from state init -> state preparing >Thread-30544::INFO::2012-12-02 14:54:11,302::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30544::INFO::2012-12-02 14:54:11,302::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.004075050354', 'lastCheck': 1354452842.0710011, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00201201438904', 'lastCheck': 1354452848.581089, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00232219696045', 'lastCheck': 1354452848.8260641, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00795793533325', 'lastCheck': 1354452848.9213271, 'code': 0, 'valid': True}} >Thread-30544::DEBUG::2012-12-02 14:54:11,303::task::1151::TaskManager.Task::(prepare) Task=`2b7d9653-f964-44ad-a773-3761013d4056`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.004075050354', 'lastCheck': 1354452842.0710011, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00201201438904', 'lastCheck': 1354452848.581089, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00232219696045', 'lastCheck': 1354452848.8260641, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00795793533325', 'lastCheck': 1354452848.9213271, 'code': 0, 'valid': True}} >Thread-30544::DEBUG::2012-12-02 14:54:11,303::task::568::TaskManager.Task::(_updateState) Task=`2b7d9653-f964-44ad-a773-3761013d4056`::moving from state preparing -> state finished >Thread-30544::DEBUG::2012-12-02 14:54:11,303::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30544::DEBUG::2012-12-02 14:54:11,303::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30544::DEBUG::2012-12-02 14:54:11,303::task::957::TaskManager.Task::(_decref) Task=`2b7d9653-f964-44ad-a773-3761013d4056`::ref 0 aborting False >Thread-30546::DEBUG::2012-12-02 14:54:11,561::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30546::DEBUG::2012-12-02 14:54:11,561::task::568::TaskManager.Task::(_updateState) Task=`da0d1855-90f7-4c38-bb33-b6183a26144a`::moving from state init -> state preparing >Thread-30546::INFO::2012-12-02 14:54:11,561::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30546::INFO::2012-12-02 14:54:11,562::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30546::DEBUG::2012-12-02 14:54:11,562::task::1151::TaskManager.Task::(prepare) Task=`da0d1855-90f7-4c38-bb33-b6183a26144a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30546::DEBUG::2012-12-02 14:54:11,562::task::568::TaskManager.Task::(_updateState) Task=`da0d1855-90f7-4c38-bb33-b6183a26144a`::moving from state preparing -> state finished >Thread-30546::DEBUG::2012-12-02 14:54:11,562::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30546::DEBUG::2012-12-02 14:54:11,562::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30546::DEBUG::2012-12-02 14:54:11,562::task::957::TaskManager.Task::(_decref) Task=`da0d1855-90f7-4c38-bb33-b6183a26144a`::ref 0 aborting False >Thread-30547::DEBUG::2012-12-02 14:54:11,565::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30547::DEBUG::2012-12-02 14:54:11,565::task::568::TaskManager.Task::(_updateState) Task=`409160be-47df-4a81-b9c9-4327f547331d`::moving from state init -> state preparing >Thread-30547::INFO::2012-12-02 14:54:11,565::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30547::DEBUG::2012-12-02 14:54:11,565::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7c89d2b9-c236-45f2-b431-7b70a49b5187`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30547::DEBUG::2012-12-02 14:54:11,565::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30547::DEBUG::2012-12-02 14:54:11,566::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30547::DEBUG::2012-12-02 14:54:11,566::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`7c89d2b9-c236-45f2-b431-7b70a49b5187`::Granted request >Thread-30547::DEBUG::2012-12-02 14:54:11,566::task::794::TaskManager.Task::(resourceAcquired) Task=`409160be-47df-4a81-b9c9-4327f547331d`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30547::DEBUG::2012-12-02 14:54:11,566::task::957::TaskManager.Task::(_decref) Task=`409160be-47df-4a81-b9c9-4327f547331d`::ref 1 aborting False >Thread-30547::INFO::2012-12-02 14:54:11,571::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30547::DEBUG::2012-12-02 14:54:11,571::task::1151::TaskManager.Task::(prepare) Task=`409160be-47df-4a81-b9c9-4327f547331d`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30547::DEBUG::2012-12-02 14:54:11,571::task::568::TaskManager.Task::(_updateState) Task=`409160be-47df-4a81-b9c9-4327f547331d`::moving from state preparing -> state finished >Thread-30547::DEBUG::2012-12-02 14:54:11,571::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30547::DEBUG::2012-12-02 14:54:11,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30547::DEBUG::2012-12-02 14:54:11,571::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30547::DEBUG::2012-12-02 14:54:11,571::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30547::DEBUG::2012-12-02 14:54:11,571::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30547::DEBUG::2012-12-02 14:54:11,571::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30547::DEBUG::2012-12-02 14:54:11,571::task::957::TaskManager.Task::(_decref) Task=`409160be-47df-4a81-b9c9-4327f547331d`::ref 0 aborting False >Thread-30552::DEBUG::2012-12-02 14:54:21,391::task::568::TaskManager.Task::(_updateState) Task=`65e10dfd-7807-4217-bd29-5d51d41a07d6`::moving from state init -> state preparing >Thread-30552::INFO::2012-12-02 14:54:21,392::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30552::INFO::2012-12-02 14:54:21,392::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.010066986084', 'lastCheck': 1354452852.090374, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00320410728455', 'lastCheck': 1354452858.594698, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00276803970337', 'lastCheck': 1354452858.8386049, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0066180229187', 'lastCheck': 1354452858.936079, 'code': 0, 'valid': True}} >Thread-30552::DEBUG::2012-12-02 14:54:21,392::task::1151::TaskManager.Task::(prepare) Task=`65e10dfd-7807-4217-bd29-5d51d41a07d6`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.010066986084', 'lastCheck': 1354452852.090374, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00320410728455', 'lastCheck': 1354452858.594698, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00276803970337', 'lastCheck': 1354452858.8386049, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0066180229187', 'lastCheck': 1354452858.936079, 'code': 0, 'valid': True}} >Thread-30552::DEBUG::2012-12-02 14:54:21,392::task::568::TaskManager.Task::(_updateState) Task=`65e10dfd-7807-4217-bd29-5d51d41a07d6`::moving from state preparing -> state finished >Thread-30552::DEBUG::2012-12-02 14:54:21,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30552::DEBUG::2012-12-02 14:54:21,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30552::DEBUG::2012-12-02 14:54:21,393::task::957::TaskManager.Task::(_decref) Task=`65e10dfd-7807-4217-bd29-5d51d41a07d6`::ref 0 aborting False >Thread-30554::DEBUG::2012-12-02 14:54:21,665::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30554::DEBUG::2012-12-02 14:54:21,666::task::568::TaskManager.Task::(_updateState) Task=`97968ffb-c096-4e37-94f0-83490a3e00c2`::moving from state init -> state preparing >Thread-30554::INFO::2012-12-02 14:54:21,666::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30554::INFO::2012-12-02 14:54:21,666::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30554::DEBUG::2012-12-02 14:54:21,666::task::1151::TaskManager.Task::(prepare) Task=`97968ffb-c096-4e37-94f0-83490a3e00c2`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30554::DEBUG::2012-12-02 14:54:21,666::task::568::TaskManager.Task::(_updateState) Task=`97968ffb-c096-4e37-94f0-83490a3e00c2`::moving from state preparing -> state finished >Thread-30554::DEBUG::2012-12-02 14:54:21,666::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30554::DEBUG::2012-12-02 14:54:21,667::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30554::DEBUG::2012-12-02 14:54:21,667::task::957::TaskManager.Task::(_decref) Task=`97968ffb-c096-4e37-94f0-83490a3e00c2`::ref 0 aborting False >Thread-30555::DEBUG::2012-12-02 14:54:21,669::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30555::DEBUG::2012-12-02 14:54:21,670::task::568::TaskManager.Task::(_updateState) Task=`653c1534-b81d-46c5-9ac6-fa28e84fc182`::moving from state init -> state preparing >Thread-30555::INFO::2012-12-02 14:54:21,670::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30555::DEBUG::2012-12-02 14:54:21,670::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5a29480b-e3cd-4ac1-9a69-ceb151652958`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30555::DEBUG::2012-12-02 14:54:21,670::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30555::DEBUG::2012-12-02 14:54:21,670::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30555::DEBUG::2012-12-02 14:54:21,671::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`5a29480b-e3cd-4ac1-9a69-ceb151652958`::Granted request >Thread-30555::DEBUG::2012-12-02 14:54:21,671::task::794::TaskManager.Task::(resourceAcquired) Task=`653c1534-b81d-46c5-9ac6-fa28e84fc182`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30555::DEBUG::2012-12-02 14:54:21,671::task::957::TaskManager.Task::(_decref) Task=`653c1534-b81d-46c5-9ac6-fa28e84fc182`::ref 1 aborting False >Thread-30555::INFO::2012-12-02 14:54:21,677::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30555::DEBUG::2012-12-02 14:54:21,677::task::1151::TaskManager.Task::(prepare) Task=`653c1534-b81d-46c5-9ac6-fa28e84fc182`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30555::DEBUG::2012-12-02 14:54:21,677::task::568::TaskManager.Task::(_updateState) Task=`653c1534-b81d-46c5-9ac6-fa28e84fc182`::moving from state preparing -> state finished >Thread-30555::DEBUG::2012-12-02 14:54:21,677::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30555::DEBUG::2012-12-02 14:54:21,677::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30555::DEBUG::2012-12-02 14:54:21,677::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30555::DEBUG::2012-12-02 14:54:21,677::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30555::DEBUG::2012-12-02 14:54:21,677::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30555::DEBUG::2012-12-02 14:54:21,677::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30555::DEBUG::2012-12-02 14:54:21,678::task::957::TaskManager.Task::(_decref) Task=`653c1534-b81d-46c5-9ac6-fa28e84fc182`::ref 0 aborting False >Thread-30560::DEBUG::2012-12-02 14:54:31,487::task::568::TaskManager.Task::(_updateState) Task=`4097c8b2-5874-4844-aa11-c73882d435ff`::moving from state init -> state preparing >Thread-30560::INFO::2012-12-02 14:54:31,488::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30560::INFO::2012-12-02 14:54:31,488::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00798511505127', 'lastCheck': 1354452862.107306, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00224995613098', 'lastCheck': 1354452868.6097081, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00217604637146', 'lastCheck': 1354452868.84955, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812101364136', 'lastCheck': 1354452868.953038, 'code': 0, 'valid': True}} >Thread-30560::DEBUG::2012-12-02 14:54:31,488::task::1151::TaskManager.Task::(prepare) Task=`4097c8b2-5874-4844-aa11-c73882d435ff`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00798511505127', 'lastCheck': 1354452862.107306, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00224995613098', 'lastCheck': 1354452868.6097081, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00217604637146', 'lastCheck': 1354452868.84955, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00812101364136', 'lastCheck': 1354452868.953038, 'code': 0, 'valid': True}} >Thread-30560::DEBUG::2012-12-02 14:54:31,488::task::568::TaskManager.Task::(_updateState) Task=`4097c8b2-5874-4844-aa11-c73882d435ff`::moving from state preparing -> state finished >Thread-30560::DEBUG::2012-12-02 14:54:31,488::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30560::DEBUG::2012-12-02 14:54:31,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30560::DEBUG::2012-12-02 14:54:31,488::task::957::TaskManager.Task::(_decref) Task=`4097c8b2-5874-4844-aa11-c73882d435ff`::ref 0 aborting False >Thread-30562::DEBUG::2012-12-02 14:54:31,778::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30562::DEBUG::2012-12-02 14:54:31,779::task::568::TaskManager.Task::(_updateState) Task=`e8b9c2f9-b471-45a3-9ec8-bd7069be9540`::moving from state init -> state preparing >Thread-30562::INFO::2012-12-02 14:54:31,779::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30562::INFO::2012-12-02 14:54:31,779::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30562::DEBUG::2012-12-02 14:54:31,779::task::1151::TaskManager.Task::(prepare) Task=`e8b9c2f9-b471-45a3-9ec8-bd7069be9540`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30562::DEBUG::2012-12-02 14:54:31,779::task::568::TaskManager.Task::(_updateState) Task=`e8b9c2f9-b471-45a3-9ec8-bd7069be9540`::moving from state preparing -> state finished >Thread-30562::DEBUG::2012-12-02 14:54:31,779::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30562::DEBUG::2012-12-02 14:54:31,779::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30562::DEBUG::2012-12-02 14:54:31,780::task::957::TaskManager.Task::(_decref) Task=`e8b9c2f9-b471-45a3-9ec8-bd7069be9540`::ref 0 aborting False >Thread-30563::DEBUG::2012-12-02 14:54:31,782::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30563::DEBUG::2012-12-02 14:54:31,782::task::568::TaskManager.Task::(_updateState) Task=`6e6c5ed7-d0ad-4207-bdfe-08d73c7fd36f`::moving from state init -> state preparing >Thread-30563::INFO::2012-12-02 14:54:31,782::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30563::DEBUG::2012-12-02 14:54:31,783::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ebd63b6f-f0e7-42b9-bb7d-0d9458a2b4a2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30563::DEBUG::2012-12-02 14:54:31,783::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30563::DEBUG::2012-12-02 14:54:31,783::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30563::DEBUG::2012-12-02 14:54:31,783::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`ebd63b6f-f0e7-42b9-bb7d-0d9458a2b4a2`::Granted request >Thread-30563::DEBUG::2012-12-02 14:54:31,783::task::794::TaskManager.Task::(resourceAcquired) Task=`6e6c5ed7-d0ad-4207-bdfe-08d73c7fd36f`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30563::DEBUG::2012-12-02 14:54:31,784::task::957::TaskManager.Task::(_decref) Task=`6e6c5ed7-d0ad-4207-bdfe-08d73c7fd36f`::ref 1 aborting False >Thread-30563::INFO::2012-12-02 14:54:31,788::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30563::DEBUG::2012-12-02 14:54:31,788::task::1151::TaskManager.Task::(prepare) Task=`6e6c5ed7-d0ad-4207-bdfe-08d73c7fd36f`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30563::DEBUG::2012-12-02 14:54:31,789::task::568::TaskManager.Task::(_updateState) Task=`6e6c5ed7-d0ad-4207-bdfe-08d73c7fd36f`::moving from state preparing -> state finished >Thread-30563::DEBUG::2012-12-02 14:54:31,789::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30563::DEBUG::2012-12-02 14:54:31,789::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30563::DEBUG::2012-12-02 14:54:31,789::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30563::DEBUG::2012-12-02 14:54:31,789::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30563::DEBUG::2012-12-02 14:54:31,789::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30563::DEBUG::2012-12-02 14:54:31,789::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30563::DEBUG::2012-12-02 14:54:31,789::task::957::TaskManager.Task::(_decref) Task=`6e6c5ed7-d0ad-4207-bdfe-08d73c7fd36f`::ref 0 aborting False >Thread-30568::DEBUG::2012-12-02 14:54:41,658::task::568::TaskManager.Task::(_updateState) Task=`3727023c-9b78-4b63-9307-d48de17ec1e9`::moving from state init -> state preparing >Thread-30568::INFO::2012-12-02 14:54:41,658::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30568::INFO::2012-12-02 14:54:41,658::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00821900367737', 'lastCheck': 1354452872.126864, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00314497947693', 'lastCheck': 1354452878.62291, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00299596786499', 'lastCheck': 1354452878.86356, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00679016113281', 'lastCheck': 1354452878.9674449, 'code': 0, 'valid': True}} >Thread-30568::DEBUG::2012-12-02 14:54:41,658::task::1151::TaskManager.Task::(prepare) Task=`3727023c-9b78-4b63-9307-d48de17ec1e9`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00821900367737', 'lastCheck': 1354452872.126864, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00314497947693', 'lastCheck': 1354452878.62291, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00299596786499', 'lastCheck': 1354452878.86356, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.00679016113281', 'lastCheck': 1354452878.9674449, 'code': 0, 'valid': True}} >Thread-30568::DEBUG::2012-12-02 14:54:41,659::task::568::TaskManager.Task::(_updateState) Task=`3727023c-9b78-4b63-9307-d48de17ec1e9`::moving from state preparing -> state finished >Thread-30568::DEBUG::2012-12-02 14:54:41,659::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30568::DEBUG::2012-12-02 14:54:41,659::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30568::DEBUG::2012-12-02 14:54:41,659::task::957::TaskManager.Task::(_decref) Task=`3727023c-9b78-4b63-9307-d48de17ec1e9`::ref 0 aborting False >Thread-30570::DEBUG::2012-12-02 14:54:41,883::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30570::DEBUG::2012-12-02 14:54:41,883::task::568::TaskManager.Task::(_updateState) Task=`ff33fc41-eb50-44bf-9ae6-532c31b25de4`::moving from state init -> state preparing >Thread-30570::INFO::2012-12-02 14:54:41,883::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30570::INFO::2012-12-02 14:54:41,884::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30570::DEBUG::2012-12-02 14:54:41,884::task::1151::TaskManager.Task::(prepare) Task=`ff33fc41-eb50-44bf-9ae6-532c31b25de4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30570::DEBUG::2012-12-02 14:54:41,884::task::568::TaskManager.Task::(_updateState) Task=`ff33fc41-eb50-44bf-9ae6-532c31b25de4`::moving from state preparing -> state finished >Thread-30570::DEBUG::2012-12-02 14:54:41,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30570::DEBUG::2012-12-02 14:54:41,884::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30570::DEBUG::2012-12-02 14:54:41,884::task::957::TaskManager.Task::(_decref) Task=`ff33fc41-eb50-44bf-9ae6-532c31b25de4`::ref 0 aborting False >Thread-30571::DEBUG::2012-12-02 14:54:41,887::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30571::DEBUG::2012-12-02 14:54:41,887::task::568::TaskManager.Task::(_updateState) Task=`a166db35-3ab2-4095-9956-0630d1b6353a`::moving from state init -> state preparing >Thread-30571::INFO::2012-12-02 14:54:41,887::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30571::DEBUG::2012-12-02 14:54:41,887::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9cba15a5-90cc-4699-9a88-c770e402ab39`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30571::DEBUG::2012-12-02 14:54:41,887::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30571::DEBUG::2012-12-02 14:54:41,888::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30571::DEBUG::2012-12-02 14:54:41,888::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`9cba15a5-90cc-4699-9a88-c770e402ab39`::Granted request >Thread-30571::DEBUG::2012-12-02 14:54:41,888::task::794::TaskManager.Task::(resourceAcquired) Task=`a166db35-3ab2-4095-9956-0630d1b6353a`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30571::DEBUG::2012-12-02 14:54:41,888::task::957::TaskManager.Task::(_decref) Task=`a166db35-3ab2-4095-9956-0630d1b6353a`::ref 1 aborting False >Thread-30571::INFO::2012-12-02 14:54:41,895::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30571::DEBUG::2012-12-02 14:54:41,895::task::1151::TaskManager.Task::(prepare) Task=`a166db35-3ab2-4095-9956-0630d1b6353a`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30571::DEBUG::2012-12-02 14:54:41,896::task::568::TaskManager.Task::(_updateState) Task=`a166db35-3ab2-4095-9956-0630d1b6353a`::moving from state preparing -> state finished >Thread-30571::DEBUG::2012-12-02 14:54:41,896::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30571::DEBUG::2012-12-02 14:54:41,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30571::DEBUG::2012-12-02 14:54:41,896::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30571::DEBUG::2012-12-02 14:54:41,896::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30571::DEBUG::2012-12-02 14:54:41,896::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30571::DEBUG::2012-12-02 14:54:41,896::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30571::DEBUG::2012-12-02 14:54:41,896::task::957::TaskManager.Task::(_decref) Task=`a166db35-3ab2-4095-9956-0630d1b6353a`::ref 0 aborting False >Thread-30576::DEBUG::2012-12-02 14:54:51,850::task::568::TaskManager.Task::(_updateState) Task=`7bfeb2e2-adde-4260-a2e6-f55674c17624`::moving from state init -> state preparing >Thread-30576::INFO::2012-12-02 14:54:51,850::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30576::INFO::2012-12-02 14:54:51,850::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00976300239563', 'lastCheck': 1354452882.1521721, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0017569065094', 'lastCheck': 1354452888.6370759, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00211882591248', 'lastCheck': 1354452888.87462, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0074679851532', 'lastCheck': 1354452888.9855371, 'code': 0, 'valid': True}} >Thread-30576::DEBUG::2012-12-02 14:54:51,850::task::1151::TaskManager.Task::(prepare) Task=`7bfeb2e2-adde-4260-a2e6-f55674c17624`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00976300239563', 'lastCheck': 1354452882.1521721, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.0017569065094', 'lastCheck': 1354452888.6370759, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00211882591248', 'lastCheck': 1354452888.87462, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0074679851532', 'lastCheck': 1354452888.9855371, 'code': 0, 'valid': True}} >Thread-30576::DEBUG::2012-12-02 14:54:51,851::task::568::TaskManager.Task::(_updateState) Task=`7bfeb2e2-adde-4260-a2e6-f55674c17624`::moving from state preparing -> state finished >Thread-30576::DEBUG::2012-12-02 14:54:51,851::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30576::DEBUG::2012-12-02 14:54:51,851::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30576::DEBUG::2012-12-02 14:54:51,851::task::957::TaskManager.Task::(_decref) Task=`7bfeb2e2-adde-4260-a2e6-f55674c17624`::ref 0 aborting False >Thread-30578::DEBUG::2012-12-02 14:54:51,988::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30578::DEBUG::2012-12-02 14:54:51,988::task::568::TaskManager.Task::(_updateState) Task=`0a4c09ae-c434-473b-98d8-2ea1729a4912`::moving from state init -> state preparing >Thread-30578::INFO::2012-12-02 14:54:51,988::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30578::INFO::2012-12-02 14:54:51,988::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30578::DEBUG::2012-12-02 14:54:51,988::task::1151::TaskManager.Task::(prepare) Task=`0a4c09ae-c434-473b-98d8-2ea1729a4912`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'SPM', 'spmLver': 8}} >Thread-30578::DEBUG::2012-12-02 14:54:51,989::task::568::TaskManager.Task::(_updateState) Task=`0a4c09ae-c434-473b-98d8-2ea1729a4912`::moving from state preparing -> state finished >Thread-30578::DEBUG::2012-12-02 14:54:51,989::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30578::DEBUG::2012-12-02 14:54:51,989::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30578::DEBUG::2012-12-02 14:54:51,989::task::957::TaskManager.Task::(_decref) Task=`0a4c09ae-c434-473b-98d8-2ea1729a4912`::ref 0 aborting False >Thread-30579::DEBUG::2012-12-02 14:54:51,991::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30579::DEBUG::2012-12-02 14:54:51,992::task::568::TaskManager.Task::(_updateState) Task=`9aa2363d-6ebd-496b-bc97-6691439a695c`::moving from state init -> state preparing >Thread-30579::INFO::2012-12-02 14:54:51,992::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30579::DEBUG::2012-12-02 14:54:51,992::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c36db7e3-0cf5-476b-afb9-455d6a2890df`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30579::DEBUG::2012-12-02 14:54:51,992::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'shared' >Thread-30579::DEBUG::2012-12-02 14:54:51,992::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'shared' (1 active user) >Thread-30579::DEBUG::2012-12-02 14:54:51,992::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`c36db7e3-0cf5-476b-afb9-455d6a2890df`::Granted request >Thread-30579::DEBUG::2012-12-02 14:54:51,993::task::794::TaskManager.Task::(resourceAcquired) Task=`9aa2363d-6ebd-496b-bc97-6691439a695c`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (shared) >Thread-30579::DEBUG::2012-12-02 14:54:51,993::task::957::TaskManager.Task::(_decref) Task=`9aa2363d-6ebd-496b-bc97-6691439a695c`::ref 1 aborting False >Thread-30579::INFO::2012-12-02 14:54:51,998::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30579::DEBUG::2012-12-02 14:54:51,998::task::1151::TaskManager.Task::(prepare) Task=`9aa2363d-6ebd-496b-bc97-6691439a695c`::finished: {'info': {'spm_id': 1, 'master_uuid': '31e5485d-d40c-4a93-baea-0ff411f7b66a', 'name': 'dc1', 'version': '3', 'domains': '625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 1, 'lver': 8}, 'dominfo': {'625b62bd-21ae-4068-bc86-657056036298': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}, '95e02403-62f6-47c7-b706-e7be59adb348': {'status': 'Active', 'diskfree': '873313861632', 'alerts': [], 'disktotal': '953884344320'}}} >Thread-30579::DEBUG::2012-12-02 14:54:51,998::task::568::TaskManager.Task::(_updateState) Task=`9aa2363d-6ebd-496b-bc97-6691439a695c`::moving from state preparing -> state finished >Thread-30579::DEBUG::2012-12-02 14:54:51,998::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30579::DEBUG::2012-12-02 14:54:51,998::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30579::DEBUG::2012-12-02 14:54:51,998::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30579::DEBUG::2012-12-02 14:54:51,998::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30579::DEBUG::2012-12-02 14:54:51,999::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30579::DEBUG::2012-12-02 14:54:51,999::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30579::DEBUG::2012-12-02 14:54:51,999::task::957::TaskManager.Task::(_decref) Task=`9aa2363d-6ebd-496b-bc97-6691439a695c`::ref 0 aborting False >Thread-30583::DEBUG::2012-12-02 14:54:59,191::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30583::DEBUG::2012-12-02 14:54:59,191::task::568::TaskManager.Task::(_updateState) Task=`cece1e40-7857-4ad5-baea-6f924f7d59fd`::moving from state init -> state preparing >Thread-30583::INFO::2012-12-02 14:54:59,191::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-30583::DEBUG::2012-12-02 14:54:59,192::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-30583::DEBUG::2012-12-02 14:54:59,192::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-30583::INFO::2012-12-02 14:54:59,192::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-30583::DEBUG::2012-12-02 14:54:59,192::task::1151::TaskManager.Task::(prepare) Task=`cece1e40-7857-4ad5-baea-6f924f7d59fd`::finished: {'allTasksStatus': {}} >Thread-30583::DEBUG::2012-12-02 14:54:59,192::task::568::TaskManager.Task::(_updateState) Task=`cece1e40-7857-4ad5-baea-6f924f7d59fd`::moving from state preparing -> state finished >Thread-30583::DEBUG::2012-12-02 14:54:59,192::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30583::DEBUG::2012-12-02 14:54:59,192::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30583::DEBUG::2012-12-02 14:54:59,192::task::957::TaskManager.Task::(_decref) Task=`cece1e40-7857-4ad5-baea-6f924f7d59fd`::ref 0 aborting False >Thread-30584::DEBUG::2012-12-02 14:54:59,304::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30584::DEBUG::2012-12-02 14:54:59,304::task::568::TaskManager.Task::(_updateState) Task=`69bc4be6-c522-4e8c-a3d9-b31a42ffaba8`::moving from state init -> state preparing >Thread-30584::INFO::2012-12-02 14:54:59,304::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-30584::DEBUG::2012-12-02 14:54:59,304::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-30584::DEBUG::2012-12-02 14:54:59,304::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-30584::INFO::2012-12-02 14:54:59,304::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-30584::DEBUG::2012-12-02 14:54:59,305::task::1151::TaskManager.Task::(prepare) Task=`69bc4be6-c522-4e8c-a3d9-b31a42ffaba8`::finished: {'allTasksStatus': {}} >Thread-30584::DEBUG::2012-12-02 14:54:59,305::task::568::TaskManager.Task::(_updateState) Task=`69bc4be6-c522-4e8c-a3d9-b31a42ffaba8`::moving from state preparing -> state finished >Thread-30584::DEBUG::2012-12-02 14:54:59,305::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30584::DEBUG::2012-12-02 14:54:59,305::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30584::DEBUG::2012-12-02 14:54:59,305::task::957::TaskManager.Task::(_decref) Task=`69bc4be6-c522-4e8c-a3d9-b31a42ffaba8`::ref 0 aborting False >Thread-30585::DEBUG::2012-12-02 14:54:59,309::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30585::DEBUG::2012-12-02 14:54:59,309::task::568::TaskManager.Task::(_updateState) Task=`1e0d94ed-ade0-45e1-af04-b99581e29fc5`::moving from state init -> state preparing >Thread-30585::INFO::2012-12-02 14:54:59,309::logUtils::37::dispatcher::(wrapper) Run and protect: spmStop(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', options=None) >Thread-30585::DEBUG::2012-12-02 14:54:59,309::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6a919fd3-d7aa-444c-ad70-e70c4ab6d19c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30585::DEBUG::2012-12-02 14:54:59,309::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'exclusive' >Thread-30585::DEBUG::2012-12-02 14:54:59,309::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'exclusive' (1 active user) >Thread-30585::DEBUG::2012-12-02 14:54:59,310::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`6a919fd3-d7aa-444c-ad70-e70c4ab6d19c`::Granted request >Thread-30585::DEBUG::2012-12-02 14:54:59,310::task::794::TaskManager.Task::(resourceAcquired) Task=`1e0d94ed-ade0-45e1-af04-b99581e29fc5`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (exclusive) >Thread-30585::DEBUG::2012-12-02 14:54:59,310::task::957::TaskManager.Task::(_decref) Task=`1e0d94ed-ade0-45e1-af04-b99581e29fc5`::ref 1 aborting False >Thread-30585::DEBUG::2012-12-02 14:54:59,310::sp::315::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >Thread-30585::DEBUG::2012-12-02 14:54:59,310::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1ed2d2df-e54a-438c-a654-42980e2582ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30585::DEBUG::2012-12-02 14:54:59,310::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'exclusive' >Thread-30585::DEBUG::2012-12-02 14:54:59,310::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'exclusive' (1 active user) >Thread-30585::DEBUG::2012-12-02 14:54:59,310::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`1ed2d2df-e54a-438c-a654-42980e2582ba`::Granted request >Thread-30585::DEBUG::2012-12-02 14:54:59,311::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30585::DEBUG::2012-12-02 14:54:59,311::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30585::DEBUG::2012-12-02 14:54:59,311::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30585::DEBUG::2012-12-02 14:54:59,311::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30585::DEBUG::2012-12-02 14:54:59,315::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >Thread-30585::DEBUG::2012-12-02 14:54:59,315::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-30585::DEBUG::2012-12-02 14:54:59,315::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-30585::DEBUG::2012-12-02 14:54:59,315::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=data1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc1', 'POOL_DOMAINS=625b62bd-21ae-4068-bc86-657056036298:Active,31e5485d-d40c-4a93-baea-0ff411f7b66a:Active,718c4127-5ab7-440b-8d64-7f96ec39fce1:Active,95e02403-62f6-47c7-b706-e7be59adb348:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=8', 'POOL_UUID=aea11d65-88fa-4113-8161-93a0c73632f2', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'ROLE=Master', 'SDUUID=31e5485d-d40c-4a93-baea-0ff411f7b66a', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd452ebef7bc0a3f787d8bee6954fe8c78432454'] >Thread-30585::DEBUG::2012-12-02 14:54:59,372::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-30585::INFO::2012-12-02 14:54:59,372::safelease::239::SANLock::(release) Releasing cluster lock for domain 31e5485d-d40c-4a93-baea-0ff411f7b66a >Thread-30585::DEBUG::2012-12-02 14:54:59,390::safelease::250::SANLock::(release) Cluster lock for domain 31e5485d-d40c-4a93-baea-0ff411f7b66a successfully released >Thread-30585::INFO::2012-12-02 14:54:59,391::logUtils::39::dispatcher::(wrapper) Run and protect: spmStop, Return response: None >Thread-30585::DEBUG::2012-12-02 14:54:59,391::task::1151::TaskManager.Task::(prepare) Task=`1e0d94ed-ade0-45e1-af04-b99581e29fc5`::finished: None >Thread-30585::DEBUG::2012-12-02 14:54:59,391::task::568::TaskManager.Task::(_updateState) Task=`1e0d94ed-ade0-45e1-af04-b99581e29fc5`::moving from state preparing -> state finished >Thread-30585::DEBUG::2012-12-02 14:54:59,391::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30585::DEBUG::2012-12-02 14:54:59,391::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30585::DEBUG::2012-12-02 14:54:59,391::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30585::DEBUG::2012-12-02 14:54:59,391::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30585::DEBUG::2012-12-02 14:54:59,391::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30585::DEBUG::2012-12-02 14:54:59,392::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30585::DEBUG::2012-12-02 14:54:59,392::task::957::TaskManager.Task::(_decref) Task=`1e0d94ed-ade0-45e1-af04-b99581e29fc5`::ref 0 aborting False >Thread-30586::DEBUG::2012-12-02 14:54:59,925::task::568::TaskManager.Task::(_updateState) Task=`2cad1b0e-fa5e-40b1-b643-12b5c820e346`::moving from state init -> state preparing >Thread-30586::INFO::2012-12-02 14:54:59,925::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30586::INFO::2012-12-02 14:54:59,925::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00989103317261', 'lastCheck': 1354452892.171206, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206303596497', 'lastCheck': 1354452898.6496201, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171899795532', 'lastCheck': 1354452898.885581, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0171339511871', 'lastCheck': 1354452899.0122049, 'code': 0, 'valid': True}} >Thread-30586::DEBUG::2012-12-02 14:54:59,926::task::1151::TaskManager.Task::(prepare) Task=`2cad1b0e-fa5e-40b1-b643-12b5c820e346`::finished: {'625b62bd-21ae-4068-bc86-657056036298': {'delay': '0.00989103317261', 'lastCheck': 1354452892.171206, 'code': 0, 'valid': True}, '31e5485d-d40c-4a93-baea-0ff411f7b66a': {'delay': '0.00206303596497', 'lastCheck': 1354452898.6496201, 'code': 0, 'valid': True}, '718c4127-5ab7-440b-8d64-7f96ec39fce1': {'delay': '0.00171899795532', 'lastCheck': 1354452898.885581, 'code': 0, 'valid': True}, '95e02403-62f6-47c7-b706-e7be59adb348': {'delay': '0.0171339511871', 'lastCheck': 1354452899.0122049, 'code': 0, 'valid': True}} >Thread-30586::DEBUG::2012-12-02 14:54:59,926::task::568::TaskManager.Task::(_updateState) Task=`2cad1b0e-fa5e-40b1-b643-12b5c820e346`::moving from state preparing -> state finished >Thread-30586::DEBUG::2012-12-02 14:54:59,926::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30586::DEBUG::2012-12-02 14:54:59,926::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30586::DEBUG::2012-12-02 14:54:59,926::task::957::TaskManager.Task::(_decref) Task=`2cad1b0e-fa5e-40b1-b643-12b5c820e346`::ref 0 aborting False >Thread-30588::DEBUG::2012-12-02 14:55:00,035::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30588::DEBUG::2012-12-02 14:55:00,036::task::568::TaskManager.Task::(_updateState) Task=`592a1449-a8b1-45c2-809f-831c4e61c819`::moving from state init -> state preparing >Thread-30588::INFO::2012-12-02 14:55:00,036::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', hostID=1, scsiKey='aea11d65-88fa-4113-8161-93a0c73632f2', remove=False, options=None) >Thread-30588::DEBUG::2012-12-02 14:55:00,036::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`491edb2b-cbc5-4bf1-ac4b-dc7cb2fadc32`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30588::DEBUG::2012-12-02 14:55:00,036::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' for lock type 'exclusive' >Thread-30588::DEBUG::2012-12-02 14:55:00,036::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free. Now locking as 'exclusive' (1 active user) >Thread-30588::DEBUG::2012-12-02 14:55:00,037::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.aea11d65-88fa-4113-8161-93a0c73632f2`ReqID=`491edb2b-cbc5-4bf1-ac4b-dc7cb2fadc32`::Granted request >Thread-30588::DEBUG::2012-12-02 14:55:00,037::task::794::TaskManager.Task::(resourceAcquired) Task=`592a1449-a8b1-45c2-809f-831c4e61c819`::_resourcesAcquired: Storage.aea11d65-88fa-4113-8161-93a0c73632f2 (exclusive) >Thread-30588::DEBUG::2012-12-02 14:55:00,037::task::957::TaskManager.Task::(_decref) Task=`592a1449-a8b1-45c2-809f-831c4e61c819`::ref 1 aborting False >Thread-30588::INFO::2012-12-02 14:55:00,037::sp::658::Storage.StoragePool::(disconnect) Disconnect from the storage pool aea11d65-88fa-4113-8161-93a0c73632f2 >Thread-11821::DEBUG::2012-12-02 14:55:00,038::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 625b62bd-21ae-4068-bc86-657056036298 is stopping >Thread-11821::INFO::2012-12-02 14:55:00,038::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 625b62bd-21ae-4068-bc86-657056036298 (id: 1) >Thread-11821::DEBUG::2012-12-02 14:55:03,039::safelease::193::SANLock::(releaseHostId) Host id for domain 625b62bd-21ae-4068-bc86-657056036298 released successfully (id: 1) >Thread-11822::DEBUG::2012-12-02 14:55:03,039::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 31e5485d-d40c-4a93-baea-0ff411f7b66a is stopping >Thread-11822::INFO::2012-12-02 14:55:03,039::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 31e5485d-d40c-4a93-baea-0ff411f7b66a (id: 1) >Thread-11822::DEBUG::2012-12-02 14:55:05,039::safelease::193::SANLock::(releaseHostId) Host id for domain 31e5485d-d40c-4a93-baea-0ff411f7b66a released successfully (id: 1) >Thread-11884::DEBUG::2012-12-02 14:55:05,040::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 718c4127-5ab7-440b-8d64-7f96ec39fce1 is stopping >Thread-11946::DEBUG::2012-12-02 14:55:05,040::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 95e02403-62f6-47c7-b706-e7be59adb348 is stopping >Thread-11946::INFO::2012-12-02 14:55:05,040::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 95e02403-62f6-47c7-b706-e7be59adb348 (id: 1) >Thread-11946::DEBUG::2012-12-02 14:55:07,041::safelease::193::SANLock::(releaseHostId) Host id for domain 95e02403-62f6-47c7-b706-e7be59adb348 released successfully (id: 1) >Thread-30588::INFO::2012-12-02 14:55:07,041::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-30588::DEBUG::2012-12-02 14:55:07,041::task::1151::TaskManager.Task::(prepare) Task=`592a1449-a8b1-45c2-809f-831c4e61c819`::finished: True >Thread-30588::DEBUG::2012-12-02 14:55:07,042::task::568::TaskManager.Task::(_updateState) Task=`592a1449-a8b1-45c2-809f-831c4e61c819`::moving from state preparing -> state finished >Thread-30588::DEBUG::2012-12-02 14:55:07,042::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.aea11d65-88fa-4113-8161-93a0c73632f2': < ResourceRef 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', isValid: 'True' obj: 'None'>} >Thread-30588::DEBUG::2012-12-02 14:55:07,042::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30588::DEBUG::2012-12-02 14:55:07,042::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' >Thread-30588::DEBUG::2012-12-02 14:55:07,042::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' (0 active users) >Thread-30588::DEBUG::2012-12-02 14:55:07,042::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2' is free, finding out if anyone is waiting for it. >Thread-30588::DEBUG::2012-12-02 14:55:07,042::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.aea11d65-88fa-4113-8161-93a0c73632f2', Clearing records. >Thread-30588::DEBUG::2012-12-02 14:55:07,043::task::957::TaskManager.Task::(_decref) Task=`592a1449-a8b1-45c2-809f-831c4e61c819`::ref 0 aborting False >Thread-30589::DEBUG::2012-12-02 14:55:07,084::BindingXMLRPC::160::vds::(wrapper) [10.35.1.234] >Thread-30589::DEBUG::2012-12-02 14:55:07,085::task::568::TaskManager.Task::(_updateState) Task=`f86fae61-1b4e-4ba3-a3e4-a46722219f4b`::moving from state init -> state preparing >Thread-30589::INFO::2012-12-02 14:55:07,085::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='aea11d65-88fa-4113-8161-93a0c73632f2', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '2cf18e57-89a2-4289-8250-8166cea114c3', 'port': ''}, {'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/export1', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '0fd83e5a-0073-488e-a285-1188f158d927', 'port': ''}, {'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data3', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '48a60794-1558-46d7-b35a-e1172edda23e', 'port': ''}, {'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/amureini/data4', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '363f822d-4935-4d65-98e9-09e01af1821d', 'port': ''}], options=None) >Thread-30589::DEBUG::2012-12-02 14:55:07,085::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data1' (cwd None) >Thread-30589::DEBUG::2012-12-02 14:55:07,115::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export1' (cwd None) >Thread-30589::DEBUG::2012-12-02 14:55:07,275::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data3' (cwd None) >Thread-30589::DEBUG::2012-12-02 14:55:07,306::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_data4' (cwd None) >Thread-30589::DEBUG::2012-12-02 14:55:07,336::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-30589::DEBUG::2012-12-02 14:55:07,339::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30589::DEBUG::2012-12-02 14:55:07,339::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-30589::DEBUG::2012-12-02 14:55:07,339::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30589::DEBUG::2012-12-02 14:55:07,339::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-30589::DEBUG::2012-12-02 14:55:07,365::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30589::DEBUG::2012-12-02 14:55:07,366::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30589::DEBUG::2012-12-02 14:55:07,999::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-30589::DEBUG::2012-12-02 14:55:08,138::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30589::DEBUG::2012-12-02 14:55:08,138::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30589::DEBUG::2012-12-02 14:55:08,139::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30589::DEBUG::2012-12-02 14:55:08,139::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30589::DEBUG::2012-12-02 14:55:08,139::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30589::DEBUG::2012-12-02 14:55:08,139::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30589::DEBUG::2012-12-02 14:55:08,139::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30589::DEBUG::2012-12-02 14:55:08,139::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30589::INFO::2012-12-02 14:55:08,140::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '2cf18e57-89a2-4289-8250-8166cea114c3'}, {'status': 0, 'id': '0fd83e5a-0073-488e-a285-1188f158d927'}, {'status': 0, 'id': '48a60794-1558-46d7-b35a-e1172edda23e'}, {'status': 0, 'id': '363f822d-4935-4d65-98e9-09e01af1821d'}]} >Thread-30589::DEBUG::2012-12-02 14:55:08,140::task::1151::TaskManager.Task::(prepare) Task=`f86fae61-1b4e-4ba3-a3e4-a46722219f4b`::finished: {'statuslist': [{'status': 0, 'id': '2cf18e57-89a2-4289-8250-8166cea114c3'}, {'status': 0, 'id': '0fd83e5a-0073-488e-a285-1188f158d927'}, {'status': 0, 'id': '48a60794-1558-46d7-b35a-e1172edda23e'}, {'status': 0, 'id': '363f822d-4935-4d65-98e9-09e01af1821d'}]} >Thread-30589::DEBUG::2012-12-02 14:55:08,140::task::568::TaskManager.Task::(_updateState) Task=`f86fae61-1b4e-4ba3-a3e4-a46722219f4b`::moving from state preparing -> state finished >Thread-30589::DEBUG::2012-12-02 14:55:08,140::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30589::DEBUG::2012-12-02 14:55:08,140::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30589::DEBUG::2012-12-02 14:55:08,140::task::957::TaskManager.Task::(_decref) Task=`f86fae61-1b4e-4ba3-a3e4-a46722219f4b`::ref 0 aborting False >Thread-30590::DEBUG::2012-12-02 14:55:51,487::BindingXMLRPC::872::vds::(wrapper) client [10.35.1.163]::call getCapabilities with () {} >Thread-30590::DEBUG::2012-12-02 14:55:51,500::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-30590::DEBUG::2012-12-02 14:55:51,511::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-30590::DEBUG::2012-12-02 14:55:51,511::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-30590::DEBUG::2012-12-02 14:55:51,521::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-30590::DEBUG::2012-12-02 14:55:51,521::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-30590::DEBUG::2012-12-02 14:55:51,533::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30590::DEBUG::2012-12-02 14:55:51,534::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-30590::DEBUG::2012-12-02 14:55:51,546::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30590::DEBUG::2012-12-02 14:55:51,547::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-30590::DEBUG::2012-12-02 14:55:51,560::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30590::DEBUG::2012-12-02 14:55:51,561::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.1.234'}} >Thread-30591::DEBUG::2012-12-02 14:55:51,715::BindingXMLRPC::872::vds::(wrapper) client [10.35.1.163]::call getCapabilities with () {} >Thread-30591::DEBUG::2012-12-02 14:55:51,726::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-30591::DEBUG::2012-12-02 14:55:51,738::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-30591::DEBUG::2012-12-02 14:55:51,739::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-30591::DEBUG::2012-12-02 14:55:51,750::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-30591::DEBUG::2012-12-02 14:55:51,750::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-30591::DEBUG::2012-12-02 14:55:51,763::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30591::DEBUG::2012-12-02 14:55:51,763::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-30591::DEBUG::2012-12-02 14:55:51,773::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30591::DEBUG::2012-12-02 14:55:51,774::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-30591::DEBUG::2012-12-02 14:55:51,786::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30591::DEBUG::2012-12-02 14:55:51,787::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.1.163'}} >Thread-30593::DEBUG::2012-12-02 14:55:51,964::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30593::DEBUG::2012-12-02 14:55:51,965::task::568::TaskManager.Task::(_updateState) Task=`6bc015a7-ecdb-46e4-8c88-3348b80f1c8c`::moving from state init -> state preparing >Thread-30593::INFO::2012-12-02 14:55:51,965::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '7f9ba62c-1ef6-4d5f-a012-d4efb9105493', 'port': ''}], options=None) >Thread-30593::INFO::2012-12-02 14:55:51,965::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '7f9ba62c-1ef6-4d5f-a012-d4efb9105493'}]} >Thread-30593::DEBUG::2012-12-02 14:55:51,965::task::1151::TaskManager.Task::(prepare) Task=`6bc015a7-ecdb-46e4-8c88-3348b80f1c8c`::finished: {'statuslist': [{'status': 0, 'id': '7f9ba62c-1ef6-4d5f-a012-d4efb9105493'}]} >Thread-30593::DEBUG::2012-12-02 14:55:51,965::task::568::TaskManager.Task::(_updateState) Task=`6bc015a7-ecdb-46e4-8c88-3348b80f1c8c`::moving from state preparing -> state finished >Thread-30593::DEBUG::2012-12-02 14:55:51,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30593::DEBUG::2012-12-02 14:55:51,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30593::DEBUG::2012-12-02 14:55:51,965::task::957::TaskManager.Task::(_decref) Task=`6bc015a7-ecdb-46e4-8c88-3348b80f1c8c`::ref 0 aborting False >Thread-30594::DEBUG::2012-12-02 14:55:51,983::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30594::DEBUG::2012-12-02 14:55:51,983::task::568::TaskManager.Task::(_updateState) Task=`b1ccd103-6530-47c0-a01d-68338a2e78be`::moving from state init -> state preparing >Thread-30594::INFO::2012-12-02 14:55:51,983::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '7f9ba62c-1ef6-4d5f-a012-d4efb9105493', 'port': ''}], options=None) >Thread-30594::DEBUG::2012-12-02 14:55:51,988::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446' (cwd None) >Thread-30594::DEBUG::2012-12-02 14:55:52,115::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30594::DEBUG::2012-12-02 14:55:52,116::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30594::DEBUG::2012-12-02 14:55:52,116::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30594::DEBUG::2012-12-02 14:55:52,116::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30594::DEBUG::2012-12-02 14:55:52,116::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30594::DEBUG::2012-12-02 14:55:52,116::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30594::INFO::2012-12-02 14:55:52,116::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '7f9ba62c-1ef6-4d5f-a012-d4efb9105493'}]} >Thread-30594::DEBUG::2012-12-02 14:55:52,117::task::1151::TaskManager.Task::(prepare) Task=`b1ccd103-6530-47c0-a01d-68338a2e78be`::finished: {'statuslist': [{'status': 0, 'id': '7f9ba62c-1ef6-4d5f-a012-d4efb9105493'}]} >Thread-30594::DEBUG::2012-12-02 14:55:52,117::task::568::TaskManager.Task::(_updateState) Task=`b1ccd103-6530-47c0-a01d-68338a2e78be`::moving from state preparing -> state finished >Thread-30594::DEBUG::2012-12-02 14:55:52,117::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30594::DEBUG::2012-12-02 14:55:52,117::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30594::DEBUG::2012-12-02 14:55:52,117::task::957::TaskManager.Task::(_decref) Task=`b1ccd103-6530-47c0-a01d-68338a2e78be`::ref 0 aborting False >Thread-30595::DEBUG::2012-12-02 14:55:52,145::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30595::DEBUG::2012-12-02 14:55:52,145::task::568::TaskManager.Task::(_updateState) Task=`8c3fbe8f-bf23-4c1e-b87c-1fe8a430848a`::moving from state init -> state preparing >Thread-30595::INFO::2012-12-02 14:55:52,145::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='a3177eae-1dc9-445a-92b1-da23ec8008bc', masterVersion=1, options=None) >Thread-30595::DEBUG::2012-12-02 14:55:52,146::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`f3bc7635-743a-446b-8a13-baeaf5f7d2c3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30595::DEBUG::2012-12-02 14:55:52,146::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-30595::DEBUG::2012-12-02 14:55:52,146::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-30595::DEBUG::2012-12-02 14:55:52,146::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`f3bc7635-743a-446b-8a13-baeaf5f7d2c3`::Granted request >Thread-30595::INFO::2012-12-02 14:55:52,147::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: a3177eae-1dc9-445a-92b1-da23ec8008bc (ver = 1) >Thread-30595::DEBUG::2012-12-02 14:55:52,147::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,147::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,147::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,147::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,147::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,148::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,148::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-30595::DEBUG::2012-12-02 14:55:52,148::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30595::DEBUG::2012-12-02 14:55:52,148::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-30595::DEBUG::2012-12-02 14:55:52,148::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30595::DEBUG::2012-12-02 14:55:52,148::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-30595::DEBUG::2012-12-02 14:55:52,173::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30595::DEBUG::2012-12-02 14:55:52,174::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30595::DEBUG::2012-12-02 14:55:52,806::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-30595::DEBUG::2012-12-02 14:55:52,890::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30595::DEBUG::2012-12-02 14:55:52,891::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,891::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,891::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,891::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,892::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,892::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,892::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30595::DEBUG::2012-12-02 14:55:52,892::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:52,894::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-30595::DEBUG::2012-12-02 14:55:53,111::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-30595::WARNING::2012-12-02 14:55:53,113::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-30595::DEBUG::2012-12-02 14:55:53,113::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:53,149::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30595::DEBUG::2012-12-02 14:55:53,149::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-30595::DEBUG::2012-12-02 14:55:53,159::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'ROLE=Master', 'SDUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=77d9cbd7cbfb7d30cd64cfa61cb7a6d547852110'] >Thread-30595::DEBUG::2012-12-02 14:55:53,193::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-30595::DEBUG::2012-12-02 14:55:53,193::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'a3177eae-1dc9-445a-92b1-da23ec8008bc_imageNS' >Thread-30595::DEBUG::2012-12-02 14:55:53,194::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'a3177eae-1dc9-445a-92b1-da23ec8008bc_volumeNS' >Thread-30595::DEBUG::2012-12-02 14:55:53,194::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain a3177eae-1dc9-445a-92b1-da23ec8008bc verified, version 1 >Thread-30595::DEBUG::2012-12-02 14:55:53,194::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-30595::DEBUG::2012-12-02 14:55:53,194::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30595::DEBUG::2012-12-02 14:55:53,202::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'ROLE=Master', 'SDUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=77d9cbd7cbfb7d30cd64cfa61cb7a6d547852110'] >Thread-30595::DEBUG::2012-12-02 14:55:53,202::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `da40b79b-9f51-4240-9cb2-2ab10e793fb6` started monitoring domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Thread-30595::DEBUG::2012-12-02 14:55:53,203::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30595::DEBUG::2012-12-02 14:55:53,206::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30595::DEBUG::2012-12-02 14:55:53,208::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free' (cwd None) >Thread-30654::INFO::2012-12-02 14:55:53,228::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain a3177eae-1dc9-445a-92b1-da23ec8008bc (id: 3) >Thread-30654::DEBUG::2012-12-02 14:55:53,229::safelease::178::SANLock::(acquireHostId) Host id for domain a3177eae-1dc9-445a-92b1-da23ec8008bc successfully acquired (id: 3) >Thread-30595::DEBUG::2012-12-02 14:55:53,598::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >Thread-30595::DEBUG::2012-12-02 14:55:53,602::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30595::INFO::2012-12-02 14:55:53,628::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30595::INFO::2012-12-02 14:55:53,629::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/mastersd >Thread-30595::DEBUG::2012-12-02 14:55:53,629::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-30595::DEBUG::2012-12-02 14:55:53,629::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-30595::DEBUG::2012-12-02 14:55:53,629::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-30595::DEBUG::2012-12-02 14:55:53,629::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-30595::INFO::2012-12-02 14:55:53,630::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-30595::DEBUG::2012-12-02 14:55:53,630::task::1151::TaskManager.Task::(prepare) Task=`8c3fbe8f-bf23-4c1e-b87c-1fe8a430848a`::finished: True >Thread-30595::DEBUG::2012-12-02 14:55:53,630::task::568::TaskManager.Task::(_updateState) Task=`8c3fbe8f-bf23-4c1e-b87c-1fe8a430848a`::moving from state preparing -> state finished >Thread-30595::DEBUG::2012-12-02 14:55:53,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30595::DEBUG::2012-12-02 14:55:53,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30595::DEBUG::2012-12-02 14:55:53,630::task::957::TaskManager.Task::(_decref) Task=`8c3fbe8f-bf23-4c1e-b87c-1fe8a430848a`::ref 0 aborting False >Thread-30712::DEBUG::2012-12-02 14:55:53,638::task::568::TaskManager.Task::(_updateState) Task=`a80bc032-51ac-46ec-9b12-aaed9bce5650`::moving from state init -> state preparing >Thread-30712::INFO::2012-12-02 14:55:53,639::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30712::INFO::2012-12-02 14:55:53,639::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00647020339966', 'lastCheck': 1354452953.2288771, 'code': 0, 'valid': True}} >Thread-30712::DEBUG::2012-12-02 14:55:53,639::task::1151::TaskManager.Task::(prepare) Task=`a80bc032-51ac-46ec-9b12-aaed9bce5650`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00647020339966', 'lastCheck': 1354452953.2288771, 'code': 0, 'valid': True}} >Thread-30712::DEBUG::2012-12-02 14:55:53,639::task::568::TaskManager.Task::(_updateState) Task=`a80bc032-51ac-46ec-9b12-aaed9bce5650`::moving from state preparing -> state finished >Thread-30712::DEBUG::2012-12-02 14:55:53,639::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30712::DEBUG::2012-12-02 14:55:53,639::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30712::DEBUG::2012-12-02 14:55:53,640::task::957::TaskManager.Task::(_decref) Task=`a80bc032-51ac-46ec-9b12-aaed9bce5650`::ref 0 aborting False >Thread-30717::DEBUG::2012-12-02 14:56:03,768::task::568::TaskManager.Task::(_updateState) Task=`086475f6-7af0-4cfc-a883-f26510397361`::moving from state init -> state preparing >Thread-30717::INFO::2012-12-02 14:56:03,768::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30717::INFO::2012-12-02 14:56:03,768::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00762605667114', 'lastCheck': 1354452963.247782, 'code': 0, 'valid': True}} >Thread-30717::DEBUG::2012-12-02 14:56:03,768::task::1151::TaskManager.Task::(prepare) Task=`086475f6-7af0-4cfc-a883-f26510397361`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00762605667114', 'lastCheck': 1354452963.247782, 'code': 0, 'valid': True}} >Thread-30717::DEBUG::2012-12-02 14:56:03,769::task::568::TaskManager.Task::(_updateState) Task=`086475f6-7af0-4cfc-a883-f26510397361`::moving from state preparing -> state finished >Thread-30717::DEBUG::2012-12-02 14:56:03,769::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30717::DEBUG::2012-12-02 14:56:03,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30717::DEBUG::2012-12-02 14:56:03,769::task::957::TaskManager.Task::(_decref) Task=`086475f6-7af0-4cfc-a883-f26510397361`::ref 0 aborting False >Thread-30723::DEBUG::2012-12-02 14:56:13,880::task::568::TaskManager.Task::(_updateState) Task=`fc847a4f-967f-4eaf-b1c5-0d914a1dce56`::moving from state init -> state preparing >Thread-30723::INFO::2012-12-02 14:56:13,881::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30723::INFO::2012-12-02 14:56:13,881::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00849318504333', 'lastCheck': 1354452973.269592, 'code': 0, 'valid': True}} >Thread-30723::DEBUG::2012-12-02 14:56:13,881::task::1151::TaskManager.Task::(prepare) Task=`fc847a4f-967f-4eaf-b1c5-0d914a1dce56`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00849318504333', 'lastCheck': 1354452973.269592, 'code': 0, 'valid': True}} >Thread-30723::DEBUG::2012-12-02 14:56:13,881::task::568::TaskManager.Task::(_updateState) Task=`fc847a4f-967f-4eaf-b1c5-0d914a1dce56`::moving from state preparing -> state finished >Thread-30723::DEBUG::2012-12-02 14:56:13,881::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30723::DEBUG::2012-12-02 14:56:13,881::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30723::DEBUG::2012-12-02 14:56:13,881::task::957::TaskManager.Task::(_decref) Task=`fc847a4f-967f-4eaf-b1c5-0d914a1dce56`::ref 0 aborting False >Thread-30729::DEBUG::2012-12-02 14:56:23,995::task::568::TaskManager.Task::(_updateState) Task=`74baddfe-c88d-4486-84a6-eaff0631d3bf`::moving from state init -> state preparing >Thread-30729::INFO::2012-12-02 14:56:23,995::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30729::INFO::2012-12-02 14:56:23,996::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00873613357544', 'lastCheck': 1354452983.2947569, 'code': 0, 'valid': True}} >Thread-30729::DEBUG::2012-12-02 14:56:23,996::task::1151::TaskManager.Task::(prepare) Task=`74baddfe-c88d-4486-84a6-eaff0631d3bf`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00873613357544', 'lastCheck': 1354452983.2947569, 'code': 0, 'valid': True}} >Thread-30729::DEBUG::2012-12-02 14:56:23,996::task::568::TaskManager.Task::(_updateState) Task=`74baddfe-c88d-4486-84a6-eaff0631d3bf`::moving from state preparing -> state finished >Thread-30729::DEBUG::2012-12-02 14:56:23,996::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30729::DEBUG::2012-12-02 14:56:23,996::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30729::DEBUG::2012-12-02 14:56:23,996::task::957::TaskManager.Task::(_decref) Task=`74baddfe-c88d-4486-84a6-eaff0631d3bf`::ref 0 aborting False >Thread-30735::DEBUG::2012-12-02 14:56:34,686::task::568::TaskManager.Task::(_updateState) Task=`701d42bd-3056-4738-80db-a616231e4d86`::moving from state init -> state preparing >Thread-30735::INFO::2012-12-02 14:56:34,686::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30735::INFO::2012-12-02 14:56:34,687::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00860786437988', 'lastCheck': 1354452993.313273, 'code': 0, 'valid': True}} >Thread-30735::DEBUG::2012-12-02 14:56:34,687::task::1151::TaskManager.Task::(prepare) Task=`701d42bd-3056-4738-80db-a616231e4d86`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00860786437988', 'lastCheck': 1354452993.313273, 'code': 0, 'valid': True}} >Thread-30735::DEBUG::2012-12-02 14:56:34,687::task::568::TaskManager.Task::(_updateState) Task=`701d42bd-3056-4738-80db-a616231e4d86`::moving from state preparing -> state finished >Thread-30735::DEBUG::2012-12-02 14:56:34,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30735::DEBUG::2012-12-02 14:56:34,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30735::DEBUG::2012-12-02 14:56:34,687::task::957::TaskManager.Task::(_decref) Task=`701d42bd-3056-4738-80db-a616231e4d86`::ref 0 aborting False >Thread-30741::DEBUG::2012-12-02 14:56:44,806::task::568::TaskManager.Task::(_updateState) Task=`c42bbd77-d99f-4e2a-b9ee-d757d3308556`::moving from state init -> state preparing >Thread-30741::INFO::2012-12-02 14:56:44,807::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30741::INFO::2012-12-02 14:56:44,807::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00842690467834', 'lastCheck': 1354453003.3306341, 'code': 0, 'valid': True}} >Thread-30741::DEBUG::2012-12-02 14:56:44,807::task::1151::TaskManager.Task::(prepare) Task=`c42bbd77-d99f-4e2a-b9ee-d757d3308556`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00842690467834', 'lastCheck': 1354453003.3306341, 'code': 0, 'valid': True}} >Thread-30741::DEBUG::2012-12-02 14:56:44,807::task::568::TaskManager.Task::(_updateState) Task=`c42bbd77-d99f-4e2a-b9ee-d757d3308556`::moving from state preparing -> state finished >Thread-30741::DEBUG::2012-12-02 14:56:44,807::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30741::DEBUG::2012-12-02 14:56:44,807::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30741::DEBUG::2012-12-02 14:56:44,807::task::957::TaskManager.Task::(_decref) Task=`c42bbd77-d99f-4e2a-b9ee-d757d3308556`::ref 0 aborting False >Thread-30743::DEBUG::2012-12-02 14:56:46,703::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30743::DEBUG::2012-12-02 14:56:46,704::task::568::TaskManager.Task::(_updateState) Task=`d85afddd-7252-4a62-82b9-69fe2602f834`::moving from state init -> state preparing >Thread-30743::INFO::2012-12-02 14:56:46,704::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175', 'port': ''}], options=None) >Thread-30743::INFO::2012-12-02 14:56:46,704::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30743::DEBUG::2012-12-02 14:56:46,704::task::1151::TaskManager.Task::(prepare) Task=`d85afddd-7252-4a62-82b9-69fe2602f834`::finished: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30743::DEBUG::2012-12-02 14:56:46,704::task::568::TaskManager.Task::(_updateState) Task=`d85afddd-7252-4a62-82b9-69fe2602f834`::moving from state preparing -> state finished >Thread-30743::DEBUG::2012-12-02 14:56:46,704::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30743::DEBUG::2012-12-02 14:56:46,704::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30743::DEBUG::2012-12-02 14:56:46,705::task::957::TaskManager.Task::(_decref) Task=`d85afddd-7252-4a62-82b9-69fe2602f834`::ref 0 aborting False >Thread-30744::DEBUG::2012-12-02 14:56:46,718::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30744::DEBUG::2012-12-02 14:56:46,718::task::568::TaskManager.Task::(_updateState) Task=`42fbb6e1-2327-41a2-8fd5-b756c7ac868c`::moving from state init -> state preparing >Thread-30744::INFO::2012-12-02 14:56:46,718::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175', 'port': ''}], options=None) >Thread-30744::DEBUG::2012-12-02 14:56:46,724::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447' (cwd None) >Thread-30744::DEBUG::2012-12-02 14:56:46,857::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30744::DEBUG::2012-12-02 14:56:46,857::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30744::DEBUG::2012-12-02 14:56:46,857::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30744::DEBUG::2012-12-02 14:56:46,857::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30744::DEBUG::2012-12-02 14:56:46,858::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30744::DEBUG::2012-12-02 14:56:46,858::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30744::INFO::2012-12-02 14:56:46,858::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30744::DEBUG::2012-12-02 14:56:46,858::task::1151::TaskManager.Task::(prepare) Task=`42fbb6e1-2327-41a2-8fd5-b756c7ac868c`::finished: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30744::DEBUG::2012-12-02 14:56:46,858::task::568::TaskManager.Task::(_updateState) Task=`42fbb6e1-2327-41a2-8fd5-b756c7ac868c`::moving from state preparing -> state finished >Thread-30744::DEBUG::2012-12-02 14:56:46,858::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30744::DEBUG::2012-12-02 14:56:46,859::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30744::DEBUG::2012-12-02 14:56:46,859::task::957::TaskManager.Task::(_decref) Task=`42fbb6e1-2327-41a2-8fd5-b756c7ac868c`::ref 0 aborting False >Thread-30749::DEBUG::2012-12-02 14:56:54,982::task::568::TaskManager.Task::(_updateState) Task=`0f830092-3176-49c7-98be-ae8788633cad`::moving from state init -> state preparing >Thread-30749::INFO::2012-12-02 14:56:54,982::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30749::INFO::2012-12-02 14:56:54,982::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0118389129639', 'lastCheck': 1354453013.3546979, 'code': 0, 'valid': True}} >Thread-30749::DEBUG::2012-12-02 14:56:54,982::task::1151::TaskManager.Task::(prepare) Task=`0f830092-3176-49c7-98be-ae8788633cad`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0118389129639', 'lastCheck': 1354453013.3546979, 'code': 0, 'valid': True}} >Thread-30749::DEBUG::2012-12-02 14:56:54,982::task::568::TaskManager.Task::(_updateState) Task=`0f830092-3176-49c7-98be-ae8788633cad`::moving from state preparing -> state finished >Thread-30749::DEBUG::2012-12-02 14:56:54,983::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30749::DEBUG::2012-12-02 14:56:54,983::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30749::DEBUG::2012-12-02 14:56:54,983::task::957::TaskManager.Task::(_decref) Task=`0f830092-3176-49c7-98be-ae8788633cad`::ref 0 aborting False >Thread-30755::DEBUG::2012-12-02 14:57:05,110::task::568::TaskManager.Task::(_updateState) Task=`af68a172-452b-4fa3-93c2-e86045a3b466`::moving from state init -> state preparing >Thread-30755::INFO::2012-12-02 14:57:05,110::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30755::INFO::2012-12-02 14:57:05,110::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00814390182495', 'lastCheck': 1354453023.373728, 'code': 0, 'valid': True}} >Thread-30755::DEBUG::2012-12-02 14:57:05,110::task::1151::TaskManager.Task::(prepare) Task=`af68a172-452b-4fa3-93c2-e86045a3b466`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00814390182495', 'lastCheck': 1354453023.373728, 'code': 0, 'valid': True}} >Thread-30755::DEBUG::2012-12-02 14:57:05,111::task::568::TaskManager.Task::(_updateState) Task=`af68a172-452b-4fa3-93c2-e86045a3b466`::moving from state preparing -> state finished >Thread-30755::DEBUG::2012-12-02 14:57:05,111::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30755::DEBUG::2012-12-02 14:57:05,111::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30755::DEBUG::2012-12-02 14:57:05,111::task::957::TaskManager.Task::(_decref) Task=`af68a172-452b-4fa3-93c2-e86045a3b466`::ref 0 aborting False >Thread-30761::DEBUG::2012-12-02 14:57:15,231::task::568::TaskManager.Task::(_updateState) Task=`eb0e4b99-4fc7-47fc-849d-b0cd5b82542a`::moving from state init -> state preparing >Thread-30761::INFO::2012-12-02 14:57:15,232::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30761::INFO::2012-12-02 14:57:15,232::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0101640224457', 'lastCheck': 1354453033.3921051, 'code': 0, 'valid': True}} >Thread-30761::DEBUG::2012-12-02 14:57:15,232::task::1151::TaskManager.Task::(prepare) Task=`eb0e4b99-4fc7-47fc-849d-b0cd5b82542a`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0101640224457', 'lastCheck': 1354453033.3921051, 'code': 0, 'valid': True}} >Thread-30761::DEBUG::2012-12-02 14:57:15,232::task::568::TaskManager.Task::(_updateState) Task=`eb0e4b99-4fc7-47fc-849d-b0cd5b82542a`::moving from state preparing -> state finished >Thread-30761::DEBUG::2012-12-02 14:57:15,232::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30761::DEBUG::2012-12-02 14:57:15,232::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30761::DEBUG::2012-12-02 14:57:15,232::task::957::TaskManager.Task::(_decref) Task=`eb0e4b99-4fc7-47fc-849d-b0cd5b82542a`::ref 0 aborting False >Thread-30767::DEBUG::2012-12-02 14:57:25,346::task::568::TaskManager.Task::(_updateState) Task=`172ba199-da1b-485a-b784-913784c214de`::moving from state init -> state preparing >Thread-30767::INFO::2012-12-02 14:57:25,346::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30767::INFO::2012-12-02 14:57:25,347::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00423908233643', 'lastCheck': 1354453043.4104431, 'code': 0, 'valid': True}} >Thread-30767::DEBUG::2012-12-02 14:57:25,347::task::1151::TaskManager.Task::(prepare) Task=`172ba199-da1b-485a-b784-913784c214de`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00423908233643', 'lastCheck': 1354453043.4104431, 'code': 0, 'valid': True}} >Thread-30767::DEBUG::2012-12-02 14:57:25,347::task::568::TaskManager.Task::(_updateState) Task=`172ba199-da1b-485a-b784-913784c214de`::moving from state preparing -> state finished >Thread-30767::DEBUG::2012-12-02 14:57:25,347::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30767::DEBUG::2012-12-02 14:57:25,347::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30767::DEBUG::2012-12-02 14:57:25,347::task::957::TaskManager.Task::(_decref) Task=`172ba199-da1b-485a-b784-913784c214de`::ref 0 aborting False >Thread-30773::DEBUG::2012-12-02 14:57:35,456::task::568::TaskManager.Task::(_updateState) Task=`9ca345a4-0b57-4de7-bafa-a8f02e40ac12`::moving from state init -> state preparing >Thread-30773::INFO::2012-12-02 14:57:35,456::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30773::INFO::2012-12-02 14:57:35,457::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00336408615112', 'lastCheck': 1354453053.423265, 'code': 0, 'valid': True}} >Thread-30773::DEBUG::2012-12-02 14:57:35,457::task::1151::TaskManager.Task::(prepare) Task=`9ca345a4-0b57-4de7-bafa-a8f02e40ac12`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00336408615112', 'lastCheck': 1354453053.423265, 'code': 0, 'valid': True}} >Thread-30773::DEBUG::2012-12-02 14:57:35,457::task::568::TaskManager.Task::(_updateState) Task=`9ca345a4-0b57-4de7-bafa-a8f02e40ac12`::moving from state preparing -> state finished >Thread-30773::DEBUG::2012-12-02 14:57:35,457::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30773::DEBUG::2012-12-02 14:57:35,457::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30773::DEBUG::2012-12-02 14:57:35,457::task::957::TaskManager.Task::(_decref) Task=`9ca345a4-0b57-4de7-bafa-a8f02e40ac12`::ref 0 aborting False >Thread-30779::DEBUG::2012-12-02 14:57:45,553::task::568::TaskManager.Task::(_updateState) Task=`1da84849-a573-4140-bcdc-e0733a08ef90`::moving from state init -> state preparing >Thread-30779::INFO::2012-12-02 14:57:45,553::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30779::INFO::2012-12-02 14:57:45,554::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00406503677368', 'lastCheck': 1354453063.436785, 'code': 0, 'valid': True}} >Thread-30779::DEBUG::2012-12-02 14:57:45,554::task::1151::TaskManager.Task::(prepare) Task=`1da84849-a573-4140-bcdc-e0733a08ef90`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00406503677368', 'lastCheck': 1354453063.436785, 'code': 0, 'valid': True}} >Thread-30779::DEBUG::2012-12-02 14:57:45,554::task::568::TaskManager.Task::(_updateState) Task=`1da84849-a573-4140-bcdc-e0733a08ef90`::moving from state preparing -> state finished >Thread-30779::DEBUG::2012-12-02 14:57:45,554::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30779::DEBUG::2012-12-02 14:57:45,554::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30779::DEBUG::2012-12-02 14:57:45,554::task::957::TaskManager.Task::(_decref) Task=`1da84849-a573-4140-bcdc-e0733a08ef90`::ref 0 aborting False >Thread-30784::DEBUG::2012-12-02 14:57:52,932::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30784::DEBUG::2012-12-02 14:57:52,932::task::568::TaskManager.Task::(_updateState) Task=`3a848317-22c3-4439-b8f3-2de876ba0fe3`::moving from state init -> state preparing >Thread-30784::INFO::2012-12-02 14:57:52,932::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175', 'port': ''}], options=None) >Thread-30784::INFO::2012-12-02 14:57:52,932::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30784::DEBUG::2012-12-02 14:57:52,932::task::1151::TaskManager.Task::(prepare) Task=`3a848317-22c3-4439-b8f3-2de876ba0fe3`::finished: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30784::DEBUG::2012-12-02 14:57:52,933::task::568::TaskManager.Task::(_updateState) Task=`3a848317-22c3-4439-b8f3-2de876ba0fe3`::moving from state preparing -> state finished >Thread-30784::DEBUG::2012-12-02 14:57:52,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30784::DEBUG::2012-12-02 14:57:52,933::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30784::DEBUG::2012-12-02 14:57:52,933::task::957::TaskManager.Task::(_decref) Task=`3a848317-22c3-4439-b8f3-2de876ba0fe3`::ref 0 aborting False >Thread-30785::DEBUG::2012-12-02 14:57:52,953::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30785::DEBUG::2012-12-02 14:57:52,954::task::568::TaskManager.Task::(_updateState) Task=`e8c3f8d3-095a-459e-9046-5061cef3fd07`::moving from state init -> state preparing >Thread-30785::INFO::2012-12-02 14:57:52,954::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175', 'port': ''}], options=None) >Thread-30785::DEBUG::2012-12-02 14:57:52,958::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30785::DEBUG::2012-12-02 14:57:52,958::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30785::DEBUG::2012-12-02 14:57:52,958::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30785::DEBUG::2012-12-02 14:57:52,959::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30785::DEBUG::2012-12-02 14:57:52,959::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30785::DEBUG::2012-12-02 14:57:52,959::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30785::INFO::2012-12-02 14:57:52,959::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30785::DEBUG::2012-12-02 14:57:52,959::task::1151::TaskManager.Task::(prepare) Task=`e8c3f8d3-095a-459e-9046-5061cef3fd07`::finished: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-30785::DEBUG::2012-12-02 14:57:52,959::task::568::TaskManager.Task::(_updateState) Task=`e8c3f8d3-095a-459e-9046-5061cef3fd07`::moving from state preparing -> state finished >Thread-30785::DEBUG::2012-12-02 14:57:52,959::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30785::DEBUG::2012-12-02 14:57:52,959::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30785::DEBUG::2012-12-02 14:57:52,959::task::957::TaskManager.Task::(_decref) Task=`e8c3f8d3-095a-459e-9046-5061cef3fd07`::ref 0 aborting False >Thread-30787::DEBUG::2012-12-02 14:57:55,706::task::568::TaskManager.Task::(_updateState) Task=`05f123a1-a611-4069-bfb7-a98665bb9823`::moving from state init -> state preparing >Thread-30787::INFO::2012-12-02 14:57:55,706::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30787::INFO::2012-12-02 14:57:55,707::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00494003295898', 'lastCheck': 1354453073.4520781, 'code': 0, 'valid': True}} >Thread-30787::DEBUG::2012-12-02 14:57:55,707::task::1151::TaskManager.Task::(prepare) Task=`05f123a1-a611-4069-bfb7-a98665bb9823`::finished: {'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00494003295898', 'lastCheck': 1354453073.4520781, 'code': 0, 'valid': True}} >Thread-30787::DEBUG::2012-12-02 14:57:55,707::task::568::TaskManager.Task::(_updateState) Task=`05f123a1-a611-4069-bfb7-a98665bb9823`::moving from state preparing -> state finished >Thread-30787::DEBUG::2012-12-02 14:57:55,707::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30787::DEBUG::2012-12-02 14:57:55,707::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30787::DEBUG::2012-12-02 14:57:55,707::task::957::TaskManager.Task::(_decref) Task=`05f123a1-a611-4069-bfb7-a98665bb9823`::ref 0 aborting False >Thread-30789::DEBUG::2012-12-02 14:57:57,740::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-30789::DEBUG::2012-12-02 14:57:57,741::task::568::TaskManager.Task::(_updateState) Task=`b15ffb01-1344-4801-9238-8e4124254060`::moving from state init -> state preparing >Thread-30789::INFO::2012-12-02 14:57:57,741::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='a3177eae-1dc9-445a-92b1-da23ec8008bc', masterVersion=1, options=None) >Thread-30789::DEBUG::2012-12-02 14:57:57,741::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`0a2c63d1-c1bc-49fa-8c28-73a2eccede59`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-30789::DEBUG::2012-12-02 14:57:57,741::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-30789::DEBUG::2012-12-02 14:57:57,742::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-30789::DEBUG::2012-12-02 14:57:57,742::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`0a2c63d1-c1bc-49fa-8c28-73a2eccede59`::Granted request >Thread-30789::DEBUG::2012-12-02 14:57:57,742::task::794::TaskManager.Task::(resourceAcquired) Task=`b15ffb01-1344-4801-9238-8e4124254060`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-30789::DEBUG::2012-12-02 14:57:57,742::task::957::TaskManager.Task::(_decref) Task=`b15ffb01-1344-4801-9238-8e4124254060`::ref 1 aborting False >Thread-30789::DEBUG::2012-12-02 14:57:57,742::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-30789::DEBUG::2012-12-02 14:57:57,742::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30789::DEBUG::2012-12-02 14:57:57,743::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-30789::DEBUG::2012-12-02 14:57:57,743::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30789::DEBUG::2012-12-02 14:57:57,743::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-30789::DEBUG::2012-12-02 14:57:57,771::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30789::DEBUG::2012-12-02 14:57:57,771::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30789::DEBUG::2012-12-02 14:57:58,412::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-30789::DEBUG::2012-12-02 14:57:58,494::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30789::DEBUG::2012-12-02 14:57:58,494::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,494::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,494::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,495::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,495::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,495::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,495::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30789::INFO::2012-12-02 14:57:58,495::fileSD::260::Storage.StorageDomain::(validate) sdUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30789::DEBUG::2012-12-02 14:57:58,514::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'ROLE=Master', 'SDUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bbdf1746b3e52fda773ad5ef53a5ddd4d9d01ec3'] >Thread-30789::DEBUG::2012-12-02 14:57:58,515::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,515::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,515::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,515::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,515::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,516::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:58,516::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-30789::DEBUG::2012-12-02 14:57:58,516::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30789::DEBUG::2012-12-02 14:57:58,516::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-30789::DEBUG::2012-12-02 14:57:58,516::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30789::DEBUG::2012-12-02 14:57:58,516::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-30789::DEBUG::2012-12-02 14:57:58,543::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30789::DEBUG::2012-12-02 14:57:58,543::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30789::DEBUG::2012-12-02 14:57:59,177::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-30789::DEBUG::2012-12-02 14:57:59,250::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-30789::DEBUG::2012-12-02 14:57:59,251::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,251::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,251::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,251::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,251::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,252::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,252::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30789::DEBUG::2012-12-02 14:57:59,252::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,253::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-30789::DEBUG::2012-12-02 14:57:59,464::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-30789::WARNING::2012-12-02 14:57:59,466::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-30789::DEBUG::2012-12-02 14:57:59,466::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,501::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30789::DEBUG::2012-12-02 14:57:59,501::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-30789::DEBUG::2012-12-02 14:57:59,511::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'ROLE=Master', 'SDUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bbdf1746b3e52fda773ad5ef53a5ddd4d9d01ec3'] >Thread-30789::DEBUG::2012-12-02 14:57:59,514::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-30789::WARNING::2012-12-02 14:57:59,514::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a3177eae-1dc9-445a-92b1-da23ec8008bc_imageNS already registered >Thread-30789::WARNING::2012-12-02 14:57:59,514::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a3177eae-1dc9-445a-92b1-da23ec8008bc_volumeNS already registered >Thread-30789::DEBUG::2012-12-02 14:57:59,514::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain a3177eae-1dc9-445a-92b1-da23ec8008bc verified, version 1 >Thread-30789::DEBUG::2012-12-02 14:57:59,514::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-30789::DEBUG::2012-12-02 14:57:59,515::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-30789::DEBUG::2012-12-02 14:57:59,521::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'ROLE=Master', 'SDUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bbdf1746b3e52fda773ad5ef53a5ddd4d9d01ec3'] >Thread-30789::DEBUG::2012-12-02 14:57:59,521::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,522::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-30789::DEBUG::2012-12-02 14:57:59,731::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-30789::WARNING::2012-12-02 14:57:59,733::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-30789::DEBUG::2012-12-02 14:57:59,733::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,767::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-30789::DEBUG::2012-12-02 14:57:59,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-30789::DEBUG::2012-12-02 14:57:59,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Regular', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7867c42a2ec25b1e27a5f200d36e7105ccf17687'] >Thread-30789::DEBUG::2012-12-02 14:57:59,811::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-30789::DEBUG::2012-12-02 14:57:59,811::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '4e891d8c-9947-468e-88ec-29e49d68e083_imageNS' >Thread-30789::DEBUG::2012-12-02 14:57:59,811::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '4e891d8c-9947-468e-88ec-29e49d68e083_volumeNS' >Thread-30789::DEBUG::2012-12-02 14:57:59,812::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `da40b79b-9f51-4240-9cb2-2ab10e793fb6` started monitoring domain `4e891d8c-9947-468e-88ec-29e49d68e083` >Thread-30789::DEBUG::2012-12-02 14:57:59,812::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-30789::DEBUG::2012-12-02 14:57:59,814::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30789::DEBUG::2012-12-02 14:57:59,815::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-30910::INFO::2012-12-02 14:57:59,832::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-30910::DEBUG::2012-12-02 14:57:59,832::safelease::178::SANLock::(acquireHostId) Host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully acquired (id: 3) >Thread-30789::DEBUG::2012-12-02 14:58:00,024::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-30789::WARNING::2012-12-02 14:58:00,026::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-30789::DEBUG::2012-12-02 14:58:00,026::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30789::INFO::2012-12-02 14:58:00,054::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-30789::INFO::2012-12-02 14:58:00,055::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30789::INFO::2012-12-02 14:58:00,055::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/mastersd >Thread-30789::INFO::2012-12-02 14:58:00,055::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-30789::DEBUG::2012-12-02 14:58:00,055::task::1151::TaskManager.Task::(prepare) Task=`b15ffb01-1344-4801-9238-8e4124254060`::finished: None >Thread-30789::DEBUG::2012-12-02 14:58:00,056::task::568::TaskManager.Task::(_updateState) Task=`b15ffb01-1344-4801-9238-8e4124254060`::moving from state preparing -> state finished >Thread-30789::DEBUG::2012-12-02 14:58:00,056::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-30789::DEBUG::2012-12-02 14:58:00,056::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30789::DEBUG::2012-12-02 14:58:00,056::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-30789::DEBUG::2012-12-02 14:58:00,056::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-30789::DEBUG::2012-12-02 14:58:00,056::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-30789::DEBUG::2012-12-02 14:58:00,056::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-30789::DEBUG::2012-12-02 14:58:00,057::task::957::TaskManager.Task::(_decref) Task=`b15ffb01-1344-4801-9238-8e4124254060`::ref 0 aborting False >Thread-30971::DEBUG::2012-12-02 14:58:05,817::task::568::TaskManager.Task::(_updateState) Task=`d8df3dae-8d96-42cc-bcaf-efda1987126a`::moving from state init -> state preparing >Thread-30971::INFO::2012-12-02 14:58:05,817::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30971::INFO::2012-12-02 14:58:05,817::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00596189498901', 'lastCheck': 1354453079.8319819, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00792908668518', 'lastCheck': 1354453083.4701481, 'code': 0, 'valid': True}} >Thread-30971::DEBUG::2012-12-02 14:58:05,817::task::1151::TaskManager.Task::(prepare) Task=`d8df3dae-8d96-42cc-bcaf-efda1987126a`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00596189498901', 'lastCheck': 1354453079.8319819, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00792908668518', 'lastCheck': 1354453083.4701481, 'code': 0, 'valid': True}} >Thread-30971::DEBUG::2012-12-02 14:58:05,818::task::568::TaskManager.Task::(_updateState) Task=`d8df3dae-8d96-42cc-bcaf-efda1987126a`::moving from state preparing -> state finished >Thread-30971::DEBUG::2012-12-02 14:58:05,818::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30971::DEBUG::2012-12-02 14:58:05,818::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30971::DEBUG::2012-12-02 14:58:05,818::task::957::TaskManager.Task::(_decref) Task=`d8df3dae-8d96-42cc-bcaf-efda1987126a`::ref 0 aborting False >Thread-30977::DEBUG::2012-12-02 14:58:15,933::task::568::TaskManager.Task::(_updateState) Task=`b3315cce-c27c-467c-8e46-a6d480adee31`::moving from state init -> state preparing >Thread-30977::INFO::2012-12-02 14:58:15,933::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30977::INFO::2012-12-02 14:58:15,934::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00760984420776', 'lastCheck': 1354453089.848804, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00876021385193', 'lastCheck': 1354453093.4892931, 'code': 0, 'valid': True}} >Thread-30977::DEBUG::2012-12-02 14:58:15,934::task::1151::TaskManager.Task::(prepare) Task=`b3315cce-c27c-467c-8e46-a6d480adee31`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00760984420776', 'lastCheck': 1354453089.848804, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00876021385193', 'lastCheck': 1354453093.4892931, 'code': 0, 'valid': True}} >Thread-30977::DEBUG::2012-12-02 14:58:15,934::task::568::TaskManager.Task::(_updateState) Task=`b3315cce-c27c-467c-8e46-a6d480adee31`::moving from state preparing -> state finished >Thread-30977::DEBUG::2012-12-02 14:58:15,934::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30977::DEBUG::2012-12-02 14:58:15,934::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30977::DEBUG::2012-12-02 14:58:15,934::task::957::TaskManager.Task::(_decref) Task=`b3315cce-c27c-467c-8e46-a6d480adee31`::ref 0 aborting False >Thread-30983::DEBUG::2012-12-02 14:58:26,047::task::568::TaskManager.Task::(_updateState) Task=`501404a2-5e32-468c-8665-e3fdf7529000`::moving from state init -> state preparing >Thread-30983::INFO::2012-12-02 14:58:26,047::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30983::INFO::2012-12-02 14:58:26,048::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00742220878601', 'lastCheck': 1354453099.8646829, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00981402397156', 'lastCheck': 1354453103.514065, 'code': 0, 'valid': True}} >Thread-30983::DEBUG::2012-12-02 14:58:26,048::task::1151::TaskManager.Task::(prepare) Task=`501404a2-5e32-468c-8665-e3fdf7529000`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00742220878601', 'lastCheck': 1354453099.8646829, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00981402397156', 'lastCheck': 1354453103.514065, 'code': 0, 'valid': True}} >Thread-30983::DEBUG::2012-12-02 14:58:26,048::task::568::TaskManager.Task::(_updateState) Task=`501404a2-5e32-468c-8665-e3fdf7529000`::moving from state preparing -> state finished >Thread-30983::DEBUG::2012-12-02 14:58:26,048::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30983::DEBUG::2012-12-02 14:58:26,048::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30983::DEBUG::2012-12-02 14:58:26,048::task::957::TaskManager.Task::(_decref) Task=`501404a2-5e32-468c-8665-e3fdf7529000`::ref 0 aborting False >Thread-30989::DEBUG::2012-12-02 14:58:36,166::task::568::TaskManager.Task::(_updateState) Task=`f7fb5b1d-796c-4c09-ad8f-1f99bebbe70b`::moving from state init -> state preparing >Thread-30989::INFO::2012-12-02 14:58:36,166::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30989::INFO::2012-12-02 14:58:36,166::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00877690315247', 'lastCheck': 1354453109.8846381, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00980186462402', 'lastCheck': 1354453113.5329449, 'code': 0, 'valid': True}} >Thread-30989::DEBUG::2012-12-02 14:58:36,167::task::1151::TaskManager.Task::(prepare) Task=`f7fb5b1d-796c-4c09-ad8f-1f99bebbe70b`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00877690315247', 'lastCheck': 1354453109.8846381, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00980186462402', 'lastCheck': 1354453113.5329449, 'code': 0, 'valid': True}} >Thread-30989::DEBUG::2012-12-02 14:58:36,167::task::568::TaskManager.Task::(_updateState) Task=`f7fb5b1d-796c-4c09-ad8f-1f99bebbe70b`::moving from state preparing -> state finished >Thread-30989::DEBUG::2012-12-02 14:58:36,167::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30989::DEBUG::2012-12-02 14:58:36,167::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30989::DEBUG::2012-12-02 14:58:36,167::task::957::TaskManager.Task::(_decref) Task=`f7fb5b1d-796c-4c09-ad8f-1f99bebbe70b`::ref 0 aborting False >Thread-30995::DEBUG::2012-12-02 14:58:46,291::task::568::TaskManager.Task::(_updateState) Task=`4b099f0e-d2a5-45fa-a00a-ca67b467da44`::moving from state init -> state preparing >Thread-30995::INFO::2012-12-02 14:58:46,292::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-30995::INFO::2012-12-02 14:58:46,292::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00811505317688', 'lastCheck': 1354453119.902509, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00926804542542', 'lastCheck': 1354453123.551492, 'code': 0, 'valid': True}} >Thread-30995::DEBUG::2012-12-02 14:58:46,292::task::1151::TaskManager.Task::(prepare) Task=`4b099f0e-d2a5-45fa-a00a-ca67b467da44`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00811505317688', 'lastCheck': 1354453119.902509, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00926804542542', 'lastCheck': 1354453123.551492, 'code': 0, 'valid': True}} >Thread-30995::DEBUG::2012-12-02 14:58:46,292::task::568::TaskManager.Task::(_updateState) Task=`4b099f0e-d2a5-45fa-a00a-ca67b467da44`::moving from state preparing -> state finished >Thread-30995::DEBUG::2012-12-02 14:58:46,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-30995::DEBUG::2012-12-02 14:58:46,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-30995::DEBUG::2012-12-02 14:58:46,292::task::957::TaskManager.Task::(_decref) Task=`4b099f0e-d2a5-45fa-a00a-ca67b467da44`::ref 0 aborting False >Thread-31001::DEBUG::2012-12-02 14:58:56,410::task::568::TaskManager.Task::(_updateState) Task=`c35c94eb-2ab9-49c5-8c91-63f2fdd82c27`::moving from state init -> state preparing >Thread-31001::INFO::2012-12-02 14:58:56,411::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31001::INFO::2012-12-02 14:58:56,411::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00810313224792', 'lastCheck': 1354453129.9189291, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0101010799408', 'lastCheck': 1354453133.572439, 'code': 0, 'valid': True}} >Thread-31001::DEBUG::2012-12-02 14:58:56,411::task::1151::TaskManager.Task::(prepare) Task=`c35c94eb-2ab9-49c5-8c91-63f2fdd82c27`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00810313224792', 'lastCheck': 1354453129.9189291, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0101010799408', 'lastCheck': 1354453133.572439, 'code': 0, 'valid': True}} >Thread-31001::DEBUG::2012-12-02 14:58:56,411::task::568::TaskManager.Task::(_updateState) Task=`c35c94eb-2ab9-49c5-8c91-63f2fdd82c27`::moving from state preparing -> state finished >Thread-31001::DEBUG::2012-12-02 14:58:56,411::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31001::DEBUG::2012-12-02 14:58:56,411::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31001::DEBUG::2012-12-02 14:58:56,411::task::957::TaskManager.Task::(_decref) Task=`c35c94eb-2ab9-49c5-8c91-63f2fdd82c27`::ref 0 aborting False >Thread-31007::DEBUG::2012-12-02 14:59:06,540::task::568::TaskManager.Task::(_updateState) Task=`5c7c6a7e-8f57-4ae5-806e-ec05d661854d`::moving from state init -> state preparing >Thread-31007::INFO::2012-12-02 14:59:06,540::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31007::INFO::2012-12-02 14:59:06,540::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00763583183289', 'lastCheck': 1354453139.9376919, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00892019271851', 'lastCheck': 1354453143.590651, 'code': 0, 'valid': True}} >Thread-31007::DEBUG::2012-12-02 14:59:06,540::task::1151::TaskManager.Task::(prepare) Task=`5c7c6a7e-8f57-4ae5-806e-ec05d661854d`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00763583183289', 'lastCheck': 1354453139.9376919, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00892019271851', 'lastCheck': 1354453143.590651, 'code': 0, 'valid': True}} >Thread-31007::DEBUG::2012-12-02 14:59:06,541::task::568::TaskManager.Task::(_updateState) Task=`5c7c6a7e-8f57-4ae5-806e-ec05d661854d`::moving from state preparing -> state finished >Thread-31007::DEBUG::2012-12-02 14:59:06,541::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31007::DEBUG::2012-12-02 14:59:06,541::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31007::DEBUG::2012-12-02 14:59:06,541::task::957::TaskManager.Task::(_decref) Task=`5c7c6a7e-8f57-4ae5-806e-ec05d661854d`::ref 0 aborting False >Thread-31013::DEBUG::2012-12-02 14:59:16,714::task::568::TaskManager.Task::(_updateState) Task=`607eee12-1370-4974-8d94-251e41d28f0b`::moving from state init -> state preparing >Thread-31013::INFO::2012-12-02 14:59:16,714::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31013::INFO::2012-12-02 14:59:16,715::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00821495056152', 'lastCheck': 1354453149.955338, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00890111923218', 'lastCheck': 1354453153.60972, 'code': 0, 'valid': True}} >Thread-31013::DEBUG::2012-12-02 14:59:16,715::task::1151::TaskManager.Task::(prepare) Task=`607eee12-1370-4974-8d94-251e41d28f0b`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00821495056152', 'lastCheck': 1354453149.955338, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00890111923218', 'lastCheck': 1354453153.60972, 'code': 0, 'valid': True}} >Thread-31013::DEBUG::2012-12-02 14:59:16,715::task::568::TaskManager.Task::(_updateState) Task=`607eee12-1370-4974-8d94-251e41d28f0b`::moving from state preparing -> state finished >Thread-31013::DEBUG::2012-12-02 14:59:16,715::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31013::DEBUG::2012-12-02 14:59:16,715::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31013::DEBUG::2012-12-02 14:59:16,715::task::957::TaskManager.Task::(_decref) Task=`607eee12-1370-4974-8d94-251e41d28f0b`::ref 0 aborting False >Thread-31019::DEBUG::2012-12-02 14:59:26,840::task::568::TaskManager.Task::(_updateState) Task=`431a9102-b63d-4cad-a79c-460b8b654899`::moving from state init -> state preparing >Thread-31019::INFO::2012-12-02 14:59:26,840::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31019::INFO::2012-12-02 14:59:26,841::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.007807970047', 'lastCheck': 1354453159.973196, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00821995735168', 'lastCheck': 1354453163.632416, 'code': 0, 'valid': True}} >Thread-31019::DEBUG::2012-12-02 14:59:26,841::task::1151::TaskManager.Task::(prepare) Task=`431a9102-b63d-4cad-a79c-460b8b654899`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.007807970047', 'lastCheck': 1354453159.973196, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00821995735168', 'lastCheck': 1354453163.632416, 'code': 0, 'valid': True}} >Thread-31019::DEBUG::2012-12-02 14:59:26,841::task::568::TaskManager.Task::(_updateState) Task=`431a9102-b63d-4cad-a79c-460b8b654899`::moving from state preparing -> state finished >Thread-31019::DEBUG::2012-12-02 14:59:26,841::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31019::DEBUG::2012-12-02 14:59:26,841::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31019::DEBUG::2012-12-02 14:59:26,841::task::957::TaskManager.Task::(_decref) Task=`431a9102-b63d-4cad-a79c-460b8b654899`::ref 0 aborting False >Thread-31025::DEBUG::2012-12-02 14:59:36,958::task::568::TaskManager.Task::(_updateState) Task=`e678f5ec-7295-4674-ad0d-1c2147f71e9a`::moving from state init -> state preparing >Thread-31025::INFO::2012-12-02 14:59:36,958::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31025::INFO::2012-12-02 14:59:36,958::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00871396064758', 'lastCheck': 1354453169.989861, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00970196723938', 'lastCheck': 1354453173.651366, 'code': 0, 'valid': True}} >Thread-31025::DEBUG::2012-12-02 14:59:36,958::task::1151::TaskManager.Task::(prepare) Task=`e678f5ec-7295-4674-ad0d-1c2147f71e9a`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00871396064758', 'lastCheck': 1354453169.989861, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00970196723938', 'lastCheck': 1354453173.651366, 'code': 0, 'valid': True}} >Thread-31025::DEBUG::2012-12-02 14:59:36,959::task::568::TaskManager.Task::(_updateState) Task=`e678f5ec-7295-4674-ad0d-1c2147f71e9a`::moving from state preparing -> state finished >Thread-31025::DEBUG::2012-12-02 14:59:36,959::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31025::DEBUG::2012-12-02 14:59:36,959::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31025::DEBUG::2012-12-02 14:59:36,959::task::957::TaskManager.Task::(_decref) Task=`e678f5ec-7295-4674-ad0d-1c2147f71e9a`::ref 0 aborting False >Thread-31031::DEBUG::2012-12-02 14:59:47,084::task::568::TaskManager.Task::(_updateState) Task=`64356811-ef99-4dff-9ba4-de37cc647f98`::moving from state init -> state preparing >Thread-31031::INFO::2012-12-02 14:59:47,084::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31031::INFO::2012-12-02 14:59:47,084::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0125200748444', 'lastCheck': 1354453180.0155699, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00886797904968', 'lastCheck': 1354453183.670871, 'code': 0, 'valid': True}} >Thread-31031::DEBUG::2012-12-02 14:59:47,084::task::1151::TaskManager.Task::(prepare) Task=`64356811-ef99-4dff-9ba4-de37cc647f98`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0125200748444', 'lastCheck': 1354453180.0155699, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00886797904968', 'lastCheck': 1354453183.670871, 'code': 0, 'valid': True}} >Thread-31031::DEBUG::2012-12-02 14:59:47,085::task::568::TaskManager.Task::(_updateState) Task=`64356811-ef99-4dff-9ba4-de37cc647f98`::moving from state preparing -> state finished >Thread-31031::DEBUG::2012-12-02 14:59:47,085::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31031::DEBUG::2012-12-02 14:59:47,085::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31031::DEBUG::2012-12-02 14:59:47,085::task::957::TaskManager.Task::(_decref) Task=`64356811-ef99-4dff-9ba4-de37cc647f98`::ref 0 aborting False >Thread-31037::DEBUG::2012-12-02 14:59:57,208::task::568::TaskManager.Task::(_updateState) Task=`3aa8b4e9-8919-474f-ad22-d35d36b58969`::moving from state init -> state preparing >Thread-31037::INFO::2012-12-02 14:59:57,208::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31037::INFO::2012-12-02 14:59:57,209::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0123999118805', 'lastCheck': 1354453190.037648, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00411009788513', 'lastCheck': 1354453193.6840751, 'code': 0, 'valid': True}} >Thread-31037::DEBUG::2012-12-02 14:59:57,209::task::1151::TaskManager.Task::(prepare) Task=`3aa8b4e9-8919-474f-ad22-d35d36b58969`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0123999118805', 'lastCheck': 1354453190.037648, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00411009788513', 'lastCheck': 1354453193.6840751, 'code': 0, 'valid': True}} >Thread-31037::DEBUG::2012-12-02 14:59:57,209::task::568::TaskManager.Task::(_updateState) Task=`3aa8b4e9-8919-474f-ad22-d35d36b58969`::moving from state preparing -> state finished >Thread-31037::DEBUG::2012-12-02 14:59:57,209::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31037::DEBUG::2012-12-02 14:59:57,209::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31037::DEBUG::2012-12-02 14:59:57,209::task::957::TaskManager.Task::(_decref) Task=`3aa8b4e9-8919-474f-ad22-d35d36b58969`::ref 0 aborting False >Thread-31043::DEBUG::2012-12-02 15:00:07,338::task::568::TaskManager.Task::(_updateState) Task=`b54dc562-ef98-4671-b400-a91deaf772fe`::moving from state init -> state preparing >Thread-31043::INFO::2012-12-02 15:00:07,338::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31043::INFO::2012-12-02 15:00:07,339::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00889205932617', 'lastCheck': 1354453200.0582299, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00862503051758', 'lastCheck': 1354453203.703208, 'code': 0, 'valid': True}} >Thread-31043::DEBUG::2012-12-02 15:00:07,339::task::1151::TaskManager.Task::(prepare) Task=`b54dc562-ef98-4671-b400-a91deaf772fe`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00889205932617', 'lastCheck': 1354453200.0582299, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00862503051758', 'lastCheck': 1354453203.703208, 'code': 0, 'valid': True}} >Thread-31043::DEBUG::2012-12-02 15:00:07,339::task::568::TaskManager.Task::(_updateState) Task=`b54dc562-ef98-4671-b400-a91deaf772fe`::moving from state preparing -> state finished >Thread-31043::DEBUG::2012-12-02 15:00:07,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31043::DEBUG::2012-12-02 15:00:07,340::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31043::DEBUG::2012-12-02 15:00:07,340::task::957::TaskManager.Task::(_decref) Task=`b54dc562-ef98-4671-b400-a91deaf772fe`::ref 0 aborting False >Thread-31049::DEBUG::2012-12-02 15:00:17,455::task::568::TaskManager.Task::(_updateState) Task=`c9a2049a-9cae-4bcd-a3d9-13186908da9d`::moving from state init -> state preparing >Thread-31049::INFO::2012-12-02 15:00:17,456::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31049::INFO::2012-12-02 15:00:17,456::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00614500045776', 'lastCheck': 1354453210.0730901, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00870990753174', 'lastCheck': 1354453213.72192, 'code': 0, 'valid': True}} >Thread-31049::DEBUG::2012-12-02 15:00:17,456::task::1151::TaskManager.Task::(prepare) Task=`c9a2049a-9cae-4bcd-a3d9-13186908da9d`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00614500045776', 'lastCheck': 1354453210.0730901, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00870990753174', 'lastCheck': 1354453213.72192, 'code': 0, 'valid': True}} >Thread-31049::DEBUG::2012-12-02 15:00:17,456::task::568::TaskManager.Task::(_updateState) Task=`c9a2049a-9cae-4bcd-a3d9-13186908da9d`::moving from state preparing -> state finished >Thread-31049::DEBUG::2012-12-02 15:00:17,456::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31049::DEBUG::2012-12-02 15:00:17,456::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31049::DEBUG::2012-12-02 15:00:17,456::task::957::TaskManager.Task::(_decref) Task=`c9a2049a-9cae-4bcd-a3d9-13186908da9d`::ref 0 aborting False >Thread-31055::DEBUG::2012-12-02 15:00:27,579::task::568::TaskManager.Task::(_updateState) Task=`9c65620a-f153-4989-8c4e-bd5a058c5f4b`::moving from state init -> state preparing >Thread-31055::INFO::2012-12-02 15:00:27,579::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31055::INFO::2012-12-02 15:00:27,579::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00655007362366', 'lastCheck': 1354453220.08834, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00977396965027', 'lastCheck': 1354453223.7457731, 'code': 0, 'valid': True}} >Thread-31055::DEBUG::2012-12-02 15:00:27,579::task::1151::TaskManager.Task::(prepare) Task=`9c65620a-f153-4989-8c4e-bd5a058c5f4b`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00655007362366', 'lastCheck': 1354453220.08834, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00977396965027', 'lastCheck': 1354453223.7457731, 'code': 0, 'valid': True}} >Thread-31055::DEBUG::2012-12-02 15:00:27,579::task::568::TaskManager.Task::(_updateState) Task=`9c65620a-f153-4989-8c4e-bd5a058c5f4b`::moving from state preparing -> state finished >Thread-31055::DEBUG::2012-12-02 15:00:27,579::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31055::DEBUG::2012-12-02 15:00:27,580::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31055::DEBUG::2012-12-02 15:00:27,580::task::957::TaskManager.Task::(_decref) Task=`9c65620a-f153-4989-8c4e-bd5a058c5f4b`::ref 0 aborting False >Thread-31061::DEBUG::2012-12-02 15:00:37,698::task::568::TaskManager.Task::(_updateState) Task=`5005a148-aa9e-4d5f-b957-76629ad7566b`::moving from state init -> state preparing >Thread-31061::INFO::2012-12-02 15:00:37,698::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31061::INFO::2012-12-02 15:00:37,698::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00932598114014', 'lastCheck': 1354453230.1083119, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00913691520691', 'lastCheck': 1354453233.765059, 'code': 0, 'valid': True}} >Thread-31061::DEBUG::2012-12-02 15:00:37,698::task::1151::TaskManager.Task::(prepare) Task=`5005a148-aa9e-4d5f-b957-76629ad7566b`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00932598114014', 'lastCheck': 1354453230.1083119, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00913691520691', 'lastCheck': 1354453233.765059, 'code': 0, 'valid': True}} >Thread-31061::DEBUG::2012-12-02 15:00:37,698::task::568::TaskManager.Task::(_updateState) Task=`5005a148-aa9e-4d5f-b957-76629ad7566b`::moving from state preparing -> state finished >Thread-31061::DEBUG::2012-12-02 15:00:37,698::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31061::DEBUG::2012-12-02 15:00:37,699::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31061::DEBUG::2012-12-02 15:00:37,699::task::957::TaskManager.Task::(_decref) Task=`5005a148-aa9e-4d5f-b957-76629ad7566b`::ref 0 aborting False >Thread-31067::DEBUG::2012-12-02 15:00:47,826::task::568::TaskManager.Task::(_updateState) Task=`504524f5-6606-4025-a97f-f4e5b74374d9`::moving from state init -> state preparing >Thread-31067::INFO::2012-12-02 15:00:47,826::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31067::INFO::2012-12-02 15:00:47,826::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00796890258789', 'lastCheck': 1354453240.1270621, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00806498527527', 'lastCheck': 1354453243.783813, 'code': 0, 'valid': True}} >Thread-31067::DEBUG::2012-12-02 15:00:47,826::task::1151::TaskManager.Task::(prepare) Task=`504524f5-6606-4025-a97f-f4e5b74374d9`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00796890258789', 'lastCheck': 1354453240.1270621, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00806498527527', 'lastCheck': 1354453243.783813, 'code': 0, 'valid': True}} >Thread-31067::DEBUG::2012-12-02 15:00:47,826::task::568::TaskManager.Task::(_updateState) Task=`504524f5-6606-4025-a97f-f4e5b74374d9`::moving from state preparing -> state finished >Thread-31067::DEBUG::2012-12-02 15:00:47,827::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31067::DEBUG::2012-12-02 15:00:47,827::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31067::DEBUG::2012-12-02 15:00:47,827::task::957::TaskManager.Task::(_decref) Task=`504524f5-6606-4025-a97f-f4e5b74374d9`::ref 0 aborting False >Thread-30654::DEBUG::2012-12-02 15:00:53,783::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30654::DEBUG::2012-12-02 15:00:53,784::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30654::DEBUG::2012-12-02 15:00:53,784::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-30654::DEBUG::2012-12-02 15:00:53,998::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-30654::WARNING::2012-12-02 15:00:54,000::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-30654::DEBUG::2012-12-02 15:00:54,000::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31113::WARNING::2012-12-02 15:00:54,019::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31119::WARNING::2012-12-02 15:00:54,022::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31121::WARNING::2012-12-02 15:00:54,023::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31122::WARNING::2012-12-02 15:00:54,023::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31123::WARNING::2012-12-02 15:00:54,024::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31124::WARNING::2012-12-02 15:00:54,024::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31125::WARNING::2012-12-02 15:00:54,025::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31126::WARNING::2012-12-02 15:00:54,026::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31127::WARNING::2012-12-02 15:00:54,027::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31128::WARNING::2012-12-02 15:00:54,027::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31129::WARNING::2012-12-02 15:00:54,028::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31130::WARNING::2012-12-02 15:00:54,028::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-30654::DEBUG::2012-12-02 15:00:54,044::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30654::DEBUG::2012-12-02 15:00:54,044::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-30654::DEBUG::2012-12-02 15:00:54,049::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a1', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1446', 'ROLE=Master', 'SDUUID=a3177eae-1dc9-445a-92b1-da23ec8008bc', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=bbdf1746b3e52fda773ad5ef53a5ddd4d9d01ec3'] >Thread-30654::DEBUG::2012-12-02 15:00:54,051::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-30654::WARNING::2012-12-02 15:00:54,051::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a3177eae-1dc9-445a-92b1-da23ec8008bc_imageNS already registered >Thread-30654::WARNING::2012-12-02 15:00:54,051::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a3177eae-1dc9-445a-92b1-da23ec8008bc_volumeNS already registered >Thread-31132::DEBUG::2012-12-02 15:00:57,936::task::568::TaskManager.Task::(_updateState) Task=`da3e48cf-14a5-48f8-a6a0-464f786ab8c9`::moving from state init -> state preparing >Thread-31132::INFO::2012-12-02 15:00:57,936::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31132::INFO::2012-12-02 15:00:57,936::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00793409347534', 'lastCheck': 1354453250.145668, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00341296195984', 'lastCheck': 1354453254.0644231, 'code': 0, 'valid': True}} >Thread-31132::DEBUG::2012-12-02 15:00:57,936::task::1151::TaskManager.Task::(prepare) Task=`da3e48cf-14a5-48f8-a6a0-464f786ab8c9`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00793409347534', 'lastCheck': 1354453250.145668, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00341296195984', 'lastCheck': 1354453254.0644231, 'code': 0, 'valid': True}} >Thread-31132::DEBUG::2012-12-02 15:00:57,936::task::568::TaskManager.Task::(_updateState) Task=`da3e48cf-14a5-48f8-a6a0-464f786ab8c9`::moving from state preparing -> state finished >Thread-31132::DEBUG::2012-12-02 15:00:57,937::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31132::DEBUG::2012-12-02 15:00:57,937::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31132::DEBUG::2012-12-02 15:00:57,937::task::957::TaskManager.Task::(_decref) Task=`da3e48cf-14a5-48f8-a6a0-464f786ab8c9`::ref 0 aborting False >Thread-31138::DEBUG::2012-12-02 15:01:08,046::task::568::TaskManager.Task::(_updateState) Task=`b1560411-00c5-4ed6-a7fc-0e3017ee8487`::moving from state init -> state preparing >Thread-31138::INFO::2012-12-02 15:01:08,049::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31138::INFO::2012-12-02 15:01:08,049::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00817584991455', 'lastCheck': 1354453260.167661, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00821614265442', 'lastCheck': 1354453264.0850251, 'code': 0, 'valid': True}} >Thread-31138::DEBUG::2012-12-02 15:01:08,049::task::1151::TaskManager.Task::(prepare) Task=`b1560411-00c5-4ed6-a7fc-0e3017ee8487`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00817584991455', 'lastCheck': 1354453260.167661, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00821614265442', 'lastCheck': 1354453264.0850251, 'code': 0, 'valid': True}} >Thread-31138::DEBUG::2012-12-02 15:01:08,050::task::568::TaskManager.Task::(_updateState) Task=`b1560411-00c5-4ed6-a7fc-0e3017ee8487`::moving from state preparing -> state finished >Thread-31138::DEBUG::2012-12-02 15:01:08,050::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31138::DEBUG::2012-12-02 15:01:08,050::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31138::DEBUG::2012-12-02 15:01:08,050::task::957::TaskManager.Task::(_decref) Task=`b1560411-00c5-4ed6-a7fc-0e3017ee8487`::ref 0 aborting False >Thread-31144::DEBUG::2012-12-02 15:01:18,204::task::568::TaskManager.Task::(_updateState) Task=`c42d9980-1965-437e-896d-36740814b7a7`::moving from state init -> state preparing >Thread-31144::INFO::2012-12-02 15:01:18,204::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31144::INFO::2012-12-02 15:01:18,205::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00879120826721', 'lastCheck': 1354453270.185293, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0120820999146', 'lastCheck': 1354453274.108454, 'code': 0, 'valid': True}} >Thread-31144::DEBUG::2012-12-02 15:01:18,205::task::1151::TaskManager.Task::(prepare) Task=`c42d9980-1965-437e-896d-36740814b7a7`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00879120826721', 'lastCheck': 1354453270.185293, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0120820999146', 'lastCheck': 1354453274.108454, 'code': 0, 'valid': True}} >Thread-31144::DEBUG::2012-12-02 15:01:18,205::task::568::TaskManager.Task::(_updateState) Task=`c42d9980-1965-437e-896d-36740814b7a7`::moving from state preparing -> state finished >Thread-31144::DEBUG::2012-12-02 15:01:18,205::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31144::DEBUG::2012-12-02 15:01:18,205::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31144::DEBUG::2012-12-02 15:01:18,205::task::957::TaskManager.Task::(_decref) Task=`c42d9980-1965-437e-896d-36740814b7a7`::ref 0 aborting False >Thread-31150::DEBUG::2012-12-02 15:01:28,324::task::568::TaskManager.Task::(_updateState) Task=`9ca930b8-2583-426a-bfe5-5a0ea5dccbb6`::moving from state init -> state preparing >Thread-31150::INFO::2012-12-02 15:01:28,324::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31150::INFO::2012-12-02 15:01:28,324::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101351737976', 'lastCheck': 1354453280.2055049, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00817513465881', 'lastCheck': 1354453284.1317699, 'code': 0, 'valid': True}} >Thread-31150::DEBUG::2012-12-02 15:01:28,324::task::1151::TaskManager.Task::(prepare) Task=`9ca930b8-2583-426a-bfe5-5a0ea5dccbb6`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101351737976', 'lastCheck': 1354453280.2055049, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00817513465881', 'lastCheck': 1354453284.1317699, 'code': 0, 'valid': True}} >Thread-31150::DEBUG::2012-12-02 15:01:28,325::task::568::TaskManager.Task::(_updateState) Task=`9ca930b8-2583-426a-bfe5-5a0ea5dccbb6`::moving from state preparing -> state finished >Thread-31150::DEBUG::2012-12-02 15:01:28,325::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31150::DEBUG::2012-12-02 15:01:28,325::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31150::DEBUG::2012-12-02 15:01:28,325::task::957::TaskManager.Task::(_decref) Task=`9ca930b8-2583-426a-bfe5-5a0ea5dccbb6`::ref 0 aborting False >Thread-31156::DEBUG::2012-12-02 15:01:38,443::task::568::TaskManager.Task::(_updateState) Task=`6b6c9cc4-53cb-465d-b54e-605a2616d80b`::moving from state init -> state preparing >Thread-31156::INFO::2012-12-02 15:01:38,443::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31156::INFO::2012-12-02 15:01:38,444::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00899291038513', 'lastCheck': 1354453290.2246399, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00956892967224', 'lastCheck': 1354453294.1507881, 'code': 0, 'valid': True}} >Thread-31156::DEBUG::2012-12-02 15:01:38,444::task::1151::TaskManager.Task::(prepare) Task=`6b6c9cc4-53cb-465d-b54e-605a2616d80b`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00899291038513', 'lastCheck': 1354453290.2246399, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00956892967224', 'lastCheck': 1354453294.1507881, 'code': 0, 'valid': True}} >Thread-31156::DEBUG::2012-12-02 15:01:38,444::task::568::TaskManager.Task::(_updateState) Task=`6b6c9cc4-53cb-465d-b54e-605a2616d80b`::moving from state preparing -> state finished >Thread-31156::DEBUG::2012-12-02 15:01:38,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31156::DEBUG::2012-12-02 15:01:38,444::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31156::DEBUG::2012-12-02 15:01:38,444::task::957::TaskManager.Task::(_decref) Task=`6b6c9cc4-53cb-465d-b54e-605a2616d80b`::ref 0 aborting False >Thread-31162::DEBUG::2012-12-02 15:01:48,566::task::568::TaskManager.Task::(_updateState) Task=`d3a2f60a-5d06-420e-93a6-11cd626e5857`::moving from state init -> state preparing >Thread-31162::INFO::2012-12-02 15:01:48,567::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31162::INFO::2012-12-02 15:01:48,567::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00897002220154', 'lastCheck': 1354453300.2428019, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00926494598389', 'lastCheck': 1354453304.1697171, 'code': 0, 'valid': True}} >Thread-31162::DEBUG::2012-12-02 15:01:48,567::task::1151::TaskManager.Task::(prepare) Task=`d3a2f60a-5d06-420e-93a6-11cd626e5857`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00897002220154', 'lastCheck': 1354453300.2428019, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00926494598389', 'lastCheck': 1354453304.1697171, 'code': 0, 'valid': True}} >Thread-31162::DEBUG::2012-12-02 15:01:48,567::task::568::TaskManager.Task::(_updateState) Task=`d3a2f60a-5d06-420e-93a6-11cd626e5857`::moving from state preparing -> state finished >Thread-31162::DEBUG::2012-12-02 15:01:48,567::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31162::DEBUG::2012-12-02 15:01:48,567::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31162::DEBUG::2012-12-02 15:01:48,568::task::957::TaskManager.Task::(_decref) Task=`d3a2f60a-5d06-420e-93a6-11cd626e5857`::ref 0 aborting False >Thread-31168::DEBUG::2012-12-02 15:01:58,694::task::568::TaskManager.Task::(_updateState) Task=`eb97fd5a-972f-48bb-926e-2eb23b58ba3d`::moving from state init -> state preparing >Thread-31168::INFO::2012-12-02 15:01:58,695::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31168::INFO::2012-12-02 15:01:58,695::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00781798362732', 'lastCheck': 1354453310.2593091, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0116329193115', 'lastCheck': 1354453314.1929269, 'code': 0, 'valid': True}} >Thread-31168::DEBUG::2012-12-02 15:01:58,695::task::1151::TaskManager.Task::(prepare) Task=`eb97fd5a-972f-48bb-926e-2eb23b58ba3d`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00781798362732', 'lastCheck': 1354453310.2593091, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.0116329193115', 'lastCheck': 1354453314.1929269, 'code': 0, 'valid': True}} >Thread-31168::DEBUG::2012-12-02 15:01:58,695::task::568::TaskManager.Task::(_updateState) Task=`eb97fd5a-972f-48bb-926e-2eb23b58ba3d`::moving from state preparing -> state finished >Thread-31168::DEBUG::2012-12-02 15:01:58,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31168::DEBUG::2012-12-02 15:01:58,695::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31168::DEBUG::2012-12-02 15:01:58,695::task::957::TaskManager.Task::(_decref) Task=`eb97fd5a-972f-48bb-926e-2eb23b58ba3d`::ref 0 aborting False >Thread-31174::DEBUG::2012-12-02 15:02:08,809::task::568::TaskManager.Task::(_updateState) Task=`03ed4fe8-4da5-4a05-ae60-3b9795e68cc3`::moving from state init -> state preparing >Thread-31174::INFO::2012-12-02 15:02:08,809::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31174::INFO::2012-12-02 15:02:08,809::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00979208946228', 'lastCheck': 1354453320.2806201, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00784707069397', 'lastCheck': 1354453324.2096, 'code': 0, 'valid': True}} >Thread-31174::DEBUG::2012-12-02 15:02:08,809::task::1151::TaskManager.Task::(prepare) Task=`03ed4fe8-4da5-4a05-ae60-3b9795e68cc3`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00979208946228', 'lastCheck': 1354453320.2806201, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00784707069397', 'lastCheck': 1354453324.2096, 'code': 0, 'valid': True}} >Thread-31174::DEBUG::2012-12-02 15:02:08,809::task::568::TaskManager.Task::(_updateState) Task=`03ed4fe8-4da5-4a05-ae60-3b9795e68cc3`::moving from state preparing -> state finished >Thread-31174::DEBUG::2012-12-02 15:02:08,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31174::DEBUG::2012-12-02 15:02:08,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31174::DEBUG::2012-12-02 15:02:08,810::task::957::TaskManager.Task::(_decref) Task=`03ed4fe8-4da5-4a05-ae60-3b9795e68cc3`::ref 0 aborting False >Thread-31180::DEBUG::2012-12-02 15:02:18,927::task::568::TaskManager.Task::(_updateState) Task=`8c57e835-169e-43f3-bfb6-82f3d8615825`::moving from state init -> state preparing >Thread-31180::INFO::2012-12-02 15:02:18,927::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31180::INFO::2012-12-02 15:02:18,928::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00885510444641', 'lastCheck': 1354453330.3006201, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00979018211365', 'lastCheck': 1354453334.229526, 'code': 0, 'valid': True}} >Thread-31180::DEBUG::2012-12-02 15:02:18,928::task::1151::TaskManager.Task::(prepare) Task=`8c57e835-169e-43f3-bfb6-82f3d8615825`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00885510444641', 'lastCheck': 1354453330.3006201, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0.00979018211365', 'lastCheck': 1354453334.229526, 'code': 0, 'valid': True}} >Thread-31180::DEBUG::2012-12-02 15:02:18,928::task::568::TaskManager.Task::(_updateState) Task=`8c57e835-169e-43f3-bfb6-82f3d8615825`::moving from state preparing -> state finished >Thread-31180::DEBUG::2012-12-02 15:02:18,928::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31180::DEBUG::2012-12-02 15:02:18,928::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31180::DEBUG::2012-12-02 15:02:18,928::task::957::TaskManager.Task::(_decref) Task=`8c57e835-169e-43f3-bfb6-82f3d8615825`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:02:24,239::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-30654::DEBUG::2012-12-02 15:02:24,242::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` changed its status to Invalid >Thread-31186::DEBUG::2012-12-02 15:02:29,052::task::568::TaskManager.Task::(_updateState) Task=`691a5c84-39e6-4211-aeb0-5caea3a791d6`::moving from state init -> state preparing >Thread-31186::INFO::2012-12-02 15:02:29,053::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31186::INFO::2012-12-02 15:02:29,053::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00875306129456', 'lastCheck': 1354453340.3203559, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453344.242733, 'code': 200, 'valid': False}} >Thread-31186::DEBUG::2012-12-02 15:02:29,053::task::1151::TaskManager.Task::(prepare) Task=`691a5c84-39e6-4211-aeb0-5caea3a791d6`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00875306129456', 'lastCheck': 1354453340.3203559, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453344.242733, 'code': 200, 'valid': False}} >Thread-31186::DEBUG::2012-12-02 15:02:29,053::task::568::TaskManager.Task::(_updateState) Task=`691a5c84-39e6-4211-aeb0-5caea3a791d6`::moving from state preparing -> state finished >Thread-31186::DEBUG::2012-12-02 15:02:29,053::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31186::DEBUG::2012-12-02 15:02:29,053::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31186::DEBUG::2012-12-02 15:02:29,054::task::957::TaskManager.Task::(_decref) Task=`691a5c84-39e6-4211-aeb0-5caea3a791d6`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:02:34,249::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31192::DEBUG::2012-12-02 15:02:39,169::task::568::TaskManager.Task::(_updateState) Task=`c571e383-be5c-4f53-9a24-e271118f56dd`::moving from state init -> state preparing >Thread-31192::INFO::2012-12-02 15:02:39,169::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31192::INFO::2012-12-02 15:02:39,169::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00831604003906', 'lastCheck': 1354453350.337755, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453354.249856, 'code': 200, 'valid': False}} >Thread-31192::DEBUG::2012-12-02 15:02:39,169::task::1151::TaskManager.Task::(prepare) Task=`c571e383-be5c-4f53-9a24-e271118f56dd`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00831604003906', 'lastCheck': 1354453350.337755, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453354.249856, 'code': 200, 'valid': False}} >Thread-31192::DEBUG::2012-12-02 15:02:39,169::task::568::TaskManager.Task::(_updateState) Task=`c571e383-be5c-4f53-9a24-e271118f56dd`::moving from state preparing -> state finished >Thread-31192::DEBUG::2012-12-02 15:02:39,170::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31192::DEBUG::2012-12-02 15:02:39,170::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31192::DEBUG::2012-12-02 15:02:39,170::task::957::TaskManager.Task::(_decref) Task=`c571e383-be5c-4f53-9a24-e271118f56dd`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:02:44,256::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31198::DEBUG::2012-12-02 15:02:49,284::task::568::TaskManager.Task::(_updateState) Task=`13e4304a-af0b-4f95-8974-e61b789d62da`::moving from state init -> state preparing >Thread-31198::INFO::2012-12-02 15:02:49,285::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31198::INFO::2012-12-02 15:02:49,285::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00740003585815', 'lastCheck': 1354453360.3545179, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453364.256264, 'code': 200, 'valid': False}} >Thread-31198::DEBUG::2012-12-02 15:02:49,285::task::1151::TaskManager.Task::(prepare) Task=`13e4304a-af0b-4f95-8974-e61b789d62da`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00740003585815', 'lastCheck': 1354453360.3545179, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453364.256264, 'code': 200, 'valid': False}} >Thread-31198::DEBUG::2012-12-02 15:02:49,285::task::568::TaskManager.Task::(_updateState) Task=`13e4304a-af0b-4f95-8974-e61b789d62da`::moving from state preparing -> state finished >Thread-31198::DEBUG::2012-12-02 15:02:49,285::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31198::DEBUG::2012-12-02 15:02:49,285::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31198::DEBUG::2012-12-02 15:02:49,286::task::957::TaskManager.Task::(_decref) Task=`13e4304a-af0b-4f95-8974-e61b789d62da`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:02:54,262::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31204::DEBUG::2012-12-02 15:02:59,424::task::568::TaskManager.Task::(_updateState) Task=`18481b5e-1627-4d82-8a65-d7f6990f5854`::moving from state init -> state preparing >Thread-31204::INFO::2012-12-02 15:02:59,424::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31204::INFO::2012-12-02 15:02:59,424::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00760793685913', 'lastCheck': 1354453370.371182, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453374.262816, 'code': 200, 'valid': False}} >Thread-31204::DEBUG::2012-12-02 15:02:59,424::task::1151::TaskManager.Task::(prepare) Task=`18481b5e-1627-4d82-8a65-d7f6990f5854`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00760793685913', 'lastCheck': 1354453370.371182, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453374.262816, 'code': 200, 'valid': False}} >Thread-31204::DEBUG::2012-12-02 15:02:59,425::task::568::TaskManager.Task::(_updateState) Task=`18481b5e-1627-4d82-8a65-d7f6990f5854`::moving from state preparing -> state finished >Thread-31204::DEBUG::2012-12-02 15:02:59,425::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31204::DEBUG::2012-12-02 15:02:59,425::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31204::DEBUG::2012-12-02 15:02:59,425::task::957::TaskManager.Task::(_decref) Task=`18481b5e-1627-4d82-8a65-d7f6990f5854`::ref 0 aborting False >Thread-30910::DEBUG::2012-12-02 15:03:00,371::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-30910::DEBUG::2012-12-02 15:03:00,371::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30910::DEBUG::2012-12-02 15:03:00,372::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-30910::DEBUG::2012-12-02 15:03:00,594::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-30910::WARNING::2012-12-02 15:03:00,596::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-30910::DEBUG::2012-12-02 15:03:00,596::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30910::DEBUG::2012-12-02 15:03:00,632::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-30910::DEBUG::2012-12-02 15:03:00,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-30910::DEBUG::2012-12-02 15:03:00,638::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Regular', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7867c42a2ec25b1e27a5f200d36e7105ccf17687'] >Thread-30910::DEBUG::2012-12-02 15:03:00,641::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-30910::WARNING::2012-12-02 15:03:00,641::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_imageNS already registered >Thread-30910::WARNING::2012-12-02 15:03:00,641::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_volumeNS already registered >Thread-30654::ERROR::2012-12-02 15:03:04,267::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31269::DEBUG::2012-12-02 15:03:09,526::task::568::TaskManager.Task::(_updateState) Task=`d6ffbff5-54ba-487c-8302-545552783878`::moving from state init -> state preparing >Thread-31269::INFO::2012-12-02 15:03:09,526::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31269::INFO::2012-12-02 15:03:09,527::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00643682479858', 'lastCheck': 1354453380.6566069, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453384.268218, 'code': 200, 'valid': False}} >Thread-31269::DEBUG::2012-12-02 15:03:09,527::task::1151::TaskManager.Task::(prepare) Task=`d6ffbff5-54ba-487c-8302-545552783878`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00643682479858', 'lastCheck': 1354453380.6566069, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453384.268218, 'code': 200, 'valid': False}} >Thread-31269::DEBUG::2012-12-02 15:03:09,527::task::568::TaskManager.Task::(_updateState) Task=`d6ffbff5-54ba-487c-8302-545552783878`::moving from state preparing -> state finished >Thread-31269::DEBUG::2012-12-02 15:03:09,527::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31269::DEBUG::2012-12-02 15:03:09,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31269::DEBUG::2012-12-02 15:03:09,527::task::957::TaskManager.Task::(_decref) Task=`d6ffbff5-54ba-487c-8302-545552783878`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:03:14,274::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31274::DEBUG::2012-12-02 15:03:16,179::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31274::DEBUG::2012-12-02 15:03:16,179::task::568::TaskManager.Task::(_updateState) Task=`781360c2-00ee-4794-a462-7b04c3590b6e`::moving from state init -> state preparing >Thread-31274::INFO::2012-12-02 15:03:16,179::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31274::INFO::2012-12-02 15:03:16,180::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31274::DEBUG::2012-12-02 15:03:16,180::task::1151::TaskManager.Task::(prepare) Task=`781360c2-00ee-4794-a462-7b04c3590b6e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31274::DEBUG::2012-12-02 15:03:16,180::task::568::TaskManager.Task::(_updateState) Task=`781360c2-00ee-4794-a462-7b04c3590b6e`::moving from state preparing -> state finished >Thread-31274::DEBUG::2012-12-02 15:03:16,180::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31274::DEBUG::2012-12-02 15:03:16,180::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31274::DEBUG::2012-12-02 15:03:16,180::task::957::TaskManager.Task::(_decref) Task=`781360c2-00ee-4794-a462-7b04c3590b6e`::ref 0 aborting False >Thread-31276::DEBUG::2012-12-02 15:03:19,627::task::568::TaskManager.Task::(_updateState) Task=`7dc9f975-92ed-4660-b42b-29f73d934ed8`::moving from state init -> state preparing >Thread-31276::INFO::2012-12-02 15:03:19,627::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31276::INFO::2012-12-02 15:03:19,627::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00862002372742', 'lastCheck': 1354453390.6747761, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453394.274579, 'code': 200, 'valid': False}} >Thread-31276::DEBUG::2012-12-02 15:03:19,627::task::1151::TaskManager.Task::(prepare) Task=`7dc9f975-92ed-4660-b42b-29f73d934ed8`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00862002372742', 'lastCheck': 1354453390.6747761, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453394.274579, 'code': 200, 'valid': False}} >Thread-31276::DEBUG::2012-12-02 15:03:19,627::task::568::TaskManager.Task::(_updateState) Task=`7dc9f975-92ed-4660-b42b-29f73d934ed8`::moving from state preparing -> state finished >Thread-31276::DEBUG::2012-12-02 15:03:19,627::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31276::DEBUG::2012-12-02 15:03:19,628::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31276::DEBUG::2012-12-02 15:03:19,628::task::957::TaskManager.Task::(_decref) Task=`7dc9f975-92ed-4660-b42b-29f73d934ed8`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:03:24,280::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31281::DEBUG::2012-12-02 15:03:27,229::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31281::DEBUG::2012-12-02 15:03:27,230::task::568::TaskManager.Task::(_updateState) Task=`515396da-aa72-4692-aaf6-5797a4819b57`::moving from state init -> state preparing >Thread-31281::INFO::2012-12-02 15:03:27,230::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31281::INFO::2012-12-02 15:03:27,230::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31281::DEBUG::2012-12-02 15:03:27,230::task::1151::TaskManager.Task::(prepare) Task=`515396da-aa72-4692-aaf6-5797a4819b57`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31281::DEBUG::2012-12-02 15:03:27,231::task::568::TaskManager.Task::(_updateState) Task=`515396da-aa72-4692-aaf6-5797a4819b57`::moving from state preparing -> state finished >Thread-31281::DEBUG::2012-12-02 15:03:27,231::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31281::DEBUG::2012-12-02 15:03:27,231::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31281::DEBUG::2012-12-02 15:03:27,231::task::957::TaskManager.Task::(_decref) Task=`515396da-aa72-4692-aaf6-5797a4819b57`::ref 0 aborting False >Thread-31283::DEBUG::2012-12-02 15:03:29,773::task::568::TaskManager.Task::(_updateState) Task=`ec459d46-093e-472f-b2e7-0228475d1088`::moving from state init -> state preparing >Thread-31283::INFO::2012-12-02 15:03:29,774::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31283::INFO::2012-12-02 15:03:29,774::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00745892524719', 'lastCheck': 1354453400.691211, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453404.28106, 'code': 200, 'valid': False}} >Thread-31283::DEBUG::2012-12-02 15:03:29,774::task::1151::TaskManager.Task::(prepare) Task=`ec459d46-093e-472f-b2e7-0228475d1088`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00745892524719', 'lastCheck': 1354453400.691211, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453404.28106, 'code': 200, 'valid': False}} >Thread-31283::DEBUG::2012-12-02 15:03:29,774::task::568::TaskManager.Task::(_updateState) Task=`ec459d46-093e-472f-b2e7-0228475d1088`::moving from state preparing -> state finished >Thread-31283::DEBUG::2012-12-02 15:03:29,774::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31283::DEBUG::2012-12-02 15:03:29,774::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31283::DEBUG::2012-12-02 15:03:29,774::task::957::TaskManager.Task::(_decref) Task=`ec459d46-093e-472f-b2e7-0228475d1088`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:03:34,288::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31288::DEBUG::2012-12-02 15:03:37,303::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31288::DEBUG::2012-12-02 15:03:37,303::task::568::TaskManager.Task::(_updateState) Task=`32aaa3aa-251f-40cb-b50a-99206292a796`::moving from state init -> state preparing >Thread-31288::INFO::2012-12-02 15:03:37,303::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31288::INFO::2012-12-02 15:03:37,304::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31288::DEBUG::2012-12-02 15:03:37,304::task::1151::TaskManager.Task::(prepare) Task=`32aaa3aa-251f-40cb-b50a-99206292a796`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31288::DEBUG::2012-12-02 15:03:37,304::task::568::TaskManager.Task::(_updateState) Task=`32aaa3aa-251f-40cb-b50a-99206292a796`::moving from state preparing -> state finished >Thread-31288::DEBUG::2012-12-02 15:03:37,304::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31288::DEBUG::2012-12-02 15:03:37,304::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31288::DEBUG::2012-12-02 15:03:37,304::task::957::TaskManager.Task::(_decref) Task=`32aaa3aa-251f-40cb-b50a-99206292a796`::ref 0 aborting False >Thread-31290::DEBUG::2012-12-02 15:03:39,967::task::568::TaskManager.Task::(_updateState) Task=`978756fa-4394-48de-9391-abd50e7b0537`::moving from state init -> state preparing >Thread-31290::INFO::2012-12-02 15:03:39,967::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31290::INFO::2012-12-02 15:03:39,967::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00953912734985', 'lastCheck': 1354453410.7094181, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453414.2886131, 'code': 200, 'valid': False}} >Thread-31290::DEBUG::2012-12-02 15:03:39,967::task::1151::TaskManager.Task::(prepare) Task=`978756fa-4394-48de-9391-abd50e7b0537`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00953912734985', 'lastCheck': 1354453410.7094181, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453414.2886131, 'code': 200, 'valid': False}} >Thread-31290::DEBUG::2012-12-02 15:03:39,967::task::568::TaskManager.Task::(_updateState) Task=`978756fa-4394-48de-9391-abd50e7b0537`::moving from state preparing -> state finished >Thread-31290::DEBUG::2012-12-02 15:03:39,967::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31290::DEBUG::2012-12-02 15:03:39,968::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31290::DEBUG::2012-12-02 15:03:39,968::task::957::TaskManager.Task::(_decref) Task=`978756fa-4394-48de-9391-abd50e7b0537`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:03:44,294::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31295::DEBUG::2012-12-02 15:03:47,367::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31295::DEBUG::2012-12-02 15:03:47,367::task::568::TaskManager.Task::(_updateState) Task=`5739b78a-772d-4f01-aa52-bf5880132341`::moving from state init -> state preparing >Thread-31295::INFO::2012-12-02 15:03:47,367::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31295::INFO::2012-12-02 15:03:47,367::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31295::DEBUG::2012-12-02 15:03:47,368::task::1151::TaskManager.Task::(prepare) Task=`5739b78a-772d-4f01-aa52-bf5880132341`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31295::DEBUG::2012-12-02 15:03:47,368::task::568::TaskManager.Task::(_updateState) Task=`5739b78a-772d-4f01-aa52-bf5880132341`::moving from state preparing -> state finished >Thread-31295::DEBUG::2012-12-02 15:03:47,368::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31295::DEBUG::2012-12-02 15:03:47,368::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31295::DEBUG::2012-12-02 15:03:47,368::task::957::TaskManager.Task::(_decref) Task=`5739b78a-772d-4f01-aa52-bf5880132341`::ref 0 aborting False >Thread-31297::DEBUG::2012-12-02 15:03:50,102::task::568::TaskManager.Task::(_updateState) Task=`c4371a03-0896-45d8-81c9-bd8891668a75`::moving from state init -> state preparing >Thread-31297::INFO::2012-12-02 15:03:50,102::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31297::INFO::2012-12-02 15:03:50,102::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00787091255188', 'lastCheck': 1354453420.727073, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453424.2950161, 'code': 200, 'valid': False}} >Thread-31297::DEBUG::2012-12-02 15:03:50,102::task::1151::TaskManager.Task::(prepare) Task=`c4371a03-0896-45d8-81c9-bd8891668a75`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00787091255188', 'lastCheck': 1354453420.727073, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453424.2950161, 'code': 200, 'valid': False}} >Thread-31297::DEBUG::2012-12-02 15:03:50,102::task::568::TaskManager.Task::(_updateState) Task=`c4371a03-0896-45d8-81c9-bd8891668a75`::moving from state preparing -> state finished >Thread-31297::DEBUG::2012-12-02 15:03:50,103::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31297::DEBUG::2012-12-02 15:03:50,103::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31297::DEBUG::2012-12-02 15:03:50,103::task::957::TaskManager.Task::(_decref) Task=`c4371a03-0896-45d8-81c9-bd8891668a75`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:03:54,301::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31302::DEBUG::2012-12-02 15:03:57,415::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31302::DEBUG::2012-12-02 15:03:57,416::task::568::TaskManager.Task::(_updateState) Task=`7382de08-65a9-49ef-b3ca-955f11a8f356`::moving from state init -> state preparing >Thread-31302::INFO::2012-12-02 15:03:57,442::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31302::INFO::2012-12-02 15:03:57,443::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31302::DEBUG::2012-12-02 15:03:57,443::task::1151::TaskManager.Task::(prepare) Task=`7382de08-65a9-49ef-b3ca-955f11a8f356`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31302::DEBUG::2012-12-02 15:03:57,443::task::568::TaskManager.Task::(_updateState) Task=`7382de08-65a9-49ef-b3ca-955f11a8f356`::moving from state preparing -> state finished >Thread-31302::DEBUG::2012-12-02 15:03:57,443::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31302::DEBUG::2012-12-02 15:03:57,443::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31302::DEBUG::2012-12-02 15:03:57,443::task::957::TaskManager.Task::(_decref) Task=`7382de08-65a9-49ef-b3ca-955f11a8f356`::ref 0 aborting False >Thread-31304::DEBUG::2012-12-02 15:04:00,231::task::568::TaskManager.Task::(_updateState) Task=`d844f306-0bd8-4d70-8a87-b01f06949666`::moving from state init -> state preparing >Thread-31304::INFO::2012-12-02 15:04:00,231::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31304::INFO::2012-12-02 15:04:00,231::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00863099098206', 'lastCheck': 1354453430.7450521, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453434.301569, 'code': 200, 'valid': False}} >Thread-31304::DEBUG::2012-12-02 15:04:00,231::task::1151::TaskManager.Task::(prepare) Task=`d844f306-0bd8-4d70-8a87-b01f06949666`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00863099098206', 'lastCheck': 1354453430.7450521, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453434.301569, 'code': 200, 'valid': False}} >Thread-31304::DEBUG::2012-12-02 15:04:00,232::task::568::TaskManager.Task::(_updateState) Task=`d844f306-0bd8-4d70-8a87-b01f06949666`::moving from state preparing -> state finished >Thread-31304::DEBUG::2012-12-02 15:04:00,232::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31304::DEBUG::2012-12-02 15:04:00,232::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31304::DEBUG::2012-12-02 15:04:00,232::task::957::TaskManager.Task::(_decref) Task=`d844f306-0bd8-4d70-8a87-b01f06949666`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:04:04,307::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31309::DEBUG::2012-12-02 15:04:07,500::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31309::DEBUG::2012-12-02 15:04:07,500::task::568::TaskManager.Task::(_updateState) Task=`e8b03ac7-d206-41c0-9e6e-d501cd278464`::moving from state init -> state preparing >Thread-31309::INFO::2012-12-02 15:04:07,500::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31309::INFO::2012-12-02 15:04:07,500::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31309::DEBUG::2012-12-02 15:04:07,500::task::1151::TaskManager.Task::(prepare) Task=`e8b03ac7-d206-41c0-9e6e-d501cd278464`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31309::DEBUG::2012-12-02 15:04:07,501::task::568::TaskManager.Task::(_updateState) Task=`e8b03ac7-d206-41c0-9e6e-d501cd278464`::moving from state preparing -> state finished >Thread-31309::DEBUG::2012-12-02 15:04:07,501::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31309::DEBUG::2012-12-02 15:04:07,501::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31309::DEBUG::2012-12-02 15:04:07,501::task::957::TaskManager.Task::(_decref) Task=`e8b03ac7-d206-41c0-9e6e-d501cd278464`::ref 0 aborting False >Thread-31311::DEBUG::2012-12-02 15:04:10,346::task::568::TaskManager.Task::(_updateState) Task=`cbf99d96-e2ac-40aa-bc6c-f75c463a1e17`::moving from state init -> state preparing >Thread-31311::INFO::2012-12-02 15:04:10,346::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31311::INFO::2012-12-02 15:04:10,346::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00852489471436', 'lastCheck': 1354453440.7646279, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453444.307312, 'code': 200, 'valid': False}} >Thread-31311::DEBUG::2012-12-02 15:04:10,346::task::1151::TaskManager.Task::(prepare) Task=`cbf99d96-e2ac-40aa-bc6c-f75c463a1e17`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00852489471436', 'lastCheck': 1354453440.7646279, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453444.307312, 'code': 200, 'valid': False}} >Thread-31311::DEBUG::2012-12-02 15:04:10,347::task::568::TaskManager.Task::(_updateState) Task=`cbf99d96-e2ac-40aa-bc6c-f75c463a1e17`::moving from state preparing -> state finished >Thread-31311::DEBUG::2012-12-02 15:04:10,347::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31311::DEBUG::2012-12-02 15:04:10,347::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31311::DEBUG::2012-12-02 15:04:10,347::task::957::TaskManager.Task::(_decref) Task=`cbf99d96-e2ac-40aa-bc6c-f75c463a1e17`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:04:14,313::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31316::DEBUG::2012-12-02 15:04:17,539::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31316::DEBUG::2012-12-02 15:04:17,540::task::568::TaskManager.Task::(_updateState) Task=`b1fddee3-74d2-499b-8ce1-19b543d7294d`::moving from state init -> state preparing >Thread-31316::INFO::2012-12-02 15:04:17,540::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31316::INFO::2012-12-02 15:04:17,540::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31316::DEBUG::2012-12-02 15:04:17,540::task::1151::TaskManager.Task::(prepare) Task=`b1fddee3-74d2-499b-8ce1-19b543d7294d`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31316::DEBUG::2012-12-02 15:04:17,540::task::568::TaskManager.Task::(_updateState) Task=`b1fddee3-74d2-499b-8ce1-19b543d7294d`::moving from state preparing -> state finished >Thread-31316::DEBUG::2012-12-02 15:04:17,540::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31316::DEBUG::2012-12-02 15:04:17,540::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31316::DEBUG::2012-12-02 15:04:17,541::task::957::TaskManager.Task::(_decref) Task=`b1fddee3-74d2-499b-8ce1-19b543d7294d`::ref 0 aborting False >Thread-31318::DEBUG::2012-12-02 15:04:20,474::task::568::TaskManager.Task::(_updateState) Task=`32c9b8bf-fe58-4cd5-82a4-ff8ab6167e7b`::moving from state init -> state preparing >Thread-31318::INFO::2012-12-02 15:04:20,474::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31318::INFO::2012-12-02 15:04:20,474::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0086932182312', 'lastCheck': 1354453450.782645, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453454.3136871, 'code': 200, 'valid': False}} >Thread-31318::DEBUG::2012-12-02 15:04:20,474::task::1151::TaskManager.Task::(prepare) Task=`32c9b8bf-fe58-4cd5-82a4-ff8ab6167e7b`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0086932182312', 'lastCheck': 1354453450.782645, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453454.3136871, 'code': 200, 'valid': False}} >Thread-31318::DEBUG::2012-12-02 15:04:20,474::task::568::TaskManager.Task::(_updateState) Task=`32c9b8bf-fe58-4cd5-82a4-ff8ab6167e7b`::moving from state preparing -> state finished >Thread-31318::DEBUG::2012-12-02 15:04:20,475::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31318::DEBUG::2012-12-02 15:04:20,475::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31318::DEBUG::2012-12-02 15:04:20,475::task::957::TaskManager.Task::(_decref) Task=`32c9b8bf-fe58-4cd5-82a4-ff8ab6167e7b`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:04:24,318::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31323::DEBUG::2012-12-02 15:04:27,590::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31323::DEBUG::2012-12-02 15:04:27,590::task::568::TaskManager.Task::(_updateState) Task=`36023046-090c-490e-a986-88b7d52d29cf`::moving from state init -> state preparing >Thread-31323::INFO::2012-12-02 15:04:27,590::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31323::INFO::2012-12-02 15:04:27,590::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31323::DEBUG::2012-12-02 15:04:27,590::task::1151::TaskManager.Task::(prepare) Task=`36023046-090c-490e-a986-88b7d52d29cf`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31323::DEBUG::2012-12-02 15:04:27,591::task::568::TaskManager.Task::(_updateState) Task=`36023046-090c-490e-a986-88b7d52d29cf`::moving from state preparing -> state finished >Thread-31323::DEBUG::2012-12-02 15:04:27,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31323::DEBUG::2012-12-02 15:04:27,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31323::DEBUG::2012-12-02 15:04:27,591::task::957::TaskManager.Task::(_decref) Task=`36023046-090c-490e-a986-88b7d52d29cf`::ref 0 aborting False >Thread-31325::DEBUG::2012-12-02 15:04:30,645::task::568::TaskManager.Task::(_updateState) Task=`e64a6322-5271-467a-aa24-7d9d3fdb9012`::moving from state init -> state preparing >Thread-31325::INFO::2012-12-02 15:04:30,645::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31325::INFO::2012-12-02 15:04:30,645::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00746393203735', 'lastCheck': 1354453460.7984729, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453464.3185489, 'code': 200, 'valid': False}} >Thread-31325::DEBUG::2012-12-02 15:04:30,645::task::1151::TaskManager.Task::(prepare) Task=`e64a6322-5271-467a-aa24-7d9d3fdb9012`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00746393203735', 'lastCheck': 1354453460.7984729, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453464.3185489, 'code': 200, 'valid': False}} >Thread-31325::DEBUG::2012-12-02 15:04:30,645::task::568::TaskManager.Task::(_updateState) Task=`e64a6322-5271-467a-aa24-7d9d3fdb9012`::moving from state preparing -> state finished >Thread-31325::DEBUG::2012-12-02 15:04:30,645::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31325::DEBUG::2012-12-02 15:04:30,645::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31325::DEBUG::2012-12-02 15:04:30,646::task::957::TaskManager.Task::(_decref) Task=`e64a6322-5271-467a-aa24-7d9d3fdb9012`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:04:34,326::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31330::DEBUG::2012-12-02 15:04:37,625::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31330::DEBUG::2012-12-02 15:04:37,626::task::568::TaskManager.Task::(_updateState) Task=`cd86de05-2de0-44a0-b39e-fc4c95c0cba4`::moving from state init -> state preparing >Thread-31330::INFO::2012-12-02 15:04:37,626::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31330::INFO::2012-12-02 15:04:37,626::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31330::DEBUG::2012-12-02 15:04:37,626::task::1151::TaskManager.Task::(prepare) Task=`cd86de05-2de0-44a0-b39e-fc4c95c0cba4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31330::DEBUG::2012-12-02 15:04:37,626::task::568::TaskManager.Task::(_updateState) Task=`cd86de05-2de0-44a0-b39e-fc4c95c0cba4`::moving from state preparing -> state finished >Thread-31330::DEBUG::2012-12-02 15:04:37,627::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31330::DEBUG::2012-12-02 15:04:37,627::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31330::DEBUG::2012-12-02 15:04:37,627::task::957::TaskManager.Task::(_decref) Task=`cd86de05-2de0-44a0-b39e-fc4c95c0cba4`::ref 0 aborting False >Thread-31332::DEBUG::2012-12-02 15:04:40,747::task::568::TaskManager.Task::(_updateState) Task=`b5dc0332-f3a2-446f-ba0d-ccec8908ae44`::moving from state init -> state preparing >Thread-31332::INFO::2012-12-02 15:04:40,748::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31332::INFO::2012-12-02 15:04:40,748::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101149082184', 'lastCheck': 1354453470.817714, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453474.326642, 'code': 200, 'valid': False}} >Thread-31332::DEBUG::2012-12-02 15:04:40,748::task::1151::TaskManager.Task::(prepare) Task=`b5dc0332-f3a2-446f-ba0d-ccec8908ae44`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101149082184', 'lastCheck': 1354453470.817714, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453474.326642, 'code': 200, 'valid': False}} >Thread-31332::DEBUG::2012-12-02 15:04:40,748::task::568::TaskManager.Task::(_updateState) Task=`b5dc0332-f3a2-446f-ba0d-ccec8908ae44`::moving from state preparing -> state finished >Thread-31332::DEBUG::2012-12-02 15:04:40,748::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31332::DEBUG::2012-12-02 15:04:40,748::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31332::DEBUG::2012-12-02 15:04:40,749::task::957::TaskManager.Task::(_decref) Task=`b5dc0332-f3a2-446f-ba0d-ccec8908ae44`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:04:44,332::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31337::DEBUG::2012-12-02 15:04:47,685::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31337::DEBUG::2012-12-02 15:04:47,685::task::568::TaskManager.Task::(_updateState) Task=`68bcd4a6-db78-4e96-9f04-98a00bfe2877`::moving from state init -> state preparing >Thread-31337::INFO::2012-12-02 15:04:47,685::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31337::INFO::2012-12-02 15:04:47,685::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31337::DEBUG::2012-12-02 15:04:47,685::task::1151::TaskManager.Task::(prepare) Task=`68bcd4a6-db78-4e96-9f04-98a00bfe2877`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31337::DEBUG::2012-12-02 15:04:47,686::task::568::TaskManager.Task::(_updateState) Task=`68bcd4a6-db78-4e96-9f04-98a00bfe2877`::moving from state preparing -> state finished >Thread-31337::DEBUG::2012-12-02 15:04:47,686::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31337::DEBUG::2012-12-02 15:04:47,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31337::DEBUG::2012-12-02 15:04:47,686::task::957::TaskManager.Task::(_decref) Task=`68bcd4a6-db78-4e96-9f04-98a00bfe2877`::ref 0 aborting False >Thread-31339::DEBUG::2012-12-02 15:04:50,927::task::568::TaskManager.Task::(_updateState) Task=`55624a68-f472-4fe8-be0b-fdaa48bc87ec`::moving from state init -> state preparing >Thread-31339::INFO::2012-12-02 15:04:50,927::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31339::INFO::2012-12-02 15:04:50,927::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00626921653748', 'lastCheck': 1354453490.8517399, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453484.3330481, 'code': 200, 'valid': False}} >Thread-31339::DEBUG::2012-12-02 15:04:50,927::task::1151::TaskManager.Task::(prepare) Task=`55624a68-f472-4fe8-be0b-fdaa48bc87ec`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00626921653748', 'lastCheck': 1354453490.8517399, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453484.3330481, 'code': 200, 'valid': False}} >Thread-31339::DEBUG::2012-12-02 15:04:50,927::task::568::TaskManager.Task::(_updateState) Task=`55624a68-f472-4fe8-be0b-fdaa48bc87ec`::moving from state preparing -> state finished >Thread-31339::DEBUG::2012-12-02 15:04:50,927::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31339::DEBUG::2012-12-02 15:04:50,928::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31339::DEBUG::2012-12-02 15:04:50,928::task::957::TaskManager.Task::(_decref) Task=`55624a68-f472-4fe8-be0b-fdaa48bc87ec`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:04:54,338::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31344::DEBUG::2012-12-02 15:04:57,744::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31344::DEBUG::2012-12-02 15:04:57,744::task::568::TaskManager.Task::(_updateState) Task=`743bde56-36c6-45d7-8360-4fb3f2d01313`::moving from state init -> state preparing >Thread-31344::INFO::2012-12-02 15:04:57,744::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31344::INFO::2012-12-02 15:04:57,744::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31344::DEBUG::2012-12-02 15:04:57,744::task::1151::TaskManager.Task::(prepare) Task=`743bde56-36c6-45d7-8360-4fb3f2d01313`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31344::DEBUG::2012-12-02 15:04:57,745::task::568::TaskManager.Task::(_updateState) Task=`743bde56-36c6-45d7-8360-4fb3f2d01313`::moving from state preparing -> state finished >Thread-31344::DEBUG::2012-12-02 15:04:57,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31344::DEBUG::2012-12-02 15:04:57,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31344::DEBUG::2012-12-02 15:04:57,745::task::957::TaskManager.Task::(_decref) Task=`743bde56-36c6-45d7-8360-4fb3f2d01313`::ref 0 aborting False >Thread-31346::DEBUG::2012-12-02 15:05:01,105::task::568::TaskManager.Task::(_updateState) Task=`4a98b308-5a64-4e75-afb6-a9011c0f0687`::moving from state init -> state preparing >Thread-31346::INFO::2012-12-02 15:05:01,105::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31346::INFO::2012-12-02 15:05:01,105::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00819206237793', 'lastCheck': 1354453500.8715241, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453494.338362, 'code': 200, 'valid': False}} >Thread-31346::DEBUG::2012-12-02 15:05:01,105::task::1151::TaskManager.Task::(prepare) Task=`4a98b308-5a64-4e75-afb6-a9011c0f0687`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00819206237793', 'lastCheck': 1354453500.8715241, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453494.338362, 'code': 200, 'valid': False}} >Thread-31346::DEBUG::2012-12-02 15:05:01,106::task::568::TaskManager.Task::(_updateState) Task=`4a98b308-5a64-4e75-afb6-a9011c0f0687`::moving from state preparing -> state finished >Thread-31346::DEBUG::2012-12-02 15:05:01,106::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31346::DEBUG::2012-12-02 15:05:01,106::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31346::DEBUG::2012-12-02 15:05:01,106::task::957::TaskManager.Task::(_decref) Task=`4a98b308-5a64-4e75-afb6-a9011c0f0687`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:05:04,344::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31351::DEBUG::2012-12-02 15:05:07,794::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31351::DEBUG::2012-12-02 15:05:07,795::task::568::TaskManager.Task::(_updateState) Task=`af6c1c24-de84-4f2a-a312-562f8f9e14fe`::moving from state init -> state preparing >Thread-31351::INFO::2012-12-02 15:05:07,795::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31351::INFO::2012-12-02 15:05:07,795::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31351::DEBUG::2012-12-02 15:05:07,795::task::1151::TaskManager.Task::(prepare) Task=`af6c1c24-de84-4f2a-a312-562f8f9e14fe`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31351::DEBUG::2012-12-02 15:05:07,795::task::568::TaskManager.Task::(_updateState) Task=`af6c1c24-de84-4f2a-a312-562f8f9e14fe`::moving from state preparing -> state finished >Thread-31351::DEBUG::2012-12-02 15:05:07,795::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31351::DEBUG::2012-12-02 15:05:07,795::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31351::DEBUG::2012-12-02 15:05:07,796::task::957::TaskManager.Task::(_decref) Task=`af6c1c24-de84-4f2a-a312-562f8f9e14fe`::ref 0 aborting False >Thread-31353::DEBUG::2012-12-02 15:05:11,230::task::568::TaskManager.Task::(_updateState) Task=`7cb703e9-7692-4b8f-a7af-0c9b5e5ab03d`::moving from state init -> state preparing >Thread-31353::INFO::2012-12-02 15:05:11,230::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31353::INFO::2012-12-02 15:05:11,230::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00869011878967', 'lastCheck': 1354453510.8897719, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453504.344754, 'code': 200, 'valid': False}} >Thread-31353::DEBUG::2012-12-02 15:05:11,230::task::1151::TaskManager.Task::(prepare) Task=`7cb703e9-7692-4b8f-a7af-0c9b5e5ab03d`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00869011878967', 'lastCheck': 1354453510.8897719, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453504.344754, 'code': 200, 'valid': False}} >Thread-31353::DEBUG::2012-12-02 15:05:11,230::task::568::TaskManager.Task::(_updateState) Task=`7cb703e9-7692-4b8f-a7af-0c9b5e5ab03d`::moving from state preparing -> state finished >Thread-31353::DEBUG::2012-12-02 15:05:11,231::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31353::DEBUG::2012-12-02 15:05:11,231::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31353::DEBUG::2012-12-02 15:05:11,231::task::957::TaskManager.Task::(_decref) Task=`7cb703e9-7692-4b8f-a7af-0c9b5e5ab03d`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:05:14,350::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31358::DEBUG::2012-12-02 15:05:17,831::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31358::DEBUG::2012-12-02 15:05:17,831::task::568::TaskManager.Task::(_updateState) Task=`538c4bba-302b-4d5a-baa5-ee11f6501487`::moving from state init -> state preparing >Thread-31358::INFO::2012-12-02 15:05:17,832::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31358::INFO::2012-12-02 15:05:17,832::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31358::DEBUG::2012-12-02 15:05:17,832::task::1151::TaskManager.Task::(prepare) Task=`538c4bba-302b-4d5a-baa5-ee11f6501487`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31358::DEBUG::2012-12-02 15:05:17,832::task::568::TaskManager.Task::(_updateState) Task=`538c4bba-302b-4d5a-baa5-ee11f6501487`::moving from state preparing -> state finished >Thread-31358::DEBUG::2012-12-02 15:05:17,832::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31358::DEBUG::2012-12-02 15:05:17,832::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31358::DEBUG::2012-12-02 15:05:17,833::task::957::TaskManager.Task::(_decref) Task=`538c4bba-302b-4d5a-baa5-ee11f6501487`::ref 0 aborting False >Thread-31360::DEBUG::2012-12-02 15:05:21,347::task::568::TaskManager.Task::(_updateState) Task=`45c03c59-1a2f-4f6c-af79-0ef6ec021ade`::moving from state init -> state preparing >Thread-31360::INFO::2012-12-02 15:05:21,347::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31360::INFO::2012-12-02 15:05:21,348::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00606608390808', 'lastCheck': 1354453520.9288599, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453514.3512111, 'code': 200, 'valid': False}} >Thread-31360::DEBUG::2012-12-02 15:05:21,348::task::1151::TaskManager.Task::(prepare) Task=`45c03c59-1a2f-4f6c-af79-0ef6ec021ade`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00606608390808', 'lastCheck': 1354453520.9288599, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453514.3512111, 'code': 200, 'valid': False}} >Thread-31360::DEBUG::2012-12-02 15:05:21,348::task::568::TaskManager.Task::(_updateState) Task=`45c03c59-1a2f-4f6c-af79-0ef6ec021ade`::moving from state preparing -> state finished >Thread-31360::DEBUG::2012-12-02 15:05:21,348::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31360::DEBUG::2012-12-02 15:05:21,348::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31360::DEBUG::2012-12-02 15:05:21,348::task::957::TaskManager.Task::(_decref) Task=`45c03c59-1a2f-4f6c-af79-0ef6ec021ade`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:05:24,356::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31365::DEBUG::2012-12-02 15:05:27,881::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31365::DEBUG::2012-12-02 15:05:27,881::task::568::TaskManager.Task::(_updateState) Task=`0bf76a15-00b9-4589-88b9-2fbfdadf1dbd`::moving from state init -> state preparing >Thread-31365::INFO::2012-12-02 15:05:27,882::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31365::INFO::2012-12-02 15:05:27,882::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31365::DEBUG::2012-12-02 15:05:27,882::task::1151::TaskManager.Task::(prepare) Task=`0bf76a15-00b9-4589-88b9-2fbfdadf1dbd`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31365::DEBUG::2012-12-02 15:05:27,882::task::568::TaskManager.Task::(_updateState) Task=`0bf76a15-00b9-4589-88b9-2fbfdadf1dbd`::moving from state preparing -> state finished >Thread-31365::DEBUG::2012-12-02 15:05:27,882::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31365::DEBUG::2012-12-02 15:05:27,882::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31365::DEBUG::2012-12-02 15:05:27,882::task::957::TaskManager.Task::(_decref) Task=`0bf76a15-00b9-4589-88b9-2fbfdadf1dbd`::ref 0 aborting False >Thread-31367::DEBUG::2012-12-02 15:05:31,502::task::568::TaskManager.Task::(_updateState) Task=`f048b797-3661-49aa-b787-737657d6c7ab`::moving from state init -> state preparing >Thread-31367::INFO::2012-12-02 15:05:31,502::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31367::INFO::2012-12-02 15:05:31,502::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00699806213379', 'lastCheck': 1354453530.9448209, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453524.3571119, 'code': 200, 'valid': False}} >Thread-31367::DEBUG::2012-12-02 15:05:31,502::task::1151::TaskManager.Task::(prepare) Task=`f048b797-3661-49aa-b787-737657d6c7ab`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00699806213379', 'lastCheck': 1354453530.9448209, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453524.3571119, 'code': 200, 'valid': False}} >Thread-31367::DEBUG::2012-12-02 15:05:31,502::task::568::TaskManager.Task::(_updateState) Task=`f048b797-3661-49aa-b787-737657d6c7ab`::moving from state preparing -> state finished >Thread-31367::DEBUG::2012-12-02 15:05:31,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31367::DEBUG::2012-12-02 15:05:31,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31367::DEBUG::2012-12-02 15:05:31,503::task::957::TaskManager.Task::(_decref) Task=`f048b797-3661-49aa-b787-737657d6c7ab`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:05:34,364::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31372::DEBUG::2012-12-02 15:05:37,924::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31372::DEBUG::2012-12-02 15:05:37,924::task::568::TaskManager.Task::(_updateState) Task=`0162efdc-08cc-42ba-8a5e-ac646186bbfc`::moving from state init -> state preparing >Thread-31372::INFO::2012-12-02 15:05:37,924::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31372::INFO::2012-12-02 15:05:37,924::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31372::DEBUG::2012-12-02 15:05:37,924::task::1151::TaskManager.Task::(prepare) Task=`0162efdc-08cc-42ba-8a5e-ac646186bbfc`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31372::DEBUG::2012-12-02 15:05:37,925::task::568::TaskManager.Task::(_updateState) Task=`0162efdc-08cc-42ba-8a5e-ac646186bbfc`::moving from state preparing -> state finished >Thread-31372::DEBUG::2012-12-02 15:05:37,925::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31372::DEBUG::2012-12-02 15:05:37,925::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31372::DEBUG::2012-12-02 15:05:37,925::task::957::TaskManager.Task::(_decref) Task=`0162efdc-08cc-42ba-8a5e-ac646186bbfc`::ref 0 aborting False >Thread-31374::DEBUG::2012-12-02 15:05:41,626::task::568::TaskManager.Task::(_updateState) Task=`ea0dcfd0-8751-4b5d-b965-29726fb249ac`::moving from state init -> state preparing >Thread-31374::INFO::2012-12-02 15:05:41,626::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31374::INFO::2012-12-02 15:05:41,626::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00900506973267', 'lastCheck': 1354453540.9644439, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453534.364944, 'code': 200, 'valid': False}} >Thread-31374::DEBUG::2012-12-02 15:05:41,626::task::1151::TaskManager.Task::(prepare) Task=`ea0dcfd0-8751-4b5d-b965-29726fb249ac`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00900506973267', 'lastCheck': 1354453540.9644439, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453534.364944, 'code': 200, 'valid': False}} >Thread-31374::DEBUG::2012-12-02 15:05:41,626::task::568::TaskManager.Task::(_updateState) Task=`ea0dcfd0-8751-4b5d-b965-29726fb249ac`::moving from state preparing -> state finished >Thread-31374::DEBUG::2012-12-02 15:05:41,627::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31374::DEBUG::2012-12-02 15:05:41,627::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31374::DEBUG::2012-12-02 15:05:41,627::task::957::TaskManager.Task::(_decref) Task=`ea0dcfd0-8751-4b5d-b965-29726fb249ac`::ref 0 aborting False >Thread-30654::ERROR::2012-12-02 15:05:44,371::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446/a3177eae-1dc9-445a-92b1-da23ec8008bc' >Thread-31379::DEBUG::2012-12-02 15:05:47,980::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31379::DEBUG::2012-12-02 15:05:47,981::task::568::TaskManager.Task::(_updateState) Task=`838e95a8-09dd-4ef8-9aa5-49a42d31d07a`::moving from state init -> state preparing >Thread-31379::INFO::2012-12-02 15:05:47,981::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31379::INFO::2012-12-02 15:05:47,981::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31379::DEBUG::2012-12-02 15:05:47,981::task::1151::TaskManager.Task::(prepare) Task=`838e95a8-09dd-4ef8-9aa5-49a42d31d07a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-31379::DEBUG::2012-12-02 15:05:47,981::task::568::TaskManager.Task::(_updateState) Task=`838e95a8-09dd-4ef8-9aa5-49a42d31d07a`::moving from state preparing -> state finished >Thread-31379::DEBUG::2012-12-02 15:05:47,982::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31379::DEBUG::2012-12-02 15:05:47,982::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31379::DEBUG::2012-12-02 15:05:47,982::task::957::TaskManager.Task::(_decref) Task=`838e95a8-09dd-4ef8-9aa5-49a42d31d07a`::ref 0 aborting False >Thread-31381::DEBUG::2012-12-02 15:05:51,752::task::568::TaskManager.Task::(_updateState) Task=`535aaf43-cd89-49f6-a17f-8ecd45fda78e`::moving from state init -> state preparing >Thread-31381::INFO::2012-12-02 15:05:51,752::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31381::INFO::2012-12-02 15:05:51,752::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00673389434814', 'lastCheck': 1354453550.9801641, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31381::DEBUG::2012-12-02 15:05:51,753::task::1151::TaskManager.Task::(prepare) Task=`535aaf43-cd89-49f6-a17f-8ecd45fda78e`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00673389434814', 'lastCheck': 1354453550.9801641, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31381::DEBUG::2012-12-02 15:05:51,753::task::568::TaskManager.Task::(_updateState) Task=`535aaf43-cd89-49f6-a17f-8ecd45fda78e`::moving from state preparing -> state finished >Thread-31381::DEBUG::2012-12-02 15:05:51,753::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31381::DEBUG::2012-12-02 15:05:51,753::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31381::DEBUG::2012-12-02 15:05:51,753::task::957::TaskManager.Task::(_decref) Task=`535aaf43-cd89-49f6-a17f-8ecd45fda78e`::ref 0 aborting False >Thread-30654::DEBUG::2012-12-02 15:05:54,371::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a3177eae-1dc9-445a-92b1-da23ec8008bc >Thread-30654::DEBUG::2012-12-02 15:05:54,371::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30654::DEBUG::2012-12-02 15:05:54,372::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-30654::DEBUG::2012-12-02 15:05:54,576::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-30654::WARNING::2012-12-02 15:05:54,577::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-30654::DEBUG::2012-12-02 15:05:54,578::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30654::DEBUG::2012-12-02 15:05:54,611::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-30654::DEBUG::2012-12-02 15:05:54,612::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-30654::DEBUG::2012-12-02 15:05:54,815::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-30654::WARNING::2012-12-02 15:05:54,816::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-30654::DEBUG::2012-12-02 15:05:54,816::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31504::DEBUG::2012-12-02 15:05:58,031::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31504::DEBUG::2012-12-02 15:05:58,032::task::568::TaskManager.Task::(_updateState) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::moving from state init -> state preparing >Thread-31504::INFO::2012-12-02 15:05:58,032::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31504::DEBUG::2012-12-02 15:05:58,032::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31504::DEBUG::2012-12-02 15:05:58,032::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31504::DEBUG::2012-12-02 15:05:58,235::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31504::WARNING::2012-12-02 15:05:58,236::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31504::DEBUG::2012-12-02 15:05:58,237::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31504::ERROR::2012-12-02 15:05:58,266::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31504::ERROR::2012-12-02 15:05:58,267::task::833::TaskManager.Task::(_setError) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31504::DEBUG::2012-12-02 15:05:58,268::task::852::TaskManager.Task::(_run) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::Task._run: 7c163b9b-13f1-48ca-a480-25491b4c4444 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31504::DEBUG::2012-12-02 15:05:58,268::task::1177::TaskManager.Task::(stop) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::stopping in state preparing (force False) >Thread-31504::DEBUG::2012-12-02 15:05:58,268::task::957::TaskManager.Task::(_decref) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::ref 1 aborting True >Thread-31504::INFO::2012-12-02 15:05:58,268::task::1134::TaskManager.Task::(prepare) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31504::DEBUG::2012-12-02 15:05:58,268::task::1139::TaskManager.Task::(prepare) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::Prepare: aborted: Error validating master storage domain >Thread-31504::DEBUG::2012-12-02 15:05:58,268::task::957::TaskManager.Task::(_decref) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::ref 0 aborting True >Thread-31504::DEBUG::2012-12-02 15:05:58,269::task::892::TaskManager.Task::(_doAbort) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::Task._doAbort: force False >Thread-31504::DEBUG::2012-12-02 15:05:58,269::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31504::DEBUG::2012-12-02 15:05:58,269::task::568::TaskManager.Task::(_updateState) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::moving from state preparing -> state aborting >Thread-31504::DEBUG::2012-12-02 15:05:58,269::task::523::TaskManager.Task::(__state_aborting) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::_aborting: recover policy none >Thread-31504::DEBUG::2012-12-02 15:05:58,269::task::568::TaskManager.Task::(_updateState) Task=`7c163b9b-13f1-48ca-a480-25491b4c4444`::moving from state aborting -> state failed >Thread-31504::DEBUG::2012-12-02 15:05:58,269::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31504::DEBUG::2012-12-02 15:05:58,269::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31504::ERROR::2012-12-02 15:05:58,269::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31565::DEBUG::2012-12-02 15:06:01,881::task::568::TaskManager.Task::(_updateState) Task=`f0d59132-da8d-4295-b109-f0d3ea36e8dc`::moving from state init -> state preparing >Thread-31565::INFO::2012-12-02 15:06:01,881::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31565::INFO::2012-12-02 15:06:01,881::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00759696960449', 'lastCheck': 1354453560.9982269, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31565::DEBUG::2012-12-02 15:06:01,881::task::1151::TaskManager.Task::(prepare) Task=`f0d59132-da8d-4295-b109-f0d3ea36e8dc`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00759696960449', 'lastCheck': 1354453560.9982269, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31565::DEBUG::2012-12-02 15:06:01,882::task::568::TaskManager.Task::(_updateState) Task=`f0d59132-da8d-4295-b109-f0d3ea36e8dc`::moving from state preparing -> state finished >Thread-31565::DEBUG::2012-12-02 15:06:01,882::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31565::DEBUG::2012-12-02 15:06:01,882::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31565::DEBUG::2012-12-02 15:06:01,882::task::957::TaskManager.Task::(_decref) Task=`f0d59132-da8d-4295-b109-f0d3ea36e8dc`::ref 0 aborting False >Thread-31570::DEBUG::2012-12-02 15:06:08,300::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31570::DEBUG::2012-12-02 15:06:08,300::task::568::TaskManager.Task::(_updateState) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::moving from state init -> state preparing >Thread-31570::INFO::2012-12-02 15:06:08,301::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31570::DEBUG::2012-12-02 15:06:08,301::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31570::DEBUG::2012-12-02 15:06:08,301::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31570::DEBUG::2012-12-02 15:06:08,504::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31570::WARNING::2012-12-02 15:06:08,506::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31570::DEBUG::2012-12-02 15:06:08,506::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31570::ERROR::2012-12-02 15:06:08,535::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31570::ERROR::2012-12-02 15:06:08,535::task::833::TaskManager.Task::(_setError) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31570::DEBUG::2012-12-02 15:06:08,535::task::852::TaskManager.Task::(_run) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::Task._run: bd9bed8d-88b9-447f-9032-d3318a432ab2 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31570::DEBUG::2012-12-02 15:06:08,535::task::1177::TaskManager.Task::(stop) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::stopping in state preparing (force False) >Thread-31570::DEBUG::2012-12-02 15:06:08,536::task::957::TaskManager.Task::(_decref) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::ref 1 aborting True >Thread-31570::INFO::2012-12-02 15:06:08,536::task::1134::TaskManager.Task::(prepare) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31570::DEBUG::2012-12-02 15:06:08,536::task::1139::TaskManager.Task::(prepare) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::Prepare: aborted: Error validating master storage domain >Thread-31570::DEBUG::2012-12-02 15:06:08,536::task::957::TaskManager.Task::(_decref) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::ref 0 aborting True >Thread-31570::DEBUG::2012-12-02 15:06:08,536::task::892::TaskManager.Task::(_doAbort) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::Task._doAbort: force False >Thread-31570::DEBUG::2012-12-02 15:06:08,536::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31570::DEBUG::2012-12-02 15:06:08,536::task::568::TaskManager.Task::(_updateState) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::moving from state preparing -> state aborting >Thread-31570::DEBUG::2012-12-02 15:06:08,537::task::523::TaskManager.Task::(__state_aborting) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::_aborting: recover policy none >Thread-31570::DEBUG::2012-12-02 15:06:08,537::task::568::TaskManager.Task::(_updateState) Task=`bd9bed8d-88b9-447f-9032-d3318a432ab2`::moving from state aborting -> state failed >Thread-31570::DEBUG::2012-12-02 15:06:08,537::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31570::DEBUG::2012-12-02 15:06:08,537::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31570::ERROR::2012-12-02 15:06:08,537::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31631::DEBUG::2012-12-02 15:06:12,010::task::568::TaskManager.Task::(_updateState) Task=`870b110a-9dda-4622-b280-09ca94698530`::moving from state init -> state preparing >Thread-31631::INFO::2012-12-02 15:06:12,010::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31631::INFO::2012-12-02 15:06:12,010::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0112338066101', 'lastCheck': 1354453571.0193419, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31631::DEBUG::2012-12-02 15:06:12,010::task::1151::TaskManager.Task::(prepare) Task=`870b110a-9dda-4622-b280-09ca94698530`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0112338066101', 'lastCheck': 1354453571.0193419, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31631::DEBUG::2012-12-02 15:06:12,010::task::568::TaskManager.Task::(_updateState) Task=`870b110a-9dda-4622-b280-09ca94698530`::moving from state preparing -> state finished >Thread-31631::DEBUG::2012-12-02 15:06:12,011::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31631::DEBUG::2012-12-02 15:06:12,011::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31631::DEBUG::2012-12-02 15:06:12,011::task::957::TaskManager.Task::(_decref) Task=`870b110a-9dda-4622-b280-09ca94698530`::ref 0 aborting False >Thread-31636::DEBUG::2012-12-02 15:06:18,568::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31636::DEBUG::2012-12-02 15:06:18,568::task::568::TaskManager.Task::(_updateState) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::moving from state init -> state preparing >Thread-31636::INFO::2012-12-02 15:06:18,568::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31636::DEBUG::2012-12-02 15:06:18,569::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31636::DEBUG::2012-12-02 15:06:18,569::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31636::DEBUG::2012-12-02 15:06:18,773::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31636::WARNING::2012-12-02 15:06:18,775::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31636::DEBUG::2012-12-02 15:06:18,775::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31636::ERROR::2012-12-02 15:06:18,805::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31636::ERROR::2012-12-02 15:06:18,805::task::833::TaskManager.Task::(_setError) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31636::DEBUG::2012-12-02 15:06:18,805::task::852::TaskManager.Task::(_run) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::Task._run: 5b76249c-01ac-4837-816e-91b452b531d3 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31636::DEBUG::2012-12-02 15:06:18,805::task::1177::TaskManager.Task::(stop) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::stopping in state preparing (force False) >Thread-31636::DEBUG::2012-12-02 15:06:18,806::task::957::TaskManager.Task::(_decref) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::ref 1 aborting True >Thread-31636::INFO::2012-12-02 15:06:18,806::task::1134::TaskManager.Task::(prepare) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31636::DEBUG::2012-12-02 15:06:18,806::task::1139::TaskManager.Task::(prepare) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::Prepare: aborted: Error validating master storage domain >Thread-31636::DEBUG::2012-12-02 15:06:18,806::task::957::TaskManager.Task::(_decref) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::ref 0 aborting True >Thread-31636::DEBUG::2012-12-02 15:06:18,806::task::892::TaskManager.Task::(_doAbort) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::Task._doAbort: force False >Thread-31636::DEBUG::2012-12-02 15:06:18,806::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31636::DEBUG::2012-12-02 15:06:18,806::task::568::TaskManager.Task::(_updateState) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::moving from state preparing -> state aborting >Thread-31636::DEBUG::2012-12-02 15:06:18,806::task::523::TaskManager.Task::(__state_aborting) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::_aborting: recover policy none >Thread-31636::DEBUG::2012-12-02 15:06:18,807::task::568::TaskManager.Task::(_updateState) Task=`5b76249c-01ac-4837-816e-91b452b531d3`::moving from state aborting -> state failed >Thread-31636::DEBUG::2012-12-02 15:06:18,807::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31636::DEBUG::2012-12-02 15:06:18,807::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31636::ERROR::2012-12-02 15:06:18,807::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31697::DEBUG::2012-12-02 15:06:22,133::task::568::TaskManager.Task::(_updateState) Task=`fb872934-a455-47ca-8b4f-13ff5d2a38bc`::moving from state init -> state preparing >Thread-31697::INFO::2012-12-02 15:06:22,133::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31697::INFO::2012-12-02 15:06:22,133::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101799964905', 'lastCheck': 1354453581.038692, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31697::DEBUG::2012-12-02 15:06:22,134::task::1151::TaskManager.Task::(prepare) Task=`fb872934-a455-47ca-8b4f-13ff5d2a38bc`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101799964905', 'lastCheck': 1354453581.038692, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31697::DEBUG::2012-12-02 15:06:22,134::task::568::TaskManager.Task::(_updateState) Task=`fb872934-a455-47ca-8b4f-13ff5d2a38bc`::moving from state preparing -> state finished >Thread-31697::DEBUG::2012-12-02 15:06:22,134::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31697::DEBUG::2012-12-02 15:06:22,134::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31697::DEBUG::2012-12-02 15:06:22,134::task::957::TaskManager.Task::(_decref) Task=`fb872934-a455-47ca-8b4f-13ff5d2a38bc`::ref 0 aborting False >Thread-31702::DEBUG::2012-12-02 15:06:28,838::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31702::DEBUG::2012-12-02 15:06:28,838::task::568::TaskManager.Task::(_updateState) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::moving from state init -> state preparing >Thread-31702::INFO::2012-12-02 15:06:28,838::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31702::DEBUG::2012-12-02 15:06:28,839::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31702::DEBUG::2012-12-02 15:06:28,839::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31702::DEBUG::2012-12-02 15:06:29,045::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31702::WARNING::2012-12-02 15:06:29,047::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31702::DEBUG::2012-12-02 15:06:29,047::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31702::ERROR::2012-12-02 15:06:29,076::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31702::ERROR::2012-12-02 15:06:29,076::task::833::TaskManager.Task::(_setError) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31702::DEBUG::2012-12-02 15:06:29,076::task::852::TaskManager.Task::(_run) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::Task._run: 9146639d-f06d-4005-a75f-6f3d84bc549d ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31702::DEBUG::2012-12-02 15:06:29,077::task::1177::TaskManager.Task::(stop) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::stopping in state preparing (force False) >Thread-31702::DEBUG::2012-12-02 15:06:29,077::task::957::TaskManager.Task::(_decref) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::ref 1 aborting True >Thread-31702::INFO::2012-12-02 15:06:29,077::task::1134::TaskManager.Task::(prepare) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31702::DEBUG::2012-12-02 15:06:29,077::task::1139::TaskManager.Task::(prepare) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::Prepare: aborted: Error validating master storage domain >Thread-31702::DEBUG::2012-12-02 15:06:29,077::task::957::TaskManager.Task::(_decref) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::ref 0 aborting True >Thread-31702::DEBUG::2012-12-02 15:06:29,077::task::892::TaskManager.Task::(_doAbort) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::Task._doAbort: force False >Thread-31702::DEBUG::2012-12-02 15:06:29,077::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31702::DEBUG::2012-12-02 15:06:29,078::task::568::TaskManager.Task::(_updateState) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::moving from state preparing -> state aborting >Thread-31702::DEBUG::2012-12-02 15:06:29,078::task::523::TaskManager.Task::(__state_aborting) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::_aborting: recover policy none >Thread-31702::DEBUG::2012-12-02 15:06:29,078::task::568::TaskManager.Task::(_updateState) Task=`9146639d-f06d-4005-a75f-6f3d84bc549d`::moving from state aborting -> state failed >Thread-31702::DEBUG::2012-12-02 15:06:29,078::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31702::DEBUG::2012-12-02 15:06:29,078::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31702::ERROR::2012-12-02 15:06:29,078::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31763::DEBUG::2012-12-02 15:06:32,244::task::568::TaskManager.Task::(_updateState) Task=`abb907c4-cdae-469a-8f3a-c916a44a4741`::moving from state init -> state preparing >Thread-31763::INFO::2012-12-02 15:06:32,244::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31763::INFO::2012-12-02 15:06:32,245::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0590589046478', 'lastCheck': 1354453591.113591, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31763::DEBUG::2012-12-02 15:06:32,245::task::1151::TaskManager.Task::(prepare) Task=`abb907c4-cdae-469a-8f3a-c916a44a4741`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0590589046478', 'lastCheck': 1354453591.113591, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31763::DEBUG::2012-12-02 15:06:32,245::task::568::TaskManager.Task::(_updateState) Task=`abb907c4-cdae-469a-8f3a-c916a44a4741`::moving from state preparing -> state finished >Thread-31763::DEBUG::2012-12-02 15:06:32,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31763::DEBUG::2012-12-02 15:06:32,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31763::DEBUG::2012-12-02 15:06:32,245::task::957::TaskManager.Task::(_decref) Task=`abb907c4-cdae-469a-8f3a-c916a44a4741`::ref 0 aborting False >Thread-31768::DEBUG::2012-12-02 15:06:39,101::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31768::DEBUG::2012-12-02 15:06:39,102::task::568::TaskManager.Task::(_updateState) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::moving from state init -> state preparing >Thread-31768::INFO::2012-12-02 15:06:39,102::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31768::DEBUG::2012-12-02 15:06:39,102::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31768::DEBUG::2012-12-02 15:06:39,102::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31768::DEBUG::2012-12-02 15:06:39,306::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31768::WARNING::2012-12-02 15:06:39,308::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31768::DEBUG::2012-12-02 15:06:39,308::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31768::ERROR::2012-12-02 15:06:39,337::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31768::ERROR::2012-12-02 15:06:39,337::task::833::TaskManager.Task::(_setError) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31768::DEBUG::2012-12-02 15:06:39,337::task::852::TaskManager.Task::(_run) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::Task._run: 27f63873-7caa-41b3-a9b5-969f9907246c ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31768::DEBUG::2012-12-02 15:06:39,337::task::1177::TaskManager.Task::(stop) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::stopping in state preparing (force False) >Thread-31768::DEBUG::2012-12-02 15:06:39,338::task::957::TaskManager.Task::(_decref) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::ref 1 aborting True >Thread-31768::INFO::2012-12-02 15:06:39,338::task::1134::TaskManager.Task::(prepare) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31768::DEBUG::2012-12-02 15:06:39,338::task::1139::TaskManager.Task::(prepare) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::Prepare: aborted: Error validating master storage domain >Thread-31768::DEBUG::2012-12-02 15:06:39,338::task::957::TaskManager.Task::(_decref) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::ref 0 aborting True >Thread-31768::DEBUG::2012-12-02 15:06:39,338::task::892::TaskManager.Task::(_doAbort) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::Task._doAbort: force False >Thread-31768::DEBUG::2012-12-02 15:06:39,338::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31768::DEBUG::2012-12-02 15:06:39,338::task::568::TaskManager.Task::(_updateState) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::moving from state preparing -> state aborting >Thread-31768::DEBUG::2012-12-02 15:06:39,339::task::523::TaskManager.Task::(__state_aborting) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::_aborting: recover policy none >Thread-31768::DEBUG::2012-12-02 15:06:39,339::task::568::TaskManager.Task::(_updateState) Task=`27f63873-7caa-41b3-a9b5-969f9907246c`::moving from state aborting -> state failed >Thread-31768::DEBUG::2012-12-02 15:06:39,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31768::DEBUG::2012-12-02 15:06:39,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31768::ERROR::2012-12-02 15:06:39,339::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31829::DEBUG::2012-12-02 15:06:42,360::task::568::TaskManager.Task::(_updateState) Task=`366185d0-79ba-48f2-91e1-332afb56a1b4`::moving from state init -> state preparing >Thread-31829::INFO::2012-12-02 15:06:42,360::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31829::INFO::2012-12-02 15:06:42,360::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0085871219635', 'lastCheck': 1354453601.13201, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31829::DEBUG::2012-12-02 15:06:42,360::task::1151::TaskManager.Task::(prepare) Task=`366185d0-79ba-48f2-91e1-332afb56a1b4`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0085871219635', 'lastCheck': 1354453601.13201, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31829::DEBUG::2012-12-02 15:06:42,360::task::568::TaskManager.Task::(_updateState) Task=`366185d0-79ba-48f2-91e1-332afb56a1b4`::moving from state preparing -> state finished >Thread-31829::DEBUG::2012-12-02 15:06:42,360::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31829::DEBUG::2012-12-02 15:06:42,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31829::DEBUG::2012-12-02 15:06:42,361::task::957::TaskManager.Task::(_decref) Task=`366185d0-79ba-48f2-91e1-332afb56a1b4`::ref 0 aborting False >Thread-31834::DEBUG::2012-12-02 15:06:49,365::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31834::DEBUG::2012-12-02 15:06:49,366::task::568::TaskManager.Task::(_updateState) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::moving from state init -> state preparing >Thread-31834::INFO::2012-12-02 15:06:49,366::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31834::DEBUG::2012-12-02 15:06:49,366::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31834::DEBUG::2012-12-02 15:06:49,366::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31834::DEBUG::2012-12-02 15:06:49,571::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31834::WARNING::2012-12-02 15:06:49,573::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31834::DEBUG::2012-12-02 15:06:49,573::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31834::ERROR::2012-12-02 15:06:49,601::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31834::ERROR::2012-12-02 15:06:49,601::task::833::TaskManager.Task::(_setError) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31834::DEBUG::2012-12-02 15:06:49,602::task::852::TaskManager.Task::(_run) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::Task._run: 9d28d41f-b1b3-4d04-86d5-3864bbd6d68e ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31834::DEBUG::2012-12-02 15:06:49,602::task::1177::TaskManager.Task::(stop) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::stopping in state preparing (force False) >Thread-31834::DEBUG::2012-12-02 15:06:49,602::task::957::TaskManager.Task::(_decref) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::ref 1 aborting True >Thread-31834::INFO::2012-12-02 15:06:49,602::task::1134::TaskManager.Task::(prepare) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31834::DEBUG::2012-12-02 15:06:49,602::task::1139::TaskManager.Task::(prepare) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::Prepare: aborted: Error validating master storage domain >Thread-31834::DEBUG::2012-12-02 15:06:49,602::task::957::TaskManager.Task::(_decref) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::ref 0 aborting True >Thread-31834::DEBUG::2012-12-02 15:06:49,602::task::892::TaskManager.Task::(_doAbort) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::Task._doAbort: force False >Thread-31834::DEBUG::2012-12-02 15:06:49,602::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31834::DEBUG::2012-12-02 15:06:49,603::task::568::TaskManager.Task::(_updateState) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::moving from state preparing -> state aborting >Thread-31834::DEBUG::2012-12-02 15:06:49,603::task::523::TaskManager.Task::(__state_aborting) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::_aborting: recover policy none >Thread-31834::DEBUG::2012-12-02 15:06:49,603::task::568::TaskManager.Task::(_updateState) Task=`9d28d41f-b1b3-4d04-86d5-3864bbd6d68e`::moving from state aborting -> state failed >Thread-31834::DEBUG::2012-12-02 15:06:49,603::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31834::DEBUG::2012-12-02 15:06:49,603::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31834::ERROR::2012-12-02 15:06:49,603::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31895::DEBUG::2012-12-02 15:06:52,541::task::568::TaskManager.Task::(_updateState) Task=`95605480-a175-42b9-a752-ada00a41c39e`::moving from state init -> state preparing >Thread-31895::INFO::2012-12-02 15:06:52,541::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31895::INFO::2012-12-02 15:06:52,541::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00812196731567', 'lastCheck': 1354453611.1511509, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31895::DEBUG::2012-12-02 15:06:52,542::task::1151::TaskManager.Task::(prepare) Task=`95605480-a175-42b9-a752-ada00a41c39e`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00812196731567', 'lastCheck': 1354453611.1511509, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31895::DEBUG::2012-12-02 15:06:52,542::task::568::TaskManager.Task::(_updateState) Task=`95605480-a175-42b9-a752-ada00a41c39e`::moving from state preparing -> state finished >Thread-31895::DEBUG::2012-12-02 15:06:52,542::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31895::DEBUG::2012-12-02 15:06:52,542::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31895::DEBUG::2012-12-02 15:06:52,542::task::957::TaskManager.Task::(_decref) Task=`95605480-a175-42b9-a752-ada00a41c39e`::ref 0 aborting False >Thread-31900::DEBUG::2012-12-02 15:06:59,630::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31900::DEBUG::2012-12-02 15:06:59,630::task::568::TaskManager.Task::(_updateState) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::moving from state init -> state preparing >Thread-31900::INFO::2012-12-02 15:06:59,631::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31900::DEBUG::2012-12-02 15:06:59,631::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31900::DEBUG::2012-12-02 15:06:59,631::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31900::DEBUG::2012-12-02 15:06:59,836::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31900::WARNING::2012-12-02 15:06:59,837::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31900::DEBUG::2012-12-02 15:06:59,838::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31900::ERROR::2012-12-02 15:06:59,872::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31900::ERROR::2012-12-02 15:06:59,872::task::833::TaskManager.Task::(_setError) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31900::DEBUG::2012-12-02 15:06:59,873::task::852::TaskManager.Task::(_run) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::Task._run: c52857f9-9bf5-420f-b5c2-b4508728d34d ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31900::DEBUG::2012-12-02 15:06:59,873::task::1177::TaskManager.Task::(stop) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::stopping in state preparing (force False) >Thread-31900::DEBUG::2012-12-02 15:06:59,873::task::957::TaskManager.Task::(_decref) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::ref 1 aborting True >Thread-31900::INFO::2012-12-02 15:06:59,873::task::1134::TaskManager.Task::(prepare) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31900::DEBUG::2012-12-02 15:06:59,873::task::1139::TaskManager.Task::(prepare) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::Prepare: aborted: Error validating master storage domain >Thread-31900::DEBUG::2012-12-02 15:06:59,873::task::957::TaskManager.Task::(_decref) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::ref 0 aborting True >Thread-31900::DEBUG::2012-12-02 15:06:59,874::task::892::TaskManager.Task::(_doAbort) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::Task._doAbort: force False >Thread-31900::DEBUG::2012-12-02 15:06:59,874::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31900::DEBUG::2012-12-02 15:06:59,874::task::568::TaskManager.Task::(_updateState) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::moving from state preparing -> state aborting >Thread-31900::DEBUG::2012-12-02 15:06:59,874::task::523::TaskManager.Task::(__state_aborting) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::_aborting: recover policy none >Thread-31900::DEBUG::2012-12-02 15:06:59,874::task::568::TaskManager.Task::(_updateState) Task=`c52857f9-9bf5-420f-b5c2-b4508728d34d`::moving from state aborting -> state failed >Thread-31900::DEBUG::2012-12-02 15:06:59,874::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31900::DEBUG::2012-12-02 15:06:59,874::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31900::ERROR::2012-12-02 15:06:59,874::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-31961::DEBUG::2012-12-02 15:07:02,655::task::568::TaskManager.Task::(_updateState) Task=`6e3f017e-0a0c-4a63-b351-8613eb461311`::moving from state init -> state preparing >Thread-31961::INFO::2012-12-02 15:07:02,655::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-31961::INFO::2012-12-02 15:07:02,655::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00893998146057', 'lastCheck': 1354453621.172276, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31961::DEBUG::2012-12-02 15:07:02,655::task::1151::TaskManager.Task::(prepare) Task=`6e3f017e-0a0c-4a63-b351-8613eb461311`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00893998146057', 'lastCheck': 1354453621.172276, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-31961::DEBUG::2012-12-02 15:07:02,655::task::568::TaskManager.Task::(_updateState) Task=`6e3f017e-0a0c-4a63-b351-8613eb461311`::moving from state preparing -> state finished >Thread-31961::DEBUG::2012-12-02 15:07:02,655::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31961::DEBUG::2012-12-02 15:07:02,656::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31961::DEBUG::2012-12-02 15:07:02,656::task::957::TaskManager.Task::(_decref) Task=`6e3f017e-0a0c-4a63-b351-8613eb461311`::ref 0 aborting False >Thread-31966::DEBUG::2012-12-02 15:07:09,904::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-31966::DEBUG::2012-12-02 15:07:09,905::task::568::TaskManager.Task::(_updateState) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::moving from state init -> state preparing >Thread-31966::INFO::2012-12-02 15:07:09,905::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-31966::DEBUG::2012-12-02 15:07:09,905::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-31966::DEBUG::2012-12-02 15:07:09,906::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-31966::DEBUG::2012-12-02 15:07:10,111::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-31966::WARNING::2012-12-02 15:07:10,113::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-31966::DEBUG::2012-12-02 15:07:10,113::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31966::ERROR::2012-12-02 15:07:10,142::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-31966::ERROR::2012-12-02 15:07:10,143::task::833::TaskManager.Task::(_setError) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-31966::DEBUG::2012-12-02 15:07:10,143::task::852::TaskManager.Task::(_run) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::Task._run: a0c34d12-3af9-4f34-bb6f-e5a7f3872910 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-31966::DEBUG::2012-12-02 15:07:10,143::task::1177::TaskManager.Task::(stop) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::stopping in state preparing (force False) >Thread-31966::DEBUG::2012-12-02 15:07:10,143::task::957::TaskManager.Task::(_decref) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::ref 1 aborting True >Thread-31966::INFO::2012-12-02 15:07:10,143::task::1134::TaskManager.Task::(prepare) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-31966::DEBUG::2012-12-02 15:07:10,143::task::1139::TaskManager.Task::(prepare) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::Prepare: aborted: Error validating master storage domain >Thread-31966::DEBUG::2012-12-02 15:07:10,144::task::957::TaskManager.Task::(_decref) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::ref 0 aborting True >Thread-31966::DEBUG::2012-12-02 15:07:10,144::task::892::TaskManager.Task::(_doAbort) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::Task._doAbort: force False >Thread-31966::DEBUG::2012-12-02 15:07:10,144::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31966::DEBUG::2012-12-02 15:07:10,144::task::568::TaskManager.Task::(_updateState) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::moving from state preparing -> state aborting >Thread-31966::DEBUG::2012-12-02 15:07:10,144::task::523::TaskManager.Task::(__state_aborting) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::_aborting: recover policy none >Thread-31966::DEBUG::2012-12-02 15:07:10,144::task::568::TaskManager.Task::(_updateState) Task=`a0c34d12-3af9-4f34-bb6f-e5a7f3872910`::moving from state aborting -> state failed >Thread-31966::DEBUG::2012-12-02 15:07:10,144::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-31966::DEBUG::2012-12-02 15:07:10,144::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-31966::ERROR::2012-12-02 15:07:10,145::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-32027::DEBUG::2012-12-02 15:07:12,784::task::568::TaskManager.Task::(_updateState) Task=`f7bb9d82-e4c4-4b2b-8309-272eef09be82`::moving from state init -> state preparing >Thread-32027::INFO::2012-12-02 15:07:12,784::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-32027::INFO::2012-12-02 15:07:12,784::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00877809524536', 'lastCheck': 1354453631.19186, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-32027::DEBUG::2012-12-02 15:07:12,785::task::1151::TaskManager.Task::(prepare) Task=`f7bb9d82-e4c4-4b2b-8309-272eef09be82`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00877809524536', 'lastCheck': 1354453631.19186, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-32027::DEBUG::2012-12-02 15:07:12,785::task::568::TaskManager.Task::(_updateState) Task=`f7bb9d82-e4c4-4b2b-8309-272eef09be82`::moving from state preparing -> state finished >Thread-32027::DEBUG::2012-12-02 15:07:12,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32027::DEBUG::2012-12-02 15:07:12,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32027::DEBUG::2012-12-02 15:07:12,785::task::957::TaskManager.Task::(_decref) Task=`f7bb9d82-e4c4-4b2b-8309-272eef09be82`::ref 0 aborting False >Thread-32032::DEBUG::2012-12-02 15:07:20,169::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32032::DEBUG::2012-12-02 15:07:20,170::task::568::TaskManager.Task::(_updateState) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::moving from state init -> state preparing >Thread-32032::INFO::2012-12-02 15:07:20,170::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32032::DEBUG::2012-12-02 15:07:20,170::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32032::DEBUG::2012-12-02 15:07:20,171::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32032::DEBUG::2012-12-02 15:07:20,375::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32032::WARNING::2012-12-02 15:07:20,377::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32032::DEBUG::2012-12-02 15:07:20,377::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32032::ERROR::2012-12-02 15:07:20,404::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32032::ERROR::2012-12-02 15:07:20,405::task::833::TaskManager.Task::(_setError) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-32032::DEBUG::2012-12-02 15:07:20,405::task::852::TaskManager.Task::(_run) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::Task._run: b89b34d6-2590-4a37-8e70-624a0130a78a ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-32032::DEBUG::2012-12-02 15:07:20,405::task::1177::TaskManager.Task::(stop) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::stopping in state preparing (force False) >Thread-32032::DEBUG::2012-12-02 15:07:20,405::task::957::TaskManager.Task::(_decref) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::ref 1 aborting True >Thread-32032::INFO::2012-12-02 15:07:20,405::task::1134::TaskManager.Task::(prepare) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-32032::DEBUG::2012-12-02 15:07:20,406::task::1139::TaskManager.Task::(prepare) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::Prepare: aborted: Error validating master storage domain >Thread-32032::DEBUG::2012-12-02 15:07:20,406::task::957::TaskManager.Task::(_decref) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::ref 0 aborting True >Thread-32032::DEBUG::2012-12-02 15:07:20,406::task::892::TaskManager.Task::(_doAbort) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::Task._doAbort: force False >Thread-32032::DEBUG::2012-12-02 15:07:20,406::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32032::DEBUG::2012-12-02 15:07:20,406::task::568::TaskManager.Task::(_updateState) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::moving from state preparing -> state aborting >Thread-32032::DEBUG::2012-12-02 15:07:20,406::task::523::TaskManager.Task::(__state_aborting) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::_aborting: recover policy none >Thread-32032::DEBUG::2012-12-02 15:07:20,406::task::568::TaskManager.Task::(_updateState) Task=`b89b34d6-2590-4a37-8e70-624a0130a78a`::moving from state aborting -> state failed >Thread-32032::DEBUG::2012-12-02 15:07:20,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32032::DEBUG::2012-12-02 15:07:20,407::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32032::ERROR::2012-12-02 15:07:20,407::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-32093::DEBUG::2012-12-02 15:07:22,885::task::568::TaskManager.Task::(_updateState) Task=`208f42fa-bd73-4ee2-a06e-a44d3109be31`::moving from state init -> state preparing >Thread-32093::INFO::2012-12-02 15:07:22,885::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-32093::INFO::2012-12-02 15:07:22,885::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0079939365387', 'lastCheck': 1354453641.2099581, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-32093::DEBUG::2012-12-02 15:07:22,885::task::1151::TaskManager.Task::(prepare) Task=`208f42fa-bd73-4ee2-a06e-a44d3109be31`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0079939365387', 'lastCheck': 1354453641.2099581, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-32093::DEBUG::2012-12-02 15:07:22,886::task::568::TaskManager.Task::(_updateState) Task=`208f42fa-bd73-4ee2-a06e-a44d3109be31`::moving from state preparing -> state finished >Thread-32093::DEBUG::2012-12-02 15:07:22,886::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32093::DEBUG::2012-12-02 15:07:22,886::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32093::DEBUG::2012-12-02 15:07:22,886::task::957::TaskManager.Task::(_decref) Task=`208f42fa-bd73-4ee2-a06e-a44d3109be31`::ref 0 aborting False >Thread-32098::DEBUG::2012-12-02 15:07:30,430::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32098::DEBUG::2012-12-02 15:07:30,431::task::568::TaskManager.Task::(_updateState) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::moving from state init -> state preparing >Thread-32098::INFO::2012-12-02 15:07:30,431::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32098::DEBUG::2012-12-02 15:07:30,431::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32098::DEBUG::2012-12-02 15:07:30,431::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32098::DEBUG::2012-12-02 15:07:30,635::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32098::WARNING::2012-12-02 15:07:30,636::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32098::DEBUG::2012-12-02 15:07:30,637::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32098::ERROR::2012-12-02 15:07:30,665::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32098::ERROR::2012-12-02 15:07:30,665::task::833::TaskManager.Task::(_setError) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-32098::DEBUG::2012-12-02 15:07:30,665::task::852::TaskManager.Task::(_run) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::Task._run: 2cee0a77-d503-4a2a-a791-965a1e2ddec6 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-32098::DEBUG::2012-12-02 15:07:30,666::task::1177::TaskManager.Task::(stop) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::stopping in state preparing (force False) >Thread-32098::DEBUG::2012-12-02 15:07:30,666::task::957::TaskManager.Task::(_decref) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::ref 1 aborting True >Thread-32098::INFO::2012-12-02 15:07:30,666::task::1134::TaskManager.Task::(prepare) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-32098::DEBUG::2012-12-02 15:07:30,666::task::1139::TaskManager.Task::(prepare) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::Prepare: aborted: Error validating master storage domain >Thread-32098::DEBUG::2012-12-02 15:07:30,666::task::957::TaskManager.Task::(_decref) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::ref 0 aborting True >Thread-32098::DEBUG::2012-12-02 15:07:30,666::task::892::TaskManager.Task::(_doAbort) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::Task._doAbort: force False >Thread-32098::DEBUG::2012-12-02 15:07:30,666::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32098::DEBUG::2012-12-02 15:07:30,666::task::568::TaskManager.Task::(_updateState) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::moving from state preparing -> state aborting >Thread-32098::DEBUG::2012-12-02 15:07:30,667::task::523::TaskManager.Task::(__state_aborting) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::_aborting: recover policy none >Thread-32098::DEBUG::2012-12-02 15:07:30,667::task::568::TaskManager.Task::(_updateState) Task=`2cee0a77-d503-4a2a-a791-965a1e2ddec6`::moving from state aborting -> state failed >Thread-32098::DEBUG::2012-12-02 15:07:30,667::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32098::DEBUG::2012-12-02 15:07:30,667::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32098::ERROR::2012-12-02 15:07:30,667::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-32159::DEBUG::2012-12-02 15:07:32,980::task::568::TaskManager.Task::(_updateState) Task=`a9854304-6a2b-4829-879c-fb81d95b0c9f`::moving from state init -> state preparing >Thread-32159::INFO::2012-12-02 15:07:32,981::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-32159::INFO::2012-12-02 15:07:32,981::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00890111923218', 'lastCheck': 1354453651.229332, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-32159::DEBUG::2012-12-02 15:07:32,981::task::1151::TaskManager.Task::(prepare) Task=`a9854304-6a2b-4829-879c-fb81d95b0c9f`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00890111923218', 'lastCheck': 1354453651.229332, 'code': 0, 'valid': True}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'delay': '0', 'lastCheck': 1354453544.3713009, 'code': 200, 'valid': False}} >Thread-32159::DEBUG::2012-12-02 15:07:32,981::task::568::TaskManager.Task::(_updateState) Task=`a9854304-6a2b-4829-879c-fb81d95b0c9f`::moving from state preparing -> state finished >Thread-32159::DEBUG::2012-12-02 15:07:32,981::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32159::DEBUG::2012-12-02 15:07:32,981::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32159::DEBUG::2012-12-02 15:07:32,981::task::957::TaskManager.Task::(_decref) Task=`a9854304-6a2b-4829-879c-fb81d95b0c9f`::ref 0 aborting False >Thread-32161::DEBUG::2012-12-02 15:07:40,700::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32161::DEBUG::2012-12-02 15:07:40,700::task::568::TaskManager.Task::(_updateState) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::moving from state init -> state preparing >Thread-32161::INFO::2012-12-02 15:07:40,700::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32161::DEBUG::2012-12-02 15:07:40,700::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32161::DEBUG::2012-12-02 15:07:40,701::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32161::DEBUG::2012-12-02 15:07:40,904::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32161::WARNING::2012-12-02 15:07:40,906::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32161::DEBUG::2012-12-02 15:07:40,906::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32161::ERROR::2012-12-02 15:07:40,936::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32161::ERROR::2012-12-02 15:07:40,937::task::833::TaskManager.Task::(_setError) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-32161::DEBUG::2012-12-02 15:07:40,937::task::852::TaskManager.Task::(_run) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::Task._run: ae900c90-a208-4aa1-be84-1161e54175ff ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-32161::DEBUG::2012-12-02 15:07:40,937::task::1177::TaskManager.Task::(stop) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::stopping in state preparing (force False) >Thread-32161::DEBUG::2012-12-02 15:07:40,937::task::957::TaskManager.Task::(_decref) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::ref 1 aborting True >Thread-32161::INFO::2012-12-02 15:07:40,937::task::1134::TaskManager.Task::(prepare) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-32161::DEBUG::2012-12-02 15:07:40,938::task::1139::TaskManager.Task::(prepare) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::Prepare: aborted: Error validating master storage domain >Thread-32161::DEBUG::2012-12-02 15:07:40,938::task::957::TaskManager.Task::(_decref) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::ref 0 aborting True >Thread-32161::DEBUG::2012-12-02 15:07:40,938::task::892::TaskManager.Task::(_doAbort) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::Task._doAbort: force False >Thread-32161::DEBUG::2012-12-02 15:07:40,938::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32161::DEBUG::2012-12-02 15:07:40,938::task::568::TaskManager.Task::(_updateState) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::moving from state preparing -> state aborting >Thread-32161::DEBUG::2012-12-02 15:07:40,938::task::523::TaskManager.Task::(__state_aborting) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::_aborting: recover policy none >Thread-32161::DEBUG::2012-12-02 15:07:40,938::task::568::TaskManager.Task::(_updateState) Task=`ae900c90-a208-4aa1-be84-1161e54175ff`::moving from state aborting -> state failed >Thread-32161::DEBUG::2012-12-02 15:07:40,938::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32161::DEBUG::2012-12-02 15:07:40,939::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32161::ERROR::2012-12-02 15:07:40,939::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-32221::DEBUG::2012-12-02 15:07:41,469::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32221::DEBUG::2012-12-02 15:07:41,469::task::568::TaskManager.Task::(_updateState) Task=`808110ff-32fb-4129-920e-7a0a605f6216`::moving from state init -> state preparing >Thread-32221::INFO::2012-12-02 15:07:41,469::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', remove=False, options=None) >Thread-32221::DEBUG::2012-12-02 15:07:41,470::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`bb1b7788-cb1f-4e0e-8cfb-14fbe5ce8391`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32221::DEBUG::2012-12-02 15:07:41,470::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-32221::DEBUG::2012-12-02 15:07:41,470::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-32221::DEBUG::2012-12-02 15:07:41,470::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`bb1b7788-cb1f-4e0e-8cfb-14fbe5ce8391`::Granted request >Thread-32221::DEBUG::2012-12-02 15:07:41,471::task::794::TaskManager.Task::(resourceAcquired) Task=`808110ff-32fb-4129-920e-7a0a605f6216`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (exclusive) >Thread-32221::DEBUG::2012-12-02 15:07:41,471::task::957::TaskManager.Task::(_decref) Task=`808110ff-32fb-4129-920e-7a0a605f6216`::ref 1 aborting False >Thread-32221::INFO::2012-12-02 15:07:41,471::sp::658::Storage.StoragePool::(disconnect) Disconnect from the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 >Thread-30910::DEBUG::2012-12-02 15:07:41,472::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 4e891d8c-9947-468e-88ec-29e49d68e083 is stopping >Thread-30910::INFO::2012-12-02 15:07:41,472::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-30910::DEBUG::2012-12-02 15:07:43,472::safelease::193::SANLock::(releaseHostId) Host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 released successfully (id: 3) >Thread-32221::INFO::2012-12-02 15:07:43,472::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-32221::DEBUG::2012-12-02 15:07:43,473::task::1151::TaskManager.Task::(prepare) Task=`808110ff-32fb-4129-920e-7a0a605f6216`::finished: True >Thread-32221::DEBUG::2012-12-02 15:07:43,473::task::568::TaskManager.Task::(_updateState) Task=`808110ff-32fb-4129-920e-7a0a605f6216`::moving from state preparing -> state finished >Thread-32221::DEBUG::2012-12-02 15:07:43,473::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-32221::DEBUG::2012-12-02 15:07:43,473::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32221::DEBUG::2012-12-02 15:07:43,473::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-32221::DEBUG::2012-12-02 15:07:43,473::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-32221::DEBUG::2012-12-02 15:07:43,474::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-32221::DEBUG::2012-12-02 15:07:43,474::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-32221::DEBUG::2012-12-02 15:07:43,474::task::957::TaskManager.Task::(_decref) Task=`808110ff-32fb-4129-920e-7a0a605f6216`::ref 0 aborting False >Thread-32222::DEBUG::2012-12-02 15:07:43,518::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32222::DEBUG::2012-12-02 15:07:43,518::task::568::TaskManager.Task::(_updateState) Task=`8693e69d-fb79-4eab-847e-2b798e3332b2`::moving from state init -> state preparing >Thread-32222::INFO::2012-12-02 15:07:43,519::logUtils::37::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', poolName='dc', masterDom='4e891d8c-9947-468e-88ec-29e49d68e083', domDict={'4e891d8c-9947-468e-88ec-29e49d68e083': 'active', 'a3177eae-1dc9-445a-92b1-da23ec8008bc': 'active'}, masterVersion=2, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=3, options=None) >Thread-32222::INFO::2012-12-02 15:07:43,519::hsm::1483::Storage.HSM::(reconstructMaster) spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6 master=4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32222::INFO::2012-12-02 15:07:43,519::fileSD::260::Storage.StorageDomain::(validate) sdUUID=4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32222::DEBUG::2012-12-02 15:07:43,529::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Regular', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7867c42a2ec25b1e27a5f200d36e7105ccf17687'] >Thread-32222::DEBUG::2012-12-02 15:07:43,529::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`e6eef2b9-89be-4388-95c8-b8f4b649c452`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32222::DEBUG::2012-12-02 15:07:43,529::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-32222::DEBUG::2012-12-02 15:07:43,530::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-32222::DEBUG::2012-12-02 15:07:43,530::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`e6eef2b9-89be-4388-95c8-b8f4b649c452`::Granted request >Thread-32222::DEBUG::2012-12-02 15:07:43,530::task::794::TaskManager.Task::(resourceAcquired) Task=`8693e69d-fb79-4eab-847e-2b798e3332b2`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (exclusive) >Thread-32222::DEBUG::2012-12-02 15:07:43,530::task::957::TaskManager.Task::(_decref) Task=`8693e69d-fb79-4eab-847e-2b798e3332b2`::ref 1 aborting False >Thread-32222::INFO::2012-12-02 15:07:43,530::sp::727::Storage.StoragePool::(reconstructMaster) spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6 hostId=3 poolName=dc msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083 domDict={'4e891d8c-9947-468e-88ec-29e49d68e083': 'active', 'a3177eae-1dc9-445a-92b1-da23ec8008bc': 'active'} masterVersion=2 leaseparams=({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5}) >Thread-32222::INFO::2012-12-02 15:07:43,530::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-32223::DEBUG::2012-12-02 15:07:50,969::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32223::DEBUG::2012-12-02 15:07:50,969::task::568::TaskManager.Task::(_updateState) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::moving from state init -> state preparing >Thread-32223::INFO::2012-12-02 15:07:50,969::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32223::ERROR::2012-12-02 15:07:50,969::task::833::TaskManager.Task::(_setError) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-32223::DEBUG::2012-12-02 15:07:50,970::task::852::TaskManager.Task::(_run) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::Task._run: db192fb7-0f6c-4501-8246-e3b34f3c027c ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-32223::DEBUG::2012-12-02 15:07:50,970::task::1177::TaskManager.Task::(stop) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::stopping in state preparing (force False) >Thread-32223::DEBUG::2012-12-02 15:07:50,970::task::957::TaskManager.Task::(_decref) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::ref 1 aborting True >Thread-32223::INFO::2012-12-02 15:07:50,970::task::1134::TaskManager.Task::(prepare) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-32223::DEBUG::2012-12-02 15:07:50,970::task::1139::TaskManager.Task::(prepare) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::Prepare: aborted: Unknown pool id, pool not connected >Thread-32223::DEBUG::2012-12-02 15:07:50,970::task::957::TaskManager.Task::(_decref) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::ref 0 aborting True >Thread-32223::DEBUG::2012-12-02 15:07:50,971::task::892::TaskManager.Task::(_doAbort) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::Task._doAbort: force False >Thread-32223::DEBUG::2012-12-02 15:07:50,971::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32223::DEBUG::2012-12-02 15:07:50,971::task::568::TaskManager.Task::(_updateState) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::moving from state preparing -> state aborting >Thread-32223::DEBUG::2012-12-02 15:07:50,971::task::523::TaskManager.Task::(__state_aborting) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::_aborting: recover policy none >Thread-32223::DEBUG::2012-12-02 15:07:50,971::task::568::TaskManager.Task::(_updateState) Task=`db192fb7-0f6c-4501-8246-e3b34f3c027c`::moving from state aborting -> state failed >Thread-32223::DEBUG::2012-12-02 15:07:50,971::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32223::DEBUG::2012-12-02 15:07:50,971::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32223::ERROR::2012-12-02 15:07:50,971::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-32224::DEBUG::2012-12-02 15:07:51,029::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32224::DEBUG::2012-12-02 15:07:51,029::task::568::TaskManager.Task::(_updateState) Task=`6da11888-8221-4130-a8e6-e2f39eca7e38`::moving from state init -> state preparing >Thread-32224::INFO::2012-12-02 15:07:51,029::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-32224::DEBUG::2012-12-02 15:07:51,029::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`a7c48e45-1090-4f7f-8ce6-90c63594d902`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32224::DEBUG::2012-12-02 15:07:51,030::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-32224::DEBUG::2012-12-02 15:07:51,030::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is currently locked, Entering queue (1 in queue) >Thread-32222::DEBUG::2012-12-02 15:08:04,531::safelease::178::SANLock::(acquireHostId) Host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully acquired (id: 3) >Thread-32222::INFO::2012-12-02 15:08:04,532::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-32222::DEBUG::2012-12-02 15:08:04,834::safelease::235::SANLock::(acquire) Cluster lock for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully acquired (id: 3) >Thread-32222::INFO::2012-12-02 15:08:04,834::sp::702::Storage.StoragePool::(createMaster) setting master domain for spUUID da40b79b-9f51-4240-9cb2-2ab10e793fb6 on sdUUID=4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32222::DEBUG::2012-12-02 15:08:04,834::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-32222::DEBUG::2012-12-02 15:08:04,834::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-32222::DEBUG::2012-12-02 15:08:04,839::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Regular', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=7867c42a2ec25b1e27a5f200d36e7105ccf17687'] >Thread-32222::WARNING::2012-12-02 15:08:04,839::sd::458::Storage.StorageDomain::(attach) domain `4e891d8c-9947-468e-88ec-29e49d68e083` is already attached to pool `da40b79b-9f51-4240-9cb2-2ab10e793fb6` >Thread-32222::DEBUG::2012-12-02 15:08:04,840::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-32222::DEBUG::2012-12-02 15:08:04,840::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-32222::DEBUG::2012-12-02 15:08:04,840::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-32222::DEBUG::2012-12-02 15:08:04,840::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b6e4b73ca495c073872653c154ce1efbeb8ec17c'] >Thread-32222::DEBUG::2012-12-02 15:08:04,867::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-32222::INFO::2012-12-02 15:08:04,867::sp::757::Storage.StoragePool::(reconstructMaster) Set storage pool domains: {'4e891d8c-9947-468e-88ec-29e49d68e083': 'Active', 'a3177eae-1dc9-445a-92b1-da23ec8008bc': 'Active'} >Thread-32222::DEBUG::2012-12-02 15:08:04,867::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-32222::DEBUG::2012-12-02 15:08:04,867::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-32222::DEBUG::2012-12-02 15:08:04,868::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >Thread-32222::DEBUG::2012-12-02 15:08:04,913::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-32222::DEBUG::2012-12-02 15:08:04,913::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:04,913::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:04,913::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:04,913::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:04,914::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:04,914::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:04,914::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-32222::DEBUG::2012-12-02 15:08:04,914::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-32222::DEBUG::2012-12-02 15:08:04,914::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-32222::DEBUG::2012-12-02 15:08:04,914::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-32222::DEBUG::2012-12-02 15:08:04,914::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-32222::DEBUG::2012-12-02 15:08:04,939::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-32222::DEBUG::2012-12-02 15:08:04,939::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-32222::DEBUG::2012-12-02 15:08:05,573::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-32222::DEBUG::2012-12-02 15:08:05,665::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-32222::DEBUG::2012-12-02 15:08:05,666::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,666::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,666::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,666::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,667::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,667::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,667::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-32222::DEBUG::2012-12-02 15:08:05,667::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,672::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-32222::DEBUG::2012-12-02 15:08:05,883::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-32222::WARNING::2012-12-02 15:08:05,885::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-32222::DEBUG::2012-12-02 15:08:05,885::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,921::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32222::DEBUG::2012-12-02 15:08:05,921::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-32222::DEBUG::2012-12-02 15:08:05,930::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >Thread-32222::DEBUG::2012-12-02 15:08:05,933::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-32222::WARNING::2012-12-02 15:08:05,934::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_imageNS already registered >Thread-32222::WARNING::2012-12-02 15:08:05,934::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_volumeNS already registered >Thread-32222::DEBUG::2012-12-02 15:08:05,934::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 4e891d8c-9947-468e-88ec-29e49d68e083 verified, version 2 >Thread-32222::DEBUG::2012-12-02 15:08:05,934::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-32222::DEBUG::2012-12-02 15:08:05,934::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-32222::DEBUG::2012-12-02 15:08:05,940::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >Thread-32222::DEBUG::2012-12-02 15:08:05,941::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `da40b79b-9f51-4240-9cb2-2ab10e793fb6` started monitoring domain `4e891d8c-9947-468e-88ec-29e49d68e083` >Thread-32222::DEBUG::2012-12-02 15:08:05,941::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:05,941::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32222::DEBUG::2012-12-02 15:08:06,150::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32222::WARNING::2012-12-02 15:08:06,152::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32222::DEBUG::2012-12-02 15:08:06,152::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32222::ERROR::2012-12-02 15:08:06,179::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32222::DEBUG::2012-12-02 15:08:06,180::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-32222::DEBUG::2012-12-02 15:08:06,180::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32222::DEBUG::2012-12-02 15:08:06,181::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32222::DEBUG::2012-12-02 15:08:06,388::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32222::WARNING::2012-12-02 15:08:06,389::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32222::DEBUG::2012-12-02 15:08:06,389::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32222::INFO::2012-12-02 15:08:06,415::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32222::INFO::2012-12-02 15:08:06,415::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/mastersd >Thread-32222::INFO::2012-12-02 15:08:06,416::safelease::239::SANLock::(release) Releasing cluster lock for domain 4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32222::DEBUG::2012-12-02 15:08:06,425::safelease::250::SANLock::(release) Cluster lock for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully released >Thread-32222::INFO::2012-12-02 15:08:06,425::logUtils::39::dispatcher::(wrapper) Run and protect: reconstructMaster, Return response: None >Thread-32222::DEBUG::2012-12-02 15:08:06,426::task::1151::TaskManager.Task::(prepare) Task=`8693e69d-fb79-4eab-847e-2b798e3332b2`::finished: None >Thread-32284::DEBUG::2012-12-02 15:08:06,426::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 4e891d8c-9947-468e-88ec-29e49d68e083 is stopping >Thread-32222::DEBUG::2012-12-02 15:08:06,426::task::568::TaskManager.Task::(_updateState) Task=`8693e69d-fb79-4eab-847e-2b798e3332b2`::moving from state preparing -> state finished >Thread-32284::INFO::2012-12-02 15:08:06,426::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-32222::DEBUG::2012-12-02 15:08:06,426::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-32222::DEBUG::2012-12-02 15:08:06,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32222::DEBUG::2012-12-02 15:08:06,427::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-32222::DEBUG::2012-12-02 15:08:06,427::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-32222::DEBUG::2012-12-02 15:08:06,427::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-32222::DEBUG::2012-12-02 15:08:06,427::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' has 1 requests in queue. Handling top request. >Thread-32222::DEBUG::2012-12-02 15:08:06,427::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`a7c48e45-1090-4f7f-8ce6-90c63594d902`::Granted request >Thread-32222::DEBUG::2012-12-02 15:08:06,428::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 - exclusive: granted' was granted >Thread-32222::DEBUG::2012-12-02 15:08:06,428::task::957::TaskManager.Task::(_decref) Task=`8693e69d-fb79-4eab-847e-2b798e3332b2`::ref 0 aborting False >Thread-32224::INFO::2012-12-02 15:08:06,428::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-32224::DEBUG::2012-12-02 15:08:06,429::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:06,430::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:06,430::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:06,430::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:06,430::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:06,430::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:06,430::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-32224::DEBUG::2012-12-02 15:08:06,430::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-32224::DEBUG::2012-12-02 15:08:06,430::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-32224::DEBUG::2012-12-02 15:08:06,430::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-32224::DEBUG::2012-12-02 15:08:06,430::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-32224::DEBUG::2012-12-02 15:08:06,455::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-32224::DEBUG::2012-12-02 15:08:06,455::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-32224::DEBUG::2012-12-02 15:08:07,095::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-32224::DEBUG::2012-12-02 15:08:07,162::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-32224::DEBUG::2012-12-02 15:08:07,163::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,163::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,163::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,163::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,163::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,164::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,164::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-32224::DEBUG::2012-12-02 15:08:07,164::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,166::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-32224::DEBUG::2012-12-02 15:08:07,370::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-32224::WARNING::2012-12-02 15:08:07,372::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-32224::DEBUG::2012-12-02 15:08:07,372::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,406::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32224::DEBUG::2012-12-02 15:08:07,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-32224::DEBUG::2012-12-02 15:08:07,413::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >Thread-32224::DEBUG::2012-12-02 15:08:07,415::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-32224::WARNING::2012-12-02 15:08:07,415::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_imageNS already registered >Thread-32224::WARNING::2012-12-02 15:08:07,415::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_volumeNS already registered >Thread-32224::DEBUG::2012-12-02 15:08:07,415::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 4e891d8c-9947-468e-88ec-29e49d68e083 verified, version 2 >Thread-32224::DEBUG::2012-12-02 15:08:07,416::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-32224::DEBUG::2012-12-02 15:08:07,416::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-32224::DEBUG::2012-12-02 15:08:07,422::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >Thread-32224::DEBUG::2012-12-02 15:08:07,423::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `da40b79b-9f51-4240-9cb2-2ab10e793fb6` started monitoring domain `4e891d8c-9947-468e-88ec-29e49d68e083` >Thread-32224::DEBUG::2012-12-02 15:08:07,423::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,425::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32224::DEBUG::2012-12-02 15:08:07,633::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32224::WARNING::2012-12-02 15:08:07,634::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32224::DEBUG::2012-12-02 15:08:07,634::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32224::ERROR::2012-12-02 15:08:07,662::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32224::DEBUG::2012-12-02 15:08:07,663::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-32224::DEBUG::2012-12-02 15:08:07,663::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32224::DEBUG::2012-12-02 15:08:07,663::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32224::DEBUG::2012-12-02 15:08:07,870::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32224::WARNING::2012-12-02 15:08:07,871::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32224::DEBUG::2012-12-02 15:08:07,871::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32224::INFO::2012-12-02 15:08:07,899::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32224::INFO::2012-12-02 15:08:07,900::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/mastersd >Thread-32224::DEBUG::2012-12-02 15:08:07,900::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-32224::DEBUG::2012-12-02 15:08:07,900::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-32224::DEBUG::2012-12-02 15:08:07,900::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-32224::DEBUG::2012-12-02 15:08:07,900::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-32224::INFO::2012-12-02 15:08:07,901::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-32224::DEBUG::2012-12-02 15:08:07,901::task::1151::TaskManager.Task::(prepare) Task=`6da11888-8221-4130-a8e6-e2f39eca7e38`::finished: True >Thread-32224::DEBUG::2012-12-02 15:08:07,901::task::568::TaskManager.Task::(_updateState) Task=`6da11888-8221-4130-a8e6-e2f39eca7e38`::moving from state preparing -> state finished >Thread-32224::DEBUG::2012-12-02 15:08:07,901::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32224::DEBUG::2012-12-02 15:08:07,901::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32224::DEBUG::2012-12-02 15:08:07,901::task::957::TaskManager.Task::(_decref) Task=`6da11888-8221-4130-a8e6-e2f39eca7e38`::ref 0 aborting False >Thread-32580::DEBUG::2012-12-02 15:08:07,910::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32580::DEBUG::2012-12-02 15:08:07,910::task::568::TaskManager.Task::(_updateState) Task=`2e9b2635-d574-427a-ba3c-0bf6a9dfd948`::moving from state init -> state preparing >Thread-32580::INFO::2012-12-02 15:08:07,910::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32580::INFO::2012-12-02 15:08:07,910::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32580::DEBUG::2012-12-02 15:08:07,911::task::1151::TaskManager.Task::(prepare) Task=`2e9b2635-d574-427a-ba3c-0bf6a9dfd948`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32580::DEBUG::2012-12-02 15:08:07,911::task::568::TaskManager.Task::(_updateState) Task=`2e9b2635-d574-427a-ba3c-0bf6a9dfd948`::moving from state preparing -> state finished >Thread-32580::DEBUG::2012-12-02 15:08:07,911::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32580::DEBUG::2012-12-02 15:08:07,911::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32580::DEBUG::2012-12-02 15:08:07,911::task::957::TaskManager.Task::(_decref) Task=`2e9b2635-d574-427a-ba3c-0bf6a9dfd948`::ref 0 aborting False >Thread-32581::DEBUG::2012-12-02 15:08:07,970::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32581::DEBUG::2012-12-02 15:08:07,970::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state init -> state preparing >Thread-32581::INFO::2012-12-02 15:08:07,970::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', prevID=-1, prevLVER='-1', recoveryMode=None, scsiFencing='false', maxHostID=250, domVersion='3', options=None) >Thread-32581::DEBUG::2012-12-02 15:08:07,971::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`35af9cb3-93b4-4070-86d2-087079e2eb9e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32581::DEBUG::2012-12-02 15:08:07,971::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-32581::DEBUG::2012-12-02 15:08:07,971::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-32581::DEBUG::2012-12-02 15:08:07,971::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`35af9cb3-93b4-4070-86d2-087079e2eb9e`::Granted request >Thread-32581::DEBUG::2012-12-02 15:08:07,972::task::794::TaskManager.Task::(resourceAcquired) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (exclusive) >Thread-32581::DEBUG::2012-12-02 15:08:07,972::task::957::TaskManager.Task::(_decref) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::ref 1 aborting False >Thread-32581::INFO::2012-12-02 15:08:07,972::logUtils::39::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-32581::DEBUG::2012-12-02 15:08:07,972::task::1146::TaskManager.Task::(prepare) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Prepare: 1 jobs exist, move to acquiring >Thread-32581::DEBUG::2012-12-02 15:08:07,972::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state preparing -> state acquiring >Thread-32581::DEBUG::2012-12-02 15:08:07,972::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state acquiring -> state queued >Thread-32581::DEBUG::2012-12-02 15:08:07,972::taskManager::50::TaskManager::(_queueTask) queuing task: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 >Thread-32581::DEBUG::2012-12-02 15:08:07,972::taskManager::56::TaskManager::(_queueTask) task queued: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 >bfd373c7-6de7-4219-97f6-acded651d2b8::DEBUG::2012-12-02 15:08:07,973::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-32581::DEBUG::2012-12-02 15:08:07,973::task::1148::TaskManager.Task::(prepare) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::returning >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,973::threadPool::212::Misc.ThreadPool.WorkerThread::(run) Task: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 running: <bound method Task.commit of <storage.task.Task instance at 0x7f17b03b1cb0>> with: None >Thread-32581::DEBUG::2012-12-02 15:08:07,973::task::957::TaskManager.Task::(_decref) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::ref 0 aborting False >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,973::task::1159::TaskManager.Task::(commit) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::committing task: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,974::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state queued -> state running >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,974::task::872::TaskManager.Task::(_runJobs) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x22ff050>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,975::task::304::TaskManager.Task::(run) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x22ff050>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) callback None >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,975::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,975::misc::1063::SamplingMethod::(__call__) Got in to sampling method >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,982::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:07,982::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,189::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >983edf27-03aa-409f-a9e0-7d7652f9f4c4::WARNING::2012-12-02 15:08:08,190::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,190::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >983edf27-03aa-409f-a9e0-7d7652f9f4c4::ERROR::2012-12-02 15:08:08,218::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,219::misc::1071::SamplingMethod::(__call__) Returning last result >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,226::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >983edf27-03aa-409f-a9e0-7d7652f9f4c4::INFO::2012-12-02 15:08:08,227::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::ERROR::2012-12-02 15:08:08,227::task::833::TaskManager.Task::(_setError) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/storage/task.py", line 307, in run > return self.cmd(*self.argslist, **self.argsdict) > File "/usr/share/vdsm/storage/sp.py", line 250, in startSpm > self.masterDomain.acquireHostId(self.id) > File "/usr/share/vdsm/storage/sd.py", line 423, in acquireHostId > self._clusterLock.acquireHostId(hostId, async) > File "/usr/share/vdsm/storage/safelease.py", line 175, in acquireHostId > raise se.AcquireHostIdFailure(self._sdUUID, e) >AcquireHostIdFailure: Cannot acquire host id: ('4e891d8c-9947-468e-88ec-29e49d68e083', SanlockException(11, 'Sanlock lockspace add failure', 'Resource temporarily unavailable')) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,228::task::852::TaskManager.Task::(_run) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Task._run: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 () {} failed - stopping task >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,228::task::1177::TaskManager.Task::(stop) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::stopping in state running (force False) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,228::task::957::TaskManager.Task::(_decref) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::ref 1 aborting True >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,228::task::883::TaskManager.Task::(_runJobs) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::aborting: Task is aborted: 'Cannot acquire host id' - code 661 >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::957::TaskManager.Task::(_decref) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::ref 0 aborting True >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::892::TaskManager.Task::(_doAbort) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Task._doAbort: force False >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state running -> state aborting >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::523::TaskManager.Task::(__state_aborting) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::_aborting: recover policy auto >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state aborting -> state racquiring >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state racquiring -> state recovering >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,229::task::765::TaskManager.Task::(_recover) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::_recover >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::task::772::TaskManager.Task::(_recover) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::running recovery None >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::task::753::TaskManager.Task::(_recoverDone) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::Recover Done: state recovering >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::task::568::TaskManager.Task::(_updateState) Task=`983edf27-03aa-409f-a9e0-7d7652f9f4c4`::moving from state recovering -> state recovered >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,230::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,231::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,231::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >983edf27-03aa-409f-a9e0-7d7652f9f4c4::DEBUG::2012-12-02 15:08:08,231::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-32284::DEBUG::2012-12-02 15:08:08,427::safelease::193::SANLock::(releaseHostId) Host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 released successfully (id: 3) >Thread-32641::DEBUG::2012-12-02 15:08:08,983::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32641::DEBUG::2012-12-02 15:08:08,983::task::568::TaskManager.Task::(_updateState) Task=`e7237ae2-bf56-41b8-be1a-522498b0b569`::moving from state init -> state preparing >Thread-32641::INFO::2012-12-02 15:08:08,983::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='983edf27-03aa-409f-a9e0-7d7652f9f4c4', spUUID=None, options=None) >Thread-32641::DEBUG::2012-12-02 15:08:08,984::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 >Thread-32641::DEBUG::2012-12-02 15:08:08,984::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 661, 'message': 'Cannot acquire host id', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '983edf27-03aa-409f-a9e0-7d7652f9f4c4'} >Thread-32641::INFO::2012-12-02 15:08:08,984::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 661, 'message': 'Cannot acquire host id', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '983edf27-03aa-409f-a9e0-7d7652f9f4c4'}} >Thread-32641::DEBUG::2012-12-02 15:08:08,984::task::1151::TaskManager.Task::(prepare) Task=`e7237ae2-bf56-41b8-be1a-522498b0b569`::finished: {'taskStatus': {'code': 661, 'message': 'Cannot acquire host id', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '983edf27-03aa-409f-a9e0-7d7652f9f4c4'}} >Thread-32641::DEBUG::2012-12-02 15:08:08,984::task::568::TaskManager.Task::(_updateState) Task=`e7237ae2-bf56-41b8-be1a-522498b0b569`::moving from state preparing -> state finished >Thread-32641::DEBUG::2012-12-02 15:08:08,984::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32641::DEBUG::2012-12-02 15:08:08,984::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32641::DEBUG::2012-12-02 15:08:08,984::task::957::TaskManager.Task::(_decref) Task=`e7237ae2-bf56-41b8-be1a-522498b0b569`::ref 0 aborting False >Thread-32642::DEBUG::2012-12-02 15:08:09,008::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32642::DEBUG::2012-12-02 15:08:09,008::task::568::TaskManager.Task::(_updateState) Task=`ad7e078d-4d3d-4729-a885-d455412c59cd`::moving from state init -> state preparing >Thread-32642::INFO::2012-12-02 15:08:09,008::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32642::INFO::2012-12-02 15:08:09,008::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32642::DEBUG::2012-12-02 15:08:09,008::task::1151::TaskManager.Task::(prepare) Task=`ad7e078d-4d3d-4729-a885-d455412c59cd`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32642::DEBUG::2012-12-02 15:08:09,009::task::568::TaskManager.Task::(_updateState) Task=`ad7e078d-4d3d-4729-a885-d455412c59cd`::moving from state preparing -> state finished >Thread-32642::DEBUG::2012-12-02 15:08:09,009::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32642::DEBUG::2012-12-02 15:08:09,009::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32642::DEBUG::2012-12-02 15:08:09,009::task::957::TaskManager.Task::(_decref) Task=`ad7e078d-4d3d-4729-a885-d455412c59cd`::ref 0 aborting False >Thread-32643::DEBUG::2012-12-02 15:08:09,018::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32643::DEBUG::2012-12-02 15:08:09,018::task::568::TaskManager.Task::(_updateState) Task=`dc958643-f331-4cfc-bfb8-9f5e7b15fd31`::moving from state init -> state preparing >Thread-32643::INFO::2012-12-02 15:08:09,018::logUtils::37::dispatcher::(wrapper) Run and protect: clearTask(taskID='983edf27-03aa-409f-a9e0-7d7652f9f4c4', spUUID=None, options=None) >Thread-32643::DEBUG::2012-12-02 15:08:09,018::taskManager::157::TaskManager::(clearTask) Entry. taskID: 983edf27-03aa-409f-a9e0-7d7652f9f4c4 >Thread-32643::DEBUG::2012-12-02 15:08:09,019::taskManager::162::TaskManager::(clearTask) Return. >Thread-32643::INFO::2012-12-02 15:08:09,019::logUtils::39::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-32643::DEBUG::2012-12-02 15:08:09,019::task::1151::TaskManager.Task::(prepare) Task=`dc958643-f331-4cfc-bfb8-9f5e7b15fd31`::finished: None >Thread-32643::DEBUG::2012-12-02 15:08:09,019::task::568::TaskManager.Task::(_updateState) Task=`dc958643-f331-4cfc-bfb8-9f5e7b15fd31`::moving from state preparing -> state finished >Thread-32643::DEBUG::2012-12-02 15:08:09,019::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32643::DEBUG::2012-12-02 15:08:09,019::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32643::DEBUG::2012-12-02 15:08:09,019::task::957::TaskManager.Task::(_decref) Task=`dc958643-f331-4cfc-bfb8-9f5e7b15fd31`::ref 0 aborting False >Thread-32462::INFO::2012-12-02 15:08:17,688::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-32462::DEBUG::2012-12-02 15:08:17,688::safelease::178::SANLock::(acquireHostId) Host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully acquired (id: 3) >Thread-32644::DEBUG::2012-12-02 15:08:19,138::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32644::DEBUG::2012-12-02 15:08:19,138::task::568::TaskManager.Task::(_updateState) Task=`6ce5d797-0385-4c67-90a7-cc122b415b81`::moving from state init -> state preparing >Thread-32644::INFO::2012-12-02 15:08:19,138::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32644::INFO::2012-12-02 15:08:19,139::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32644::DEBUG::2012-12-02 15:08:19,139::task::1151::TaskManager.Task::(prepare) Task=`6ce5d797-0385-4c67-90a7-cc122b415b81`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32644::DEBUG::2012-12-02 15:08:19,139::task::568::TaskManager.Task::(_updateState) Task=`6ce5d797-0385-4c67-90a7-cc122b415b81`::moving from state preparing -> state finished >Thread-32644::DEBUG::2012-12-02 15:08:19,139::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32644::DEBUG::2012-12-02 15:08:19,139::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32644::DEBUG::2012-12-02 15:08:19,139::task::957::TaskManager.Task::(_decref) Task=`6ce5d797-0385-4c67-90a7-cc122b415b81`::ref 0 aborting False >Thread-32645::DEBUG::2012-12-02 15:08:19,148::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32645::DEBUG::2012-12-02 15:08:19,148::task::568::TaskManager.Task::(_updateState) Task=`e9629474-0fc6-4ad0-a587-2fef19708822`::moving from state init -> state preparing >Thread-32645::INFO::2012-12-02 15:08:19,149::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-32645::DEBUG::2012-12-02 15:08:19,149::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-32645::DEBUG::2012-12-02 15:08:19,149::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-32645::INFO::2012-12-02 15:08:19,149::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-32645::DEBUG::2012-12-02 15:08:19,149::task::1151::TaskManager.Task::(prepare) Task=`e9629474-0fc6-4ad0-a587-2fef19708822`::finished: {'allTasksStatus': {}} >Thread-32645::DEBUG::2012-12-02 15:08:19,149::task::568::TaskManager.Task::(_updateState) Task=`e9629474-0fc6-4ad0-a587-2fef19708822`::moving from state preparing -> state finished >Thread-32645::DEBUG::2012-12-02 15:08:19,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32645::DEBUG::2012-12-02 15:08:19,149::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32645::DEBUG::2012-12-02 15:08:19,150::task::957::TaskManager.Task::(_decref) Task=`e9629474-0fc6-4ad0-a587-2fef19708822`::ref 0 aborting False >Thread-32646::DEBUG::2012-12-02 15:08:19,160::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32646::DEBUG::2012-12-02 15:08:19,160::task::568::TaskManager.Task::(_updateState) Task=`190c770e-aa46-466f-8fdc-01880c9b0ae5`::moving from state init -> state preparing >Thread-32646::INFO::2012-12-02 15:08:19,160::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32646::INFO::2012-12-02 15:08:19,160::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32646::DEBUG::2012-12-02 15:08:19,160::task::1151::TaskManager.Task::(prepare) Task=`190c770e-aa46-466f-8fdc-01880c9b0ae5`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-32646::DEBUG::2012-12-02 15:08:19,161::task::568::TaskManager.Task::(_updateState) Task=`190c770e-aa46-466f-8fdc-01880c9b0ae5`::moving from state preparing -> state finished >Thread-32646::DEBUG::2012-12-02 15:08:19,161::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32646::DEBUG::2012-12-02 15:08:19,161::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32646::DEBUG::2012-12-02 15:08:19,161::task::957::TaskManager.Task::(_decref) Task=`190c770e-aa46-466f-8fdc-01880c9b0ae5`::ref 0 aborting False >Thread-32647::DEBUG::2012-12-02 15:08:19,192::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32647::DEBUG::2012-12-02 15:08:19,192::task::568::TaskManager.Task::(_updateState) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::moving from state init -> state preparing >Thread-32647::INFO::2012-12-02 15:08:19,193::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', prevID=-1, prevLVER='-1', recoveryMode=None, scsiFencing='false', maxHostID=250, domVersion='3', options=None) >Thread-32647::DEBUG::2012-12-02 15:08:19,193::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`210c03f8-224b-41ba-91e0-ef66c8654122`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32647::DEBUG::2012-12-02 15:08:19,193::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-32647::DEBUG::2012-12-02 15:08:19,193::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-32647::DEBUG::2012-12-02 15:08:19,193::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`210c03f8-224b-41ba-91e0-ef66c8654122`::Granted request >Thread-32647::DEBUG::2012-12-02 15:08:19,194::task::794::TaskManager.Task::(resourceAcquired) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (exclusive) >Thread-32647::DEBUG::2012-12-02 15:08:19,194::task::957::TaskManager.Task::(_decref) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::ref 1 aborting False >Thread-32647::INFO::2012-12-02 15:08:19,194::logUtils::39::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-32647::DEBUG::2012-12-02 15:08:19,194::task::1146::TaskManager.Task::(prepare) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::Prepare: 1 jobs exist, move to acquiring >Thread-32647::DEBUG::2012-12-02 15:08:19,194::task::568::TaskManager.Task::(_updateState) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::moving from state preparing -> state acquiring >Thread-32647::DEBUG::2012-12-02 15:08:19,194::task::568::TaskManager.Task::(_updateState) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::moving from state acquiring -> state queued >Thread-32647::DEBUG::2012-12-02 15:08:19,194::taskManager::50::TaskManager::(_queueTask) queuing task: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32647::DEBUG::2012-12-02 15:08:19,195::taskManager::56::TaskManager::(_queueTask) task queued: e143b971-cde5-4f18-92eb-a3750573504d >Thread-6::DEBUG::2012-12-02 15:08:19,195::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-32647::DEBUG::2012-12-02 15:08:19,195::task::1148::TaskManager.Task::(prepare) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::returning >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,195::threadPool::212::Misc.ThreadPool.WorkerThread::(run) Task: e143b971-cde5-4f18-92eb-a3750573504d running: <bound method Task.commit of <storage.task.Task instance at 0x7f17b039cd40>> with: None >Thread-32647::DEBUG::2012-12-02 15:08:19,195::task::957::TaskManager.Task::(_decref) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::ref 0 aborting False >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,196::task::1159::TaskManager.Task::(commit) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::committing task: e143b971-cde5-4f18-92eb-a3750573504d >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,196::task::568::TaskManager.Task::(_updateState) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::moving from state queued -> state running >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,196::task::872::TaskManager.Task::(_runJobs) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x22ff050>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,197::task::304::TaskManager.Task::(run) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x22ff050>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) callback None >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,197::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,197::misc::1063::SamplingMethod::(__call__) Got in to sampling method >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,206::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,207::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,207::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,413::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:19,414::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,415::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::ERROR::2012-12-02 15:08:19,444::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,444::misc::1071::SamplingMethod::(__call__) Returning last result >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:19,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >e143b971-cde5-4f18-92eb-a3750573504d::INFO::2012-12-02 15:08:19,451::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >Thread-32707::DEBUG::2012-12-02 15:08:20,205::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32707::DEBUG::2012-12-02 15:08:20,205::task::568::TaskManager.Task::(_updateState) Task=`b0c851aa-6d99-41f6-8a43-e7cdfc1934e8`::moving from state init -> state preparing >Thread-32707::INFO::2012-12-02 15:08:20,205::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32707::DEBUG::2012-12-02 15:08:20,206::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32707::DEBUG::2012-12-02 15:08:20,206::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32707::INFO::2012-12-02 15:08:20,206::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32707::DEBUG::2012-12-02 15:08:20,206::task::1151::TaskManager.Task::(prepare) Task=`b0c851aa-6d99-41f6-8a43-e7cdfc1934e8`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32707::DEBUG::2012-12-02 15:08:20,206::task::568::TaskManager.Task::(_updateState) Task=`b0c851aa-6d99-41f6-8a43-e7cdfc1934e8`::moving from state preparing -> state finished >Thread-32707::DEBUG::2012-12-02 15:08:20,206::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32707::DEBUG::2012-12-02 15:08:20,206::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32707::DEBUG::2012-12-02 15:08:20,207::task::957::TaskManager.Task::(_decref) Task=`b0c851aa-6d99-41f6-8a43-e7cdfc1934e8`::ref 0 aborting False >Thread-32708::DEBUG::2012-12-02 15:08:21,215::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32708::DEBUG::2012-12-02 15:08:21,215::task::568::TaskManager.Task::(_updateState) Task=`d3aae123-a529-4682-bb59-1d60c0b66323`::moving from state init -> state preparing >Thread-32708::INFO::2012-12-02 15:08:21,215::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32708::DEBUG::2012-12-02 15:08:21,216::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32708::DEBUG::2012-12-02 15:08:21,216::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32708::INFO::2012-12-02 15:08:21,216::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32708::DEBUG::2012-12-02 15:08:21,216::task::1151::TaskManager.Task::(prepare) Task=`d3aae123-a529-4682-bb59-1d60c0b66323`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32708::DEBUG::2012-12-02 15:08:21,216::task::568::TaskManager.Task::(_updateState) Task=`d3aae123-a529-4682-bb59-1d60c0b66323`::moving from state preparing -> state finished >Thread-32708::DEBUG::2012-12-02 15:08:21,216::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32708::DEBUG::2012-12-02 15:08:21,216::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32708::DEBUG::2012-12-02 15:08:21,216::task::957::TaskManager.Task::(_decref) Task=`d3aae123-a529-4682-bb59-1d60c0b66323`::ref 0 aborting False >Thread-32709::DEBUG::2012-12-02 15:08:22,225::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32709::DEBUG::2012-12-02 15:08:22,225::task::568::TaskManager.Task::(_updateState) Task=`5339ef80-952b-42ef-9f88-7eae6bb3b40a`::moving from state init -> state preparing >Thread-32709::INFO::2012-12-02 15:08:22,225::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32709::DEBUG::2012-12-02 15:08:22,225::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32709::DEBUG::2012-12-02 15:08:22,225::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32709::INFO::2012-12-02 15:08:22,226::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32709::DEBUG::2012-12-02 15:08:22,226::task::1151::TaskManager.Task::(prepare) Task=`5339ef80-952b-42ef-9f88-7eae6bb3b40a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32709::DEBUG::2012-12-02 15:08:22,226::task::568::TaskManager.Task::(_updateState) Task=`5339ef80-952b-42ef-9f88-7eae6bb3b40a`::moving from state preparing -> state finished >Thread-32709::DEBUG::2012-12-02 15:08:22,226::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32709::DEBUG::2012-12-02 15:08:22,226::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32709::DEBUG::2012-12-02 15:08:22,226::task::957::TaskManager.Task::(_decref) Task=`5339ef80-952b-42ef-9f88-7eae6bb3b40a`::ref 0 aborting False >Thread-32710::DEBUG::2012-12-02 15:08:23,235::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32710::DEBUG::2012-12-02 15:08:23,235::task::568::TaskManager.Task::(_updateState) Task=`850ba615-c084-4c7d-960f-baec70bece72`::moving from state init -> state preparing >Thread-32710::INFO::2012-12-02 15:08:23,235::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32710::DEBUG::2012-12-02 15:08:23,235::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32710::DEBUG::2012-12-02 15:08:23,235::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32710::INFO::2012-12-02 15:08:23,236::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32710::DEBUG::2012-12-02 15:08:23,236::task::1151::TaskManager.Task::(prepare) Task=`850ba615-c084-4c7d-960f-baec70bece72`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32710::DEBUG::2012-12-02 15:08:23,236::task::568::TaskManager.Task::(_updateState) Task=`850ba615-c084-4c7d-960f-baec70bece72`::moving from state preparing -> state finished >Thread-32710::DEBUG::2012-12-02 15:08:23,236::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32710::DEBUG::2012-12-02 15:08:23,236::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32710::DEBUG::2012-12-02 15:08:23,236::task::957::TaskManager.Task::(_decref) Task=`850ba615-c084-4c7d-960f-baec70bece72`::ref 0 aborting False >Thread-32711::DEBUG::2012-12-02 15:08:24,244::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32711::DEBUG::2012-12-02 15:08:24,244::task::568::TaskManager.Task::(_updateState) Task=`c7a5a4a5-5d22-4a4e-902f-2a2dbd72241b`::moving from state init -> state preparing >Thread-32711::INFO::2012-12-02 15:08:24,245::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32711::DEBUG::2012-12-02 15:08:24,245::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32711::DEBUG::2012-12-02 15:08:24,245::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32711::INFO::2012-12-02 15:08:24,245::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32711::DEBUG::2012-12-02 15:08:24,245::task::1151::TaskManager.Task::(prepare) Task=`c7a5a4a5-5d22-4a4e-902f-2a2dbd72241b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32711::DEBUG::2012-12-02 15:08:24,245::task::568::TaskManager.Task::(_updateState) Task=`c7a5a4a5-5d22-4a4e-902f-2a2dbd72241b`::moving from state preparing -> state finished >Thread-32711::DEBUG::2012-12-02 15:08:24,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32711::DEBUG::2012-12-02 15:08:24,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32711::DEBUG::2012-12-02 15:08:24,245::task::957::TaskManager.Task::(_decref) Task=`c7a5a4a5-5d22-4a4e-902f-2a2dbd72241b`::ref 0 aborting False >Thread-32712::DEBUG::2012-12-02 15:08:25,252::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32712::DEBUG::2012-12-02 15:08:25,252::task::568::TaskManager.Task::(_updateState) Task=`374dd472-2f9a-4f76-9a9a-5d2a1029a184`::moving from state init -> state preparing >Thread-32712::INFO::2012-12-02 15:08:25,252::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32712::DEBUG::2012-12-02 15:08:25,252::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32712::DEBUG::2012-12-02 15:08:25,252::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32712::INFO::2012-12-02 15:08:25,252::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32712::DEBUG::2012-12-02 15:08:25,252::task::1151::TaskManager.Task::(prepare) Task=`374dd472-2f9a-4f76-9a9a-5d2a1029a184`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32712::DEBUG::2012-12-02 15:08:25,253::task::568::TaskManager.Task::(_updateState) Task=`374dd472-2f9a-4f76-9a9a-5d2a1029a184`::moving from state preparing -> state finished >Thread-32712::DEBUG::2012-12-02 15:08:25,253::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32712::DEBUG::2012-12-02 15:08:25,253::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32712::DEBUG::2012-12-02 15:08:25,253::task::957::TaskManager.Task::(_decref) Task=`374dd472-2f9a-4f76-9a9a-5d2a1029a184`::ref 0 aborting False >Thread-32713::DEBUG::2012-12-02 15:08:26,261::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32713::DEBUG::2012-12-02 15:08:26,261::task::568::TaskManager.Task::(_updateState) Task=`056c6958-6ef0-45d0-876d-2611c1967f34`::moving from state init -> state preparing >Thread-32713::INFO::2012-12-02 15:08:26,261::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32713::DEBUG::2012-12-02 15:08:26,261::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32713::DEBUG::2012-12-02 15:08:26,261::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32713::INFO::2012-12-02 15:08:26,261::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32713::DEBUG::2012-12-02 15:08:26,262::task::1151::TaskManager.Task::(prepare) Task=`056c6958-6ef0-45d0-876d-2611c1967f34`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32713::DEBUG::2012-12-02 15:08:26,262::task::568::TaskManager.Task::(_updateState) Task=`056c6958-6ef0-45d0-876d-2611c1967f34`::moving from state preparing -> state finished >Thread-32713::DEBUG::2012-12-02 15:08:26,262::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32713::DEBUG::2012-12-02 15:08:26,262::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32713::DEBUG::2012-12-02 15:08:26,262::task::957::TaskManager.Task::(_decref) Task=`056c6958-6ef0-45d0-876d-2611c1967f34`::ref 0 aborting False >Thread-32714::DEBUG::2012-12-02 15:08:27,270::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32714::DEBUG::2012-12-02 15:08:27,270::task::568::TaskManager.Task::(_updateState) Task=`1e8c7747-6342-444b-86ab-5777aafcb8a7`::moving from state init -> state preparing >Thread-32714::INFO::2012-12-02 15:08:27,271::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32714::DEBUG::2012-12-02 15:08:27,271::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32714::DEBUG::2012-12-02 15:08:27,271::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32714::INFO::2012-12-02 15:08:27,271::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32714::DEBUG::2012-12-02 15:08:27,271::task::1151::TaskManager.Task::(prepare) Task=`1e8c7747-6342-444b-86ab-5777aafcb8a7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32714::DEBUG::2012-12-02 15:08:27,271::task::568::TaskManager.Task::(_updateState) Task=`1e8c7747-6342-444b-86ab-5777aafcb8a7`::moving from state preparing -> state finished >Thread-32714::DEBUG::2012-12-02 15:08:27,271::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32714::DEBUG::2012-12-02 15:08:27,272::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32714::DEBUG::2012-12-02 15:08:27,272::task::957::TaskManager.Task::(_decref) Task=`1e8c7747-6342-444b-86ab-5777aafcb8a7`::ref 0 aborting False >Thread-32715::DEBUG::2012-12-02 15:08:28,280::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32715::DEBUG::2012-12-02 15:08:28,280::task::568::TaskManager.Task::(_updateState) Task=`dbb2ebd3-b163-49d4-a408-c61414de5ec7`::moving from state init -> state preparing >Thread-32715::INFO::2012-12-02 15:08:28,280::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32715::DEBUG::2012-12-02 15:08:28,280::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32715::DEBUG::2012-12-02 15:08:28,280::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32715::INFO::2012-12-02 15:08:28,281::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32715::DEBUG::2012-12-02 15:08:28,281::task::1151::TaskManager.Task::(prepare) Task=`dbb2ebd3-b163-49d4-a408-c61414de5ec7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32715::DEBUG::2012-12-02 15:08:28,281::task::568::TaskManager.Task::(_updateState) Task=`dbb2ebd3-b163-49d4-a408-c61414de5ec7`::moving from state preparing -> state finished >Thread-32715::DEBUG::2012-12-02 15:08:28,281::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32715::DEBUG::2012-12-02 15:08:28,281::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32715::DEBUG::2012-12-02 15:08:28,281::task::957::TaskManager.Task::(_decref) Task=`dbb2ebd3-b163-49d4-a408-c61414de5ec7`::ref 0 aborting False >Thread-32716::DEBUG::2012-12-02 15:08:29,289::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32716::DEBUG::2012-12-02 15:08:29,290::task::568::TaskManager.Task::(_updateState) Task=`f51d40ee-60d2-4bd1-8210-191ccd3348ec`::moving from state init -> state preparing >Thread-32716::INFO::2012-12-02 15:08:29,290::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32716::DEBUG::2012-12-02 15:08:29,290::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32716::DEBUG::2012-12-02 15:08:29,290::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32716::INFO::2012-12-02 15:08:29,290::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32716::DEBUG::2012-12-02 15:08:29,290::task::1151::TaskManager.Task::(prepare) Task=`f51d40ee-60d2-4bd1-8210-191ccd3348ec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32716::DEBUG::2012-12-02 15:08:29,290::task::568::TaskManager.Task::(_updateState) Task=`f51d40ee-60d2-4bd1-8210-191ccd3348ec`::moving from state preparing -> state finished >Thread-32716::DEBUG::2012-12-02 15:08:29,291::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32716::DEBUG::2012-12-02 15:08:29,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32716::DEBUG::2012-12-02 15:08:29,291::task::957::TaskManager.Task::(_decref) Task=`f51d40ee-60d2-4bd1-8210-191ccd3348ec`::ref 0 aborting False >Thread-32717::DEBUG::2012-12-02 15:08:30,300::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32717::DEBUG::2012-12-02 15:08:30,300::task::568::TaskManager.Task::(_updateState) Task=`5acce01b-c83a-41a7-a04c-cda49582f930`::moving from state init -> state preparing >Thread-32717::INFO::2012-12-02 15:08:30,300::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32717::DEBUG::2012-12-02 15:08:30,300::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32717::DEBUG::2012-12-02 15:08:30,300::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32717::INFO::2012-12-02 15:08:30,301::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32717::DEBUG::2012-12-02 15:08:30,301::task::1151::TaskManager.Task::(prepare) Task=`5acce01b-c83a-41a7-a04c-cda49582f930`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32717::DEBUG::2012-12-02 15:08:30,301::task::568::TaskManager.Task::(_updateState) Task=`5acce01b-c83a-41a7-a04c-cda49582f930`::moving from state preparing -> state finished >Thread-32717::DEBUG::2012-12-02 15:08:30,301::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32717::DEBUG::2012-12-02 15:08:30,301::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32717::DEBUG::2012-12-02 15:08:30,301::task::957::TaskManager.Task::(_decref) Task=`5acce01b-c83a-41a7-a04c-cda49582f930`::ref 0 aborting False >Thread-32718::DEBUG::2012-12-02 15:08:31,309::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32718::DEBUG::2012-12-02 15:08:31,309::task::568::TaskManager.Task::(_updateState) Task=`66158a23-f60f-4c97-a0e6-6ca086c0e0ab`::moving from state init -> state preparing >Thread-32718::INFO::2012-12-02 15:08:31,309::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32718::DEBUG::2012-12-02 15:08:31,309::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32718::DEBUG::2012-12-02 15:08:31,309::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32718::INFO::2012-12-02 15:08:31,309::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32718::DEBUG::2012-12-02 15:08:31,309::task::1151::TaskManager.Task::(prepare) Task=`66158a23-f60f-4c97-a0e6-6ca086c0e0ab`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32718::DEBUG::2012-12-02 15:08:31,310::task::568::TaskManager.Task::(_updateState) Task=`66158a23-f60f-4c97-a0e6-6ca086c0e0ab`::moving from state preparing -> state finished >Thread-32718::DEBUG::2012-12-02 15:08:31,310::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32718::DEBUG::2012-12-02 15:08:31,310::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32718::DEBUG::2012-12-02 15:08:31,310::task::957::TaskManager.Task::(_decref) Task=`66158a23-f60f-4c97-a0e6-6ca086c0e0ab`::ref 0 aborting False >Thread-32719::DEBUG::2012-12-02 15:08:32,317::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32719::DEBUG::2012-12-02 15:08:32,317::task::568::TaskManager.Task::(_updateState) Task=`17bc6148-7e79-44fd-95b4-a25da19da1c5`::moving from state init -> state preparing >Thread-32719::INFO::2012-12-02 15:08:32,317::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32719::DEBUG::2012-12-02 15:08:32,318::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32719::DEBUG::2012-12-02 15:08:32,318::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32719::INFO::2012-12-02 15:08:32,318::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32719::DEBUG::2012-12-02 15:08:32,318::task::1151::TaskManager.Task::(prepare) Task=`17bc6148-7e79-44fd-95b4-a25da19da1c5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32719::DEBUG::2012-12-02 15:08:32,318::task::568::TaskManager.Task::(_updateState) Task=`17bc6148-7e79-44fd-95b4-a25da19da1c5`::moving from state preparing -> state finished >Thread-32719::DEBUG::2012-12-02 15:08:32,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32719::DEBUG::2012-12-02 15:08:32,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32719::DEBUG::2012-12-02 15:08:32,318::task::957::TaskManager.Task::(_decref) Task=`17bc6148-7e79-44fd-95b4-a25da19da1c5`::ref 0 aborting False >Thread-32720::DEBUG::2012-12-02 15:08:33,327::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32720::DEBUG::2012-12-02 15:08:33,328::task::568::TaskManager.Task::(_updateState) Task=`3f3cf42b-ed14-412f-9abf-6870d60a60cd`::moving from state init -> state preparing >Thread-32720::INFO::2012-12-02 15:08:33,328::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32720::DEBUG::2012-12-02 15:08:33,328::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32720::DEBUG::2012-12-02 15:08:33,328::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32720::INFO::2012-12-02 15:08:33,328::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32720::DEBUG::2012-12-02 15:08:33,328::task::1151::TaskManager.Task::(prepare) Task=`3f3cf42b-ed14-412f-9abf-6870d60a60cd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32720::DEBUG::2012-12-02 15:08:33,328::task::568::TaskManager.Task::(_updateState) Task=`3f3cf42b-ed14-412f-9abf-6870d60a60cd`::moving from state preparing -> state finished >Thread-32720::DEBUG::2012-12-02 15:08:33,329::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32720::DEBUG::2012-12-02 15:08:33,329::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32720::DEBUG::2012-12-02 15:08:33,329::task::957::TaskManager.Task::(_decref) Task=`3f3cf42b-ed14-412f-9abf-6870d60a60cd`::ref 0 aborting False >Thread-32721::DEBUG::2012-12-02 15:08:34,338::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32721::DEBUG::2012-12-02 15:08:34,338::task::568::TaskManager.Task::(_updateState) Task=`2bd3f286-d253-4e6a-9810-0bbec2eeaaf7`::moving from state init -> state preparing >Thread-32721::INFO::2012-12-02 15:08:34,339::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32721::DEBUG::2012-12-02 15:08:34,339::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32721::DEBUG::2012-12-02 15:08:34,339::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32721::INFO::2012-12-02 15:08:34,339::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32721::DEBUG::2012-12-02 15:08:34,339::task::1151::TaskManager.Task::(prepare) Task=`2bd3f286-d253-4e6a-9810-0bbec2eeaaf7`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32721::DEBUG::2012-12-02 15:08:34,339::task::568::TaskManager.Task::(_updateState) Task=`2bd3f286-d253-4e6a-9810-0bbec2eeaaf7`::moving from state preparing -> state finished >Thread-32721::DEBUG::2012-12-02 15:08:34,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32721::DEBUG::2012-12-02 15:08:34,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32721::DEBUG::2012-12-02 15:08:34,340::task::957::TaskManager.Task::(_decref) Task=`2bd3f286-d253-4e6a-9810-0bbec2eeaaf7`::ref 0 aborting False >Thread-32722::DEBUG::2012-12-02 15:08:35,348::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32722::DEBUG::2012-12-02 15:08:35,348::task::568::TaskManager.Task::(_updateState) Task=`e084c02b-3490-436d-8a7c-2682f7e1e072`::moving from state init -> state preparing >Thread-32722::INFO::2012-12-02 15:08:35,348::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32722::DEBUG::2012-12-02 15:08:35,348::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32722::DEBUG::2012-12-02 15:08:35,348::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32722::INFO::2012-12-02 15:08:35,348::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32722::DEBUG::2012-12-02 15:08:35,349::task::1151::TaskManager.Task::(prepare) Task=`e084c02b-3490-436d-8a7c-2682f7e1e072`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32722::DEBUG::2012-12-02 15:08:35,349::task::568::TaskManager.Task::(_updateState) Task=`e084c02b-3490-436d-8a7c-2682f7e1e072`::moving from state preparing -> state finished >Thread-32722::DEBUG::2012-12-02 15:08:35,349::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32722::DEBUG::2012-12-02 15:08:35,349::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32722::DEBUG::2012-12-02 15:08:35,349::task::957::TaskManager.Task::(_decref) Task=`e084c02b-3490-436d-8a7c-2682f7e1e072`::ref 0 aborting False >Thread-32723::DEBUG::2012-12-02 15:08:36,357::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32723::DEBUG::2012-12-02 15:08:36,358::task::568::TaskManager.Task::(_updateState) Task=`63c15a01-286e-4aac-ba12-f406e20eb759`::moving from state init -> state preparing >Thread-32723::INFO::2012-12-02 15:08:36,358::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32723::DEBUG::2012-12-02 15:08:36,358::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32723::DEBUG::2012-12-02 15:08:36,358::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32723::INFO::2012-12-02 15:08:36,358::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32723::DEBUG::2012-12-02 15:08:36,358::task::1151::TaskManager.Task::(prepare) Task=`63c15a01-286e-4aac-ba12-f406e20eb759`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32723::DEBUG::2012-12-02 15:08:36,358::task::568::TaskManager.Task::(_updateState) Task=`63c15a01-286e-4aac-ba12-f406e20eb759`::moving from state preparing -> state finished >Thread-32723::DEBUG::2012-12-02 15:08:36,358::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32723::DEBUG::2012-12-02 15:08:36,359::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32723::DEBUG::2012-12-02 15:08:36,359::task::957::TaskManager.Task::(_decref) Task=`63c15a01-286e-4aac-ba12-f406e20eb759`::ref 0 aborting False >Thread-32724::DEBUG::2012-12-02 15:08:37,367::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32724::DEBUG::2012-12-02 15:08:37,367::task::568::TaskManager.Task::(_updateState) Task=`bb6239e4-d3fd-4d0f-b8e7-eada4f717a6f`::moving from state init -> state preparing >Thread-32724::INFO::2012-12-02 15:08:37,367::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32724::DEBUG::2012-12-02 15:08:37,367::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32724::DEBUG::2012-12-02 15:08:37,367::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32724::INFO::2012-12-02 15:08:37,368::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32724::DEBUG::2012-12-02 15:08:37,368::task::1151::TaskManager.Task::(prepare) Task=`bb6239e4-d3fd-4d0f-b8e7-eada4f717a6f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32724::DEBUG::2012-12-02 15:08:37,368::task::568::TaskManager.Task::(_updateState) Task=`bb6239e4-d3fd-4d0f-b8e7-eada4f717a6f`::moving from state preparing -> state finished >Thread-32724::DEBUG::2012-12-02 15:08:37,368::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32724::DEBUG::2012-12-02 15:08:37,368::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32724::DEBUG::2012-12-02 15:08:37,368::task::957::TaskManager.Task::(_decref) Task=`bb6239e4-d3fd-4d0f-b8e7-eada4f717a6f`::ref 0 aborting False >Thread-32725::DEBUG::2012-12-02 15:08:38,376::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32725::DEBUG::2012-12-02 15:08:38,377::task::568::TaskManager.Task::(_updateState) Task=`49f940e3-14ae-4381-8350-65b9251a3e09`::moving from state init -> state preparing >Thread-32725::INFO::2012-12-02 15:08:38,377::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32725::DEBUG::2012-12-02 15:08:38,377::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32725::DEBUG::2012-12-02 15:08:38,377::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32725::INFO::2012-12-02 15:08:38,377::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32725::DEBUG::2012-12-02 15:08:38,378::task::1151::TaskManager.Task::(prepare) Task=`49f940e3-14ae-4381-8350-65b9251a3e09`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32725::DEBUG::2012-12-02 15:08:38,378::task::568::TaskManager.Task::(_updateState) Task=`49f940e3-14ae-4381-8350-65b9251a3e09`::moving from state preparing -> state finished >Thread-32725::DEBUG::2012-12-02 15:08:38,378::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32725::DEBUG::2012-12-02 15:08:38,378::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32725::DEBUG::2012-12-02 15:08:38,378::task::957::TaskManager.Task::(_decref) Task=`49f940e3-14ae-4381-8350-65b9251a3e09`::ref 0 aborting False >Thread-32726::DEBUG::2012-12-02 15:08:39,386::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32726::DEBUG::2012-12-02 15:08:39,386::task::568::TaskManager.Task::(_updateState) Task=`ded67c7a-689a-40fd-91cc-40001670d9c5`::moving from state init -> state preparing >Thread-32726::INFO::2012-12-02 15:08:39,386::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32726::DEBUG::2012-12-02 15:08:39,386::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32726::DEBUG::2012-12-02 15:08:39,386::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32726::INFO::2012-12-02 15:08:39,387::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32726::DEBUG::2012-12-02 15:08:39,387::task::1151::TaskManager.Task::(prepare) Task=`ded67c7a-689a-40fd-91cc-40001670d9c5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32726::DEBUG::2012-12-02 15:08:39,387::task::568::TaskManager.Task::(_updateState) Task=`ded67c7a-689a-40fd-91cc-40001670d9c5`::moving from state preparing -> state finished >Thread-32726::DEBUG::2012-12-02 15:08:39,387::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32726::DEBUG::2012-12-02 15:08:39,387::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32726::DEBUG::2012-12-02 15:08:39,387::task::957::TaskManager.Task::(_decref) Task=`ded67c7a-689a-40fd-91cc-40001670d9c5`::ref 0 aborting False >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,452::safelease::178::SANLock::(acquireHostId) Host id for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully acquired (id: 3) >e143b971-cde5-4f18-92eb-a3750573504d::INFO::2012-12-02 15:08:39,453::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 4e891d8c-9947-468e-88ec-29e49d68e083 (id: 3) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,809::safelease::235::SANLock::(acquire) Cluster lock for domain 4e891d8c-9947-468e-88ec-29e49d68e083 successfully acquired (id: 3) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,810::sp::256::Storage.StoragePool::(startSpm) spm lock acquired successfully >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,814::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=85a2705fa8446fb94041fd3fe46cc7b7e35e784f'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,815::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,815::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,815::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b8f426a604115978d1ec7e997fd36ba2e68b6393'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,853::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,853::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`190368a8-a3d0-4ce8-b7fc-1b8d7ba15f80`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,854::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,854::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,854::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`190368a8-a3d0-4ce8-b7fc-1b8d7ba15f80`::Granted request >e143b971-cde5-4f18-92eb-a3750573504d::INFO::2012-12-02 15:08:39,854::sp::403::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `4e891d8c-9947-468e-88ec-29e49d68e083` >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,855::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.4e891d8c-9947-468e-88ec-29e49d68e083`ReqID=`b7a0f6af-3d50-42f9-94d8-4c1b55fa7775`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,855::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.4e891d8c-9947-468e-88ec-29e49d68e083' for lock type 'exclusive' >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,855::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.4e891d8c-9947-468e-88ec-29e49d68e083' is free. Now locking as 'exclusive' (1 active user) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,855::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.4e891d8c-9947-468e-88ec-29e49d68e083`ReqID=`b7a0f6af-3d50-42f9-94d8-4c1b55fa7775`::Granted request >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,855::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.4e891d8c-9947-468e-88ec-29e49d68e083' >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,856::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.4e891d8c-9947-468e-88ec-29e49d68e083' (0 active users) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,856::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.4e891d8c-9947-468e-88ec-29e49d68e083' is free, finding out if anyone is waiting for it. >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,856::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.4e891d8c-9947-468e-88ec-29e49d68e083', Clearing records. >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,856::sp::407::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,856::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:39,857::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,064::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:40,065::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,066::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,100::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,100::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,112::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b8f426a604115978d1ec7e997fd36ba2e68b6393'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,115::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:40,115::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_imageNS already registered >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:40,115::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_volumeNS already registered >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,115::sp::414::Storage.StoragePool::(_upgradePool) Registering with state change event >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,115::sp::416::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-32786::DEBUG::2012-12-02 15:08:40,116::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,116::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-32786::DEBUG::2012-12-02 15:08:40,117::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,117::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,121::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,122::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-32786::DEBUG::2012-12-02 15:08:40,321::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32786::WARNING::2012-12-02 15:08:40,323::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32786::DEBUG::2012-12-02 15:08:40,323::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:40,363::sd::583::Storage.StorageDomain::(createMasterTree) vms dir not found, creating (/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083/master/vms) >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:40,376::sd::588::Storage.StorageDomain::(createMasterTree) tasks dir not found, creating (/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083/master/tasks) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,387::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,387::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32846::DEBUG::2012-12-02 15:08:40,395::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32846::DEBUG::2012-12-02 15:08:40,396::task::568::TaskManager.Task::(_updateState) Task=`0a37ad21-78eb-4454-bf2f-19234d23ff5b`::moving from state init -> state preparing >Thread-32846::INFO::2012-12-02 15:08:40,396::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32846::DEBUG::2012-12-02 15:08:40,396::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32846::DEBUG::2012-12-02 15:08:40,396::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32846::INFO::2012-12-02 15:08:40,396::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32846::DEBUG::2012-12-02 15:08:40,397::task::1151::TaskManager.Task::(prepare) Task=`0a37ad21-78eb-4454-bf2f-19234d23ff5b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32846::DEBUG::2012-12-02 15:08:40,397::task::568::TaskManager.Task::(_updateState) Task=`0a37ad21-78eb-4454-bf2f-19234d23ff5b`::moving from state preparing -> state finished >Thread-32846::DEBUG::2012-12-02 15:08:40,397::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32846::DEBUG::2012-12-02 15:08:40,397::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32846::DEBUG::2012-12-02 15:08:40,397::task::957::TaskManager.Task::(_decref) Task=`0a37ad21-78eb-4454-bf2f-19234d23ff5b`::ref 0 aborting False >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,593::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >e143b971-cde5-4f18-92eb-a3750573504d::WARNING::2012-12-02 15:08:40,595::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,595::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >e143b971-cde5-4f18-92eb-a3750573504d::ERROR::2012-12-02 15:08:40,624::sp::277::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 275, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper > return f(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1568, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,629::sp::305::Storage.StoragePool::(startSpm) ended. >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,629::task::568::TaskManager.Task::(_updateState) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::moving from state running -> state finished >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,630::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,630::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,630::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,630::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,631::task::880::TaskManager.Task::(_runJobs) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::Task.run: exit - success: result >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,631::task::957::TaskManager.Task::(_decref) Task=`e143b971-cde5-4f18-92eb-a3750573504d`::ref 0 aborting False >e143b971-cde5-4f18-92eb-a3750573504d::DEBUG::2012-12-02 15:08:40,631::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-32906::DEBUG::2012-12-02 15:08:41,405::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32906::DEBUG::2012-12-02 15:08:41,405::task::568::TaskManager.Task::(_updateState) Task=`bf952fd7-2614-4507-b2c0-e7039ed2a019`::moving from state init -> state preparing >Thread-32906::INFO::2012-12-02 15:08:41,406::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32906::DEBUG::2012-12-02 15:08:41,406::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32906::DEBUG::2012-12-02 15:08:41,406::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'} >Thread-32906::INFO::2012-12-02 15:08:41,406::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32906::DEBUG::2012-12-02 15:08:41,406::task::1151::TaskManager.Task::(prepare) Task=`bf952fd7-2614-4507-b2c0-e7039ed2a019`::finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'e143b971-cde5-4f18-92eb-a3750573504d'}} >Thread-32906::DEBUG::2012-12-02 15:08:41,406::task::568::TaskManager.Task::(_updateState) Task=`bf952fd7-2614-4507-b2c0-e7039ed2a019`::moving from state preparing -> state finished >Thread-32906::DEBUG::2012-12-02 15:08:41,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32906::DEBUG::2012-12-02 15:08:41,407::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32906::DEBUG::2012-12-02 15:08:41,407::task::957::TaskManager.Task::(_decref) Task=`bf952fd7-2614-4507-b2c0-e7039ed2a019`::ref 0 aborting False >Thread-32907::DEBUG::2012-12-02 15:08:41,415::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32907::DEBUG::2012-12-02 15:08:41,416::task::568::TaskManager.Task::(_updateState) Task=`ebb0df2d-2d02-48bc-8213-bf177d111fe2`::moving from state init -> state preparing >Thread-32907::INFO::2012-12-02 15:08:41,416::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32907::INFO::2012-12-02 15:08:41,416::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-32907::DEBUG::2012-12-02 15:08:41,416::task::1151::TaskManager.Task::(prepare) Task=`ebb0df2d-2d02-48bc-8213-bf177d111fe2`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-32907::DEBUG::2012-12-02 15:08:41,416::task::568::TaskManager.Task::(_updateState) Task=`ebb0df2d-2d02-48bc-8213-bf177d111fe2`::moving from state preparing -> state finished >Thread-32907::DEBUG::2012-12-02 15:08:41,416::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32907::DEBUG::2012-12-02 15:08:41,417::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32907::DEBUG::2012-12-02 15:08:41,417::task::957::TaskManager.Task::(_decref) Task=`ebb0df2d-2d02-48bc-8213-bf177d111fe2`::ref 0 aborting False >Thread-32908::DEBUG::2012-12-02 15:08:41,432::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32908::DEBUG::2012-12-02 15:08:41,432::task::568::TaskManager.Task::(_updateState) Task=`99404b77-2826-4c49-9e50-4dab88a3a8c1`::moving from state init -> state preparing >Thread-32908::INFO::2012-12-02 15:08:41,432::logUtils::37::dispatcher::(wrapper) Run and protect: clearTask(taskID='e143b971-cde5-4f18-92eb-a3750573504d', spUUID=None, options=None) >Thread-32908::DEBUG::2012-12-02 15:08:41,432::taskManager::157::TaskManager::(clearTask) Entry. taskID: e143b971-cde5-4f18-92eb-a3750573504d >Thread-32908::DEBUG::2012-12-02 15:08:41,432::taskManager::162::TaskManager::(clearTask) Return. >Thread-32908::INFO::2012-12-02 15:08:41,433::logUtils::39::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-32908::DEBUG::2012-12-02 15:08:41,433::task::1151::TaskManager.Task::(prepare) Task=`99404b77-2826-4c49-9e50-4dab88a3a8c1`::finished: None >Thread-32908::DEBUG::2012-12-02 15:08:41,433::task::568::TaskManager.Task::(_updateState) Task=`99404b77-2826-4c49-9e50-4dab88a3a8c1`::moving from state preparing -> state finished >Thread-32908::DEBUG::2012-12-02 15:08:41,433::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32908::DEBUG::2012-12-02 15:08:41,433::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32908::DEBUG::2012-12-02 15:08:41,433::task::957::TaskManager.Task::(_decref) Task=`99404b77-2826-4c49-9e50-4dab88a3a8c1`::ref 0 aborting False >Thread-32909::DEBUG::2012-12-02 15:08:41,499::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32909::DEBUG::2012-12-02 15:08:41,499::task::568::TaskManager.Task::(_updateState) Task=`55c315e2-12f3-41e4-b7aa-827146624193`::moving from state init -> state preparing >Thread-32909::INFO::2012-12-02 15:08:41,499::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32909::DEBUG::2012-12-02 15:08:41,500::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c900f2e3-b11e-4507-a92b-1f401ceeefa1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32909::DEBUG::2012-12-02 15:08:41,500::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-32909::DEBUG::2012-12-02 15:08:41,500::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-32909::DEBUG::2012-12-02 15:08:41,500::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c900f2e3-b11e-4507-a92b-1f401ceeefa1`::Granted request >Thread-32909::DEBUG::2012-12-02 15:08:41,500::task::794::TaskManager.Task::(resourceAcquired) Task=`55c315e2-12f3-41e4-b7aa-827146624193`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-32909::DEBUG::2012-12-02 15:08:41,500::task::957::TaskManager.Task::(_decref) Task=`55c315e2-12f3-41e4-b7aa-827146624193`::ref 1 aborting False >Thread-32909::DEBUG::2012-12-02 15:08:41,506::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32909::DEBUG::2012-12-02 15:08:41,506::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32909::DEBUG::2012-12-02 15:08:41,712::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32909::WARNING::2012-12-02 15:08:41,713::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32909::DEBUG::2012-12-02 15:08:41,713::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32909::WARNING::2012-12-02 15:08:41,742::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32909::INFO::2012-12-02 15:08:41,742::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-32909::DEBUG::2012-12-02 15:08:41,742::task::1151::TaskManager.Task::(prepare) Task=`55c315e2-12f3-41e4-b7aa-827146624193`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-32909::DEBUG::2012-12-02 15:08:41,742::task::568::TaskManager.Task::(_updateState) Task=`55c315e2-12f3-41e4-b7aa-827146624193`::moving from state preparing -> state finished >Thread-32909::DEBUG::2012-12-02 15:08:41,743::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-32909::DEBUG::2012-12-02 15:08:41,743::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32909::DEBUG::2012-12-02 15:08:41,743::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-32909::DEBUG::2012-12-02 15:08:41,743::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-32909::DEBUG::2012-12-02 15:08:41,743::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-32909::DEBUG::2012-12-02 15:08:41,743::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-32909::DEBUG::2012-12-02 15:08:41,744::task::957::TaskManager.Task::(_decref) Task=`55c315e2-12f3-41e4-b7aa-827146624193`::ref 0 aborting False >Thread-32969::DEBUG::2012-12-02 15:08:41,915::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32969::DEBUG::2012-12-02 15:08:41,916::task::568::TaskManager.Task::(_updateState) Task=`321010cf-36b8-4adb-8296-aa76a17d299c`::moving from state init -> state preparing >Thread-32969::INFO::2012-12-02 15:08:41,916::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-32969::DEBUG::2012-12-02 15:08:41,916::taskManager::177::TaskManager::(getAllTasksInfo) Entry. >Thread-32969::DEBUG::2012-12-02 15:08:41,916::taskManager::188::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-32969::INFO::2012-12-02 15:08:41,916::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-32969::DEBUG::2012-12-02 15:08:41,916::task::1151::TaskManager.Task::(prepare) Task=`321010cf-36b8-4adb-8296-aa76a17d299c`::finished: {'allTasksInfo': {}} >Thread-32969::DEBUG::2012-12-02 15:08:41,916::task::568::TaskManager.Task::(_updateState) Task=`321010cf-36b8-4adb-8296-aa76a17d299c`::moving from state preparing -> state finished >Thread-32969::DEBUG::2012-12-02 15:08:41,917::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32969::DEBUG::2012-12-02 15:08:41,917::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32969::DEBUG::2012-12-02 15:08:41,917::task::957::TaskManager.Task::(_decref) Task=`321010cf-36b8-4adb-8296-aa76a17d299c`::ref 0 aborting False >Thread-32970::DEBUG::2012-12-02 15:08:51,908::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32970::DEBUG::2012-12-02 15:08:51,909::task::568::TaskManager.Task::(_updateState) Task=`e7538bc8-6c1e-4588-bf52-cf285b5c06ec`::moving from state init -> state preparing >Thread-32970::INFO::2012-12-02 15:08:51,909::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32970::INFO::2012-12-02 15:08:51,909::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-32970::DEBUG::2012-12-02 15:08:51,909::task::1151::TaskManager.Task::(prepare) Task=`e7538bc8-6c1e-4588-bf52-cf285b5c06ec`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-32970::DEBUG::2012-12-02 15:08:51,909::task::568::TaskManager.Task::(_updateState) Task=`e7538bc8-6c1e-4588-bf52-cf285b5c06ec`::moving from state preparing -> state finished >Thread-32970::DEBUG::2012-12-02 15:08:51,909::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-32970::DEBUG::2012-12-02 15:08:51,909::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32970::DEBUG::2012-12-02 15:08:51,910::task::957::TaskManager.Task::(_decref) Task=`e7538bc8-6c1e-4588-bf52-cf285b5c06ec`::ref 0 aborting False >Thread-32971::DEBUG::2012-12-02 15:08:51,917::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-32971::DEBUG::2012-12-02 15:08:51,917::task::568::TaskManager.Task::(_updateState) Task=`c7c9ba91-7dbf-4200-8c3d-56488b9223a7`::moving from state init -> state preparing >Thread-32971::INFO::2012-12-02 15:08:51,917::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-32971::DEBUG::2012-12-02 15:08:51,918::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`36a34e65-7d23-41b1-b9c0-2985bf5d72f2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-32971::DEBUG::2012-12-02 15:08:51,918::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-32971::DEBUG::2012-12-02 15:08:51,918::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-32971::DEBUG::2012-12-02 15:08:51,918::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`36a34e65-7d23-41b1-b9c0-2985bf5d72f2`::Granted request >Thread-32971::DEBUG::2012-12-02 15:08:51,918::task::794::TaskManager.Task::(resourceAcquired) Task=`c7c9ba91-7dbf-4200-8c3d-56488b9223a7`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-32971::DEBUG::2012-12-02 15:08:51,919::task::957::TaskManager.Task::(_decref) Task=`c7c9ba91-7dbf-4200-8c3d-56488b9223a7`::ref 1 aborting False >Thread-32971::DEBUG::2012-12-02 15:08:51,925::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32971::DEBUG::2012-12-02 15:08:51,925::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-32971::DEBUG::2012-12-02 15:08:52,132::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-32971::WARNING::2012-12-02 15:08:52,134::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-32971::DEBUG::2012-12-02 15:08:52,134::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32971::WARNING::2012-12-02 15:08:52,162::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-32971::INFO::2012-12-02 15:08:52,163::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-32971::DEBUG::2012-12-02 15:08:52,163::task::1151::TaskManager.Task::(prepare) Task=`c7c9ba91-7dbf-4200-8c3d-56488b9223a7`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-32971::DEBUG::2012-12-02 15:08:52,163::task::568::TaskManager.Task::(_updateState) Task=`c7c9ba91-7dbf-4200-8c3d-56488b9223a7`::moving from state preparing -> state finished >Thread-32971::DEBUG::2012-12-02 15:08:52,163::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-32971::DEBUG::2012-12-02 15:08:52,163::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-32971::DEBUG::2012-12-02 15:08:52,163::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-32971::DEBUG::2012-12-02 15:08:52,164::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-32971::DEBUG::2012-12-02 15:08:52,164::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-32971::DEBUG::2012-12-02 15:08:52,164::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-32971::DEBUG::2012-12-02 15:08:52,164::task::957::TaskManager.Task::(_decref) Task=`c7c9ba91-7dbf-4200-8c3d-56488b9223a7`::ref 0 aborting False >Thread-33031::DEBUG::2012-12-02 15:08:54,538::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33031::DEBUG::2012-12-02 15:08:54,539::task::568::TaskManager.Task::(_updateState) Task=`e8053a2e-aaed-4cfa-8104-466ddbf1fb7d`::moving from state init -> state preparing >Thread-33031::INFO::2012-12-02 15:08:54,539::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-33031::DEBUG::2012-12-02 15:08:54,539::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c3f32d61-3cb3-464e-8304-8093d7ccf285`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33031::DEBUG::2012-12-02 15:08:54,539::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33031::DEBUG::2012-12-02 15:08:54,539::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33031::DEBUG::2012-12-02 15:08:54,540::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c3f32d61-3cb3-464e-8304-8093d7ccf285`::Granted request >Thread-33031::DEBUG::2012-12-02 15:08:54,540::task::794::TaskManager.Task::(resourceAcquired) Task=`e8053a2e-aaed-4cfa-8104-466ddbf1fb7d`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33031::DEBUG::2012-12-02 15:08:54,540::task::957::TaskManager.Task::(_decref) Task=`e8053a2e-aaed-4cfa-8104-466ddbf1fb7d`::ref 1 aborting False >Thread-33031::INFO::2012-12-02 15:08:54,540::fileSD::260::Storage.StorageDomain::(validate) sdUUID=4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-33031::DEBUG::2012-12-02 15:08:54,548::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b8f426a604115978d1ec7e997fd36ba2e68b6393'] >Thread-33031::DEBUG::2012-12-02 15:08:54,549::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:54,549::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:54,549::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:54,549::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:54,550::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:54,550::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:54,550::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-33031::DEBUG::2012-12-02 15:08:54,550::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-33031::DEBUG::2012-12-02 15:08:54,550::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-33031::DEBUG::2012-12-02 15:08:54,550::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-33031::DEBUG::2012-12-02 15:08:54,551::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-33031::DEBUG::2012-12-02 15:08:54,576::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-33031::DEBUG::2012-12-02 15:08:54,576::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-33031::DEBUG::2012-12-02 15:08:55,209::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-33031::DEBUG::2012-12-02 15:08:55,283::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-33031::DEBUG::2012-12-02 15:08:55,284::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,284::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,284::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,284::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,284::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,285::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,285::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-33031::DEBUG::2012-12-02 15:08:55,285::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,287::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-33031::DEBUG::2012-12-02 15:08:55,492::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-33031::WARNING::2012-12-02 15:08:55,494::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-33031::DEBUG::2012-12-02 15:08:55,494::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,526::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-33031::DEBUG::2012-12-02 15:08:55,527::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-33031::DEBUG::2012-12-02 15:08:55,534::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b8f426a604115978d1ec7e997fd36ba2e68b6393'] >Thread-33031::DEBUG::2012-12-02 15:08:55,536::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-33031::WARNING::2012-12-02 15:08:55,536::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_imageNS already registered >Thread-33031::WARNING::2012-12-02 15:08:55,536::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4e891d8c-9947-468e-88ec-29e49d68e083_volumeNS already registered >Thread-33031::DEBUG::2012-12-02 15:08:55,537::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 4e891d8c-9947-468e-88ec-29e49d68e083 verified, version 2 >Thread-33031::DEBUG::2012-12-02 15:08:55,537::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-33031::DEBUG::2012-12-02 15:08:55,537::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-33031::DEBUG::2012-12-02 15:08:55,537::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,538::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33031::DEBUG::2012-12-02 15:08:55,743::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33031::WARNING::2012-12-02 15:08:55,744::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33031::DEBUG::2012-12-02 15:08:55,745::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33031::ERROR::2012-12-02 15:08:55,773::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33031::DEBUG::2012-12-02 15:08:55,773::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-33031::DEBUG::2012-12-02 15:08:55,774::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33031::DEBUG::2012-12-02 15:08:55,774::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33031::DEBUG::2012-12-02 15:08:55,978::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33031::WARNING::2012-12-02 15:08:55,979::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33031::DEBUG::2012-12-02 15:08:55,980::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33031::INFO::2012-12-02 15:08:56,035::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-33031::INFO::2012-12-02 15:08:56,036::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447/4e891d8c-9947-468e-88ec-29e49d68e083 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/mastersd >Thread-33031::INFO::2012-12-02 15:08:56,036::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-33031::DEBUG::2012-12-02 15:08:56,036::task::1151::TaskManager.Task::(prepare) Task=`e8053a2e-aaed-4cfa-8104-466ddbf1fb7d`::finished: None >Thread-33031::DEBUG::2012-12-02 15:08:56,036::task::568::TaskManager.Task::(_updateState) Task=`e8053a2e-aaed-4cfa-8104-466ddbf1fb7d`::moving from state preparing -> state finished >Thread-33031::DEBUG::2012-12-02 15:08:56,036::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33031::DEBUG::2012-12-02 15:08:56,037::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33031::DEBUG::2012-12-02 15:08:56,037::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33031::DEBUG::2012-12-02 15:08:56,037::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33031::DEBUG::2012-12-02 15:08:56,037::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33031::DEBUG::2012-12-02 15:08:56,037::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33031::DEBUG::2012-12-02 15:08:56,037::task::957::TaskManager.Task::(_decref) Task=`e8053a2e-aaed-4cfa-8104-466ddbf1fb7d`::ref 0 aborting False >Thread-33208::DEBUG::2012-12-02 15:09:02,209::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33208::DEBUG::2012-12-02 15:09:02,209::task::568::TaskManager.Task::(_updateState) Task=`43584b39-3cdf-4eea-8b88-478872faea2c`::moving from state init -> state preparing >Thread-33208::INFO::2012-12-02 15:09:02,209::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33208::INFO::2012-12-02 15:09:02,209::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33208::DEBUG::2012-12-02 15:09:02,209::task::1151::TaskManager.Task::(prepare) Task=`43584b39-3cdf-4eea-8b88-478872faea2c`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33208::DEBUG::2012-12-02 15:09:02,210::task::568::TaskManager.Task::(_updateState) Task=`43584b39-3cdf-4eea-8b88-478872faea2c`::moving from state preparing -> state finished >Thread-33208::DEBUG::2012-12-02 15:09:02,210::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33208::DEBUG::2012-12-02 15:09:02,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33208::DEBUG::2012-12-02 15:09:02,210::task::957::TaskManager.Task::(_decref) Task=`43584b39-3cdf-4eea-8b88-478872faea2c`::ref 0 aborting False >Thread-33209::DEBUG::2012-12-02 15:09:02,217::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33209::DEBUG::2012-12-02 15:09:02,217::task::568::TaskManager.Task::(_updateState) Task=`4d8ed829-6ec4-4072-a662-0b740384b364`::moving from state init -> state preparing >Thread-33209::INFO::2012-12-02 15:09:02,217::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33209::DEBUG::2012-12-02 15:09:02,217::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`45f8339b-83f2-4892-acba-272755edfad3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33209::DEBUG::2012-12-02 15:09:02,218::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33209::DEBUG::2012-12-02 15:09:02,218::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33209::DEBUG::2012-12-02 15:09:02,218::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`45f8339b-83f2-4892-acba-272755edfad3`::Granted request >Thread-33209::DEBUG::2012-12-02 15:09:02,218::task::794::TaskManager.Task::(resourceAcquired) Task=`4d8ed829-6ec4-4072-a662-0b740384b364`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33209::DEBUG::2012-12-02 15:09:02,218::task::957::TaskManager.Task::(_decref) Task=`4d8ed829-6ec4-4072-a662-0b740384b364`::ref 1 aborting False >Thread-33209::DEBUG::2012-12-02 15:09:02,225::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33209::DEBUG::2012-12-02 15:09:02,225::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33209::DEBUG::2012-12-02 15:09:02,427::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33209::WARNING::2012-12-02 15:09:02,429::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33209::DEBUG::2012-12-02 15:09:02,429::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33209::WARNING::2012-12-02 15:09:02,456::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33209::INFO::2012-12-02 15:09:02,456::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33209::DEBUG::2012-12-02 15:09:02,457::task::1151::TaskManager.Task::(prepare) Task=`4d8ed829-6ec4-4072-a662-0b740384b364`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33209::DEBUG::2012-12-02 15:09:02,457::task::568::TaskManager.Task::(_updateState) Task=`4d8ed829-6ec4-4072-a662-0b740384b364`::moving from state preparing -> state finished >Thread-33209::DEBUG::2012-12-02 15:09:02,457::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33209::DEBUG::2012-12-02 15:09:02,457::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33209::DEBUG::2012-12-02 15:09:02,457::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33209::DEBUG::2012-12-02 15:09:02,457::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33209::DEBUG::2012-12-02 15:09:02,458::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33209::DEBUG::2012-12-02 15:09:02,458::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33209::DEBUG::2012-12-02 15:09:02,458::task::957::TaskManager.Task::(_decref) Task=`4d8ed829-6ec4-4072-a662-0b740384b364`::ref 0 aborting False >Thread-33269::DEBUG::2012-12-02 15:09:12,533::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33269::DEBUG::2012-12-02 15:09:12,534::task::568::TaskManager.Task::(_updateState) Task=`fea092eb-4771-4f6a-bac6-9f70c428fef6`::moving from state init -> state preparing >Thread-33269::INFO::2012-12-02 15:09:12,534::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33269::INFO::2012-12-02 15:09:12,534::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33269::DEBUG::2012-12-02 15:09:12,534::task::1151::TaskManager.Task::(prepare) Task=`fea092eb-4771-4f6a-bac6-9f70c428fef6`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33269::DEBUG::2012-12-02 15:09:12,534::task::568::TaskManager.Task::(_updateState) Task=`fea092eb-4771-4f6a-bac6-9f70c428fef6`::moving from state preparing -> state finished >Thread-33269::DEBUG::2012-12-02 15:09:12,534::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33269::DEBUG::2012-12-02 15:09:12,534::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33269::DEBUG::2012-12-02 15:09:12,535::task::957::TaskManager.Task::(_decref) Task=`fea092eb-4771-4f6a-bac6-9f70c428fef6`::ref 0 aborting False >Thread-33270::DEBUG::2012-12-02 15:09:12,542::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33270::DEBUG::2012-12-02 15:09:12,542::task::568::TaskManager.Task::(_updateState) Task=`a4926267-54ff-4f43-b980-d7a00d1675ad`::moving from state init -> state preparing >Thread-33270::INFO::2012-12-02 15:09:12,542::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33270::DEBUG::2012-12-02 15:09:12,542::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`19dbbf11-ac2c-48ec-83f4-cc8522bfcb5d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33270::DEBUG::2012-12-02 15:09:12,542::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33270::DEBUG::2012-12-02 15:09:12,542::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33270::DEBUG::2012-12-02 15:09:12,542::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`19dbbf11-ac2c-48ec-83f4-cc8522bfcb5d`::Granted request >Thread-33270::DEBUG::2012-12-02 15:09:12,543::task::794::TaskManager.Task::(resourceAcquired) Task=`a4926267-54ff-4f43-b980-d7a00d1675ad`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33270::DEBUG::2012-12-02 15:09:12,543::task::957::TaskManager.Task::(_decref) Task=`a4926267-54ff-4f43-b980-d7a00d1675ad`::ref 1 aborting False >Thread-33270::DEBUG::2012-12-02 15:09:12,548::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33270::DEBUG::2012-12-02 15:09:12,548::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33270::DEBUG::2012-12-02 15:09:12,753::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33270::WARNING::2012-12-02 15:09:12,755::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33270::DEBUG::2012-12-02 15:09:12,755::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33270::WARNING::2012-12-02 15:09:12,789::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33270::INFO::2012-12-02 15:09:12,790::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33270::DEBUG::2012-12-02 15:09:12,790::task::1151::TaskManager.Task::(prepare) Task=`a4926267-54ff-4f43-b980-d7a00d1675ad`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33270::DEBUG::2012-12-02 15:09:12,790::task::568::TaskManager.Task::(_updateState) Task=`a4926267-54ff-4f43-b980-d7a00d1675ad`::moving from state preparing -> state finished >Thread-33270::DEBUG::2012-12-02 15:09:12,790::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33270::DEBUG::2012-12-02 15:09:12,790::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33270::DEBUG::2012-12-02 15:09:12,790::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33270::DEBUG::2012-12-02 15:09:12,791::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33270::DEBUG::2012-12-02 15:09:12,791::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33270::DEBUG::2012-12-02 15:09:12,791::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33270::DEBUG::2012-12-02 15:09:12,791::task::957::TaskManager.Task::(_decref) Task=`a4926267-54ff-4f43-b980-d7a00d1675ad`::ref 0 aborting False >Thread-33330::DEBUG::2012-12-02 15:09:22,860::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33330::DEBUG::2012-12-02 15:09:22,860::task::568::TaskManager.Task::(_updateState) Task=`606dea95-ac73-4473-8bc5-8e79d69d43ee`::moving from state init -> state preparing >Thread-33330::INFO::2012-12-02 15:09:22,861::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33330::INFO::2012-12-02 15:09:22,861::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33330::DEBUG::2012-12-02 15:09:22,861::task::1151::TaskManager.Task::(prepare) Task=`606dea95-ac73-4473-8bc5-8e79d69d43ee`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33330::DEBUG::2012-12-02 15:09:22,861::task::568::TaskManager.Task::(_updateState) Task=`606dea95-ac73-4473-8bc5-8e79d69d43ee`::moving from state preparing -> state finished >Thread-33330::DEBUG::2012-12-02 15:09:22,861::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33330::DEBUG::2012-12-02 15:09:22,861::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33330::DEBUG::2012-12-02 15:09:22,862::task::957::TaskManager.Task::(_decref) Task=`606dea95-ac73-4473-8bc5-8e79d69d43ee`::ref 0 aborting False >Thread-33331::DEBUG::2012-12-02 15:09:22,868::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33331::DEBUG::2012-12-02 15:09:22,869::task::568::TaskManager.Task::(_updateState) Task=`ccbb3092-2b4d-44b3-8e3a-6db8a65b3dba`::moving from state init -> state preparing >Thread-33331::INFO::2012-12-02 15:09:22,869::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33331::DEBUG::2012-12-02 15:09:22,869::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`9eb91425-ddcd-4002-bb96-cecee63294cb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33331::DEBUG::2012-12-02 15:09:22,869::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33331::DEBUG::2012-12-02 15:09:22,869::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33331::DEBUG::2012-12-02 15:09:22,870::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`9eb91425-ddcd-4002-bb96-cecee63294cb`::Granted request >Thread-33331::DEBUG::2012-12-02 15:09:22,870::task::794::TaskManager.Task::(resourceAcquired) Task=`ccbb3092-2b4d-44b3-8e3a-6db8a65b3dba`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33331::DEBUG::2012-12-02 15:09:22,870::task::957::TaskManager.Task::(_decref) Task=`ccbb3092-2b4d-44b3-8e3a-6db8a65b3dba`::ref 1 aborting False >Thread-33331::DEBUG::2012-12-02 15:09:22,876::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33331::DEBUG::2012-12-02 15:09:22,876::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33331::DEBUG::2012-12-02 15:09:23,079::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33331::WARNING::2012-12-02 15:09:23,080::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33331::DEBUG::2012-12-02 15:09:23,080::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33331::WARNING::2012-12-02 15:09:23,110::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33331::INFO::2012-12-02 15:09:23,110::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33331::DEBUG::2012-12-02 15:09:23,111::task::1151::TaskManager.Task::(prepare) Task=`ccbb3092-2b4d-44b3-8e3a-6db8a65b3dba`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33331::DEBUG::2012-12-02 15:09:23,111::task::568::TaskManager.Task::(_updateState) Task=`ccbb3092-2b4d-44b3-8e3a-6db8a65b3dba`::moving from state preparing -> state finished >Thread-33331::DEBUG::2012-12-02 15:09:23,111::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33331::DEBUG::2012-12-02 15:09:23,111::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33331::DEBUG::2012-12-02 15:09:23,111::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33331::DEBUG::2012-12-02 15:09:23,112::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33331::DEBUG::2012-12-02 15:09:23,112::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33331::DEBUG::2012-12-02 15:09:23,112::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33331::DEBUG::2012-12-02 15:09:23,112::task::957::TaskManager.Task::(_decref) Task=`ccbb3092-2b4d-44b3-8e3a-6db8a65b3dba`::ref 0 aborting False >Thread-33391::DEBUG::2012-12-02 15:09:33,194::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33391::DEBUG::2012-12-02 15:09:33,195::task::568::TaskManager.Task::(_updateState) Task=`8763d260-f229-488b-9d04-dbb39a6e1d49`::moving from state init -> state preparing >Thread-33391::INFO::2012-12-02 15:09:33,195::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33391::INFO::2012-12-02 15:09:33,195::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33391::DEBUG::2012-12-02 15:09:33,195::task::1151::TaskManager.Task::(prepare) Task=`8763d260-f229-488b-9d04-dbb39a6e1d49`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33391::DEBUG::2012-12-02 15:09:33,195::task::568::TaskManager.Task::(_updateState) Task=`8763d260-f229-488b-9d04-dbb39a6e1d49`::moving from state preparing -> state finished >Thread-33391::DEBUG::2012-12-02 15:09:33,195::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33391::DEBUG::2012-12-02 15:09:33,195::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33391::DEBUG::2012-12-02 15:09:33,195::task::957::TaskManager.Task::(_decref) Task=`8763d260-f229-488b-9d04-dbb39a6e1d49`::ref 0 aborting False >Thread-33392::DEBUG::2012-12-02 15:09:33,202::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33392::DEBUG::2012-12-02 15:09:33,202::task::568::TaskManager.Task::(_updateState) Task=`641c1f0b-c1f7-4a78-8084-d1c99e9653f1`::moving from state init -> state preparing >Thread-33392::INFO::2012-12-02 15:09:33,203::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33392::DEBUG::2012-12-02 15:09:33,203::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`034a1c99-ef0b-4c4a-8683-ff83f33d7651`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33392::DEBUG::2012-12-02 15:09:33,203::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33392::DEBUG::2012-12-02 15:09:33,203::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33392::DEBUG::2012-12-02 15:09:33,203::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`034a1c99-ef0b-4c4a-8683-ff83f33d7651`::Granted request >Thread-33392::DEBUG::2012-12-02 15:09:33,203::task::794::TaskManager.Task::(resourceAcquired) Task=`641c1f0b-c1f7-4a78-8084-d1c99e9653f1`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33392::DEBUG::2012-12-02 15:09:33,203::task::957::TaskManager.Task::(_decref) Task=`641c1f0b-c1f7-4a78-8084-d1c99e9653f1`::ref 1 aborting False >Thread-33392::DEBUG::2012-12-02 15:09:33,209::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33392::DEBUG::2012-12-02 15:09:33,209::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33392::DEBUG::2012-12-02 15:09:33,418::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33392::WARNING::2012-12-02 15:09:33,419::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33392::DEBUG::2012-12-02 15:09:33,420::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33392::WARNING::2012-12-02 15:09:33,448::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33392::INFO::2012-12-02 15:09:33,448::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33392::DEBUG::2012-12-02 15:09:33,448::task::1151::TaskManager.Task::(prepare) Task=`641c1f0b-c1f7-4a78-8084-d1c99e9653f1`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33392::DEBUG::2012-12-02 15:09:33,449::task::568::TaskManager.Task::(_updateState) Task=`641c1f0b-c1f7-4a78-8084-d1c99e9653f1`::moving from state preparing -> state finished >Thread-33392::DEBUG::2012-12-02 15:09:33,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33392::DEBUG::2012-12-02 15:09:33,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33392::DEBUG::2012-12-02 15:09:33,449::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33392::DEBUG::2012-12-02 15:09:33,449::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33392::DEBUG::2012-12-02 15:09:33,449::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33392::DEBUG::2012-12-02 15:09:33,450::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33392::DEBUG::2012-12-02 15:09:33,450::task::957::TaskManager.Task::(_decref) Task=`641c1f0b-c1f7-4a78-8084-d1c99e9653f1`::ref 0 aborting False >Thread-33452::DEBUG::2012-12-02 15:09:35,632::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33452::DEBUG::2012-12-02 15:09:35,632::task::568::TaskManager.Task::(_updateState) Task=`2bc135c5-f39f-4739-b292-1165c486fdd4`::moving from state init -> state preparing >Thread-33452::INFO::2012-12-02 15:09:35,632::logUtils::37::dispatcher::(wrapper) Run and protect: updateVM(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', vmList=[], sdUUID=None, options=None) >Thread-33452::DEBUG::2012-12-02 15:09:35,633::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`d22d347a-7386-4205-b8d2-995d74516c5d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33452::DEBUG::2012-12-02 15:09:35,633::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33452::DEBUG::2012-12-02 15:09:35,633::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33452::DEBUG::2012-12-02 15:09:35,633::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`d22d347a-7386-4205-b8d2-995d74516c5d`::Granted request >Thread-33452::DEBUG::2012-12-02 15:09:35,633::task::794::TaskManager.Task::(resourceAcquired) Task=`2bc135c5-f39f-4739-b292-1165c486fdd4`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33452::DEBUG::2012-12-02 15:09:35,633::task::957::TaskManager.Task::(_decref) Task=`2bc135c5-f39f-4739-b292-1165c486fdd4`::ref 1 aborting False >Thread-33452::INFO::2012-12-02 15:09:35,634::sp::1247::Storage.StoragePool::(updateVM) spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6 sdUUID=4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-33452::INFO::2012-12-02 15:09:35,634::logUtils::39::dispatcher::(wrapper) Run and protect: updateVM, Return response: None >Thread-33452::DEBUG::2012-12-02 15:09:35,634::task::1151::TaskManager.Task::(prepare) Task=`2bc135c5-f39f-4739-b292-1165c486fdd4`::finished: None >Thread-33452::DEBUG::2012-12-02 15:09:35,634::task::568::TaskManager.Task::(_updateState) Task=`2bc135c5-f39f-4739-b292-1165c486fdd4`::moving from state preparing -> state finished >Thread-33452::DEBUG::2012-12-02 15:09:35,634::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33452::DEBUG::2012-12-02 15:09:35,634::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33452::DEBUG::2012-12-02 15:09:35,634::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33452::DEBUG::2012-12-02 15:09:35,635::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33452::DEBUG::2012-12-02 15:09:35,635::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33452::DEBUG::2012-12-02 15:09:35,635::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33452::DEBUG::2012-12-02 15:09:35,635::task::957::TaskManager.Task::(_decref) Task=`2bc135c5-f39f-4739-b292-1165c486fdd4`::ref 0 aborting False >Thread-33456::DEBUG::2012-12-02 15:09:43,525::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33456::DEBUG::2012-12-02 15:09:43,526::task::568::TaskManager.Task::(_updateState) Task=`e085aa7d-bdcf-4a47-9bbd-b3887c6b6f57`::moving from state init -> state preparing >Thread-33456::INFO::2012-12-02 15:09:43,526::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33456::INFO::2012-12-02 15:09:43,526::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33456::DEBUG::2012-12-02 15:09:43,526::task::1151::TaskManager.Task::(prepare) Task=`e085aa7d-bdcf-4a47-9bbd-b3887c6b6f57`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33456::DEBUG::2012-12-02 15:09:43,526::task::568::TaskManager.Task::(_updateState) Task=`e085aa7d-bdcf-4a47-9bbd-b3887c6b6f57`::moving from state preparing -> state finished >Thread-33456::DEBUG::2012-12-02 15:09:43,526::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33456::DEBUG::2012-12-02 15:09:43,527::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33456::DEBUG::2012-12-02 15:09:43,527::task::957::TaskManager.Task::(_decref) Task=`e085aa7d-bdcf-4a47-9bbd-b3887c6b6f57`::ref 0 aborting False >Thread-33457::DEBUG::2012-12-02 15:09:43,531::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33457::DEBUG::2012-12-02 15:09:43,531::task::568::TaskManager.Task::(_updateState) Task=`ee418bfc-0743-48b1-91ea-27020aaa43bc`::moving from state init -> state preparing >Thread-33457::INFO::2012-12-02 15:09:43,531::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33457::DEBUG::2012-12-02 15:09:43,531::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`21fcb278-05d4-4da5-8c64-40892c8b3de0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33457::DEBUG::2012-12-02 15:09:43,532::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33457::DEBUG::2012-12-02 15:09:43,532::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33457::DEBUG::2012-12-02 15:09:43,532::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`21fcb278-05d4-4da5-8c64-40892c8b3de0`::Granted request >Thread-33457::DEBUG::2012-12-02 15:09:43,532::task::794::TaskManager.Task::(resourceAcquired) Task=`ee418bfc-0743-48b1-91ea-27020aaa43bc`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33457::DEBUG::2012-12-02 15:09:43,532::task::957::TaskManager.Task::(_decref) Task=`ee418bfc-0743-48b1-91ea-27020aaa43bc`::ref 1 aborting False >Thread-33457::DEBUG::2012-12-02 15:09:43,537::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33457::DEBUG::2012-12-02 15:09:43,538::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33457::DEBUG::2012-12-02 15:09:43,743::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33457::WARNING::2012-12-02 15:09:43,744::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33457::DEBUG::2012-12-02 15:09:43,744::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33457::WARNING::2012-12-02 15:09:43,774::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33457::INFO::2012-12-02 15:09:43,774::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33457::DEBUG::2012-12-02 15:09:43,774::task::1151::TaskManager.Task::(prepare) Task=`ee418bfc-0743-48b1-91ea-27020aaa43bc`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33457::DEBUG::2012-12-02 15:09:43,774::task::568::TaskManager.Task::(_updateState) Task=`ee418bfc-0743-48b1-91ea-27020aaa43bc`::moving from state preparing -> state finished >Thread-33457::DEBUG::2012-12-02 15:09:43,775::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33457::DEBUG::2012-12-02 15:09:43,775::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33457::DEBUG::2012-12-02 15:09:43,775::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33457::DEBUG::2012-12-02 15:09:43,775::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33457::DEBUG::2012-12-02 15:09:43,775::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33457::DEBUG::2012-12-02 15:09:43,775::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33457::DEBUG::2012-12-02 15:09:43,776::task::957::TaskManager.Task::(_decref) Task=`ee418bfc-0743-48b1-91ea-27020aaa43bc`::ref 0 aborting False >Thread-33518::DEBUG::2012-12-02 15:09:45,743::task::568::TaskManager.Task::(_updateState) Task=`cd3e620a-9585-40ed-885c-b676d2b1ad03`::moving from state init -> state preparing >Thread-33518::INFO::2012-12-02 15:09:45,744::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33518::INFO::2012-12-02 15:09:45,744::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00723004341125', 'lastCheck': 1354453779.594697, 'code': 0, 'valid': True}} >Thread-33518::DEBUG::2012-12-02 15:09:45,744::task::1151::TaskManager.Task::(prepare) Task=`cd3e620a-9585-40ed-885c-b676d2b1ad03`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00723004341125', 'lastCheck': 1354453779.594697, 'code': 0, 'valid': True}} >Thread-33518::DEBUG::2012-12-02 15:09:45,744::task::568::TaskManager.Task::(_updateState) Task=`cd3e620a-9585-40ed-885c-b676d2b1ad03`::moving from state preparing -> state finished >Thread-33518::DEBUG::2012-12-02 15:09:45,744::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33518::DEBUG::2012-12-02 15:09:45,744::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33518::DEBUG::2012-12-02 15:09:45,745::task::957::TaskManager.Task::(_decref) Task=`cd3e620a-9585-40ed-885c-b676d2b1ad03`::ref 0 aborting False >Thread-33524::DEBUG::2012-12-02 15:09:53,893::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33524::DEBUG::2012-12-02 15:09:53,893::task::568::TaskManager.Task::(_updateState) Task=`1e99be91-1911-48b5-98e1-fcb4bd4d36d3`::moving from state init -> state preparing >Thread-33524::INFO::2012-12-02 15:09:53,894::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33524::INFO::2012-12-02 15:09:53,894::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33524::DEBUG::2012-12-02 15:09:53,894::task::1151::TaskManager.Task::(prepare) Task=`1e99be91-1911-48b5-98e1-fcb4bd4d36d3`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33524::DEBUG::2012-12-02 15:09:53,894::task::568::TaskManager.Task::(_updateState) Task=`1e99be91-1911-48b5-98e1-fcb4bd4d36d3`::moving from state preparing -> state finished >Thread-33524::DEBUG::2012-12-02 15:09:53,894::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33524::DEBUG::2012-12-02 15:09:53,894::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33524::DEBUG::2012-12-02 15:09:53,894::task::957::TaskManager.Task::(_decref) Task=`1e99be91-1911-48b5-98e1-fcb4bd4d36d3`::ref 0 aborting False >Thread-33525::DEBUG::2012-12-02 15:09:53,899::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33525::DEBUG::2012-12-02 15:09:53,899::task::568::TaskManager.Task::(_updateState) Task=`b60773d5-e152-4561-9822-c5a3eb900a2c`::moving from state init -> state preparing >Thread-33525::INFO::2012-12-02 15:09:53,899::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33525::DEBUG::2012-12-02 15:09:53,900::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`a74f6580-988f-4765-a7c1-828bddb72640`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33525::DEBUG::2012-12-02 15:09:53,900::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33525::DEBUG::2012-12-02 15:09:53,900::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33525::DEBUG::2012-12-02 15:09:53,900::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`a74f6580-988f-4765-a7c1-828bddb72640`::Granted request >Thread-33525::DEBUG::2012-12-02 15:09:53,900::task::794::TaskManager.Task::(resourceAcquired) Task=`b60773d5-e152-4561-9822-c5a3eb900a2c`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33525::DEBUG::2012-12-02 15:09:53,901::task::957::TaskManager.Task::(_decref) Task=`b60773d5-e152-4561-9822-c5a3eb900a2c`::ref 1 aborting False >Thread-33525::DEBUG::2012-12-02 15:09:53,906::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33525::DEBUG::2012-12-02 15:09:53,907::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33525::DEBUG::2012-12-02 15:09:54,114::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33525::WARNING::2012-12-02 15:09:54,115::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33525::DEBUG::2012-12-02 15:09:54,115::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33525::WARNING::2012-12-02 15:09:54,144::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33525::INFO::2012-12-02 15:09:54,144::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33525::DEBUG::2012-12-02 15:09:54,144::task::1151::TaskManager.Task::(prepare) Task=`b60773d5-e152-4561-9822-c5a3eb900a2c`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33525::DEBUG::2012-12-02 15:09:54,144::task::568::TaskManager.Task::(_updateState) Task=`b60773d5-e152-4561-9822-c5a3eb900a2c`::moving from state preparing -> state finished >Thread-33525::DEBUG::2012-12-02 15:09:54,145::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33525::DEBUG::2012-12-02 15:09:54,145::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33525::DEBUG::2012-12-02 15:09:54,145::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33525::DEBUG::2012-12-02 15:09:54,145::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33525::DEBUG::2012-12-02 15:09:54,145::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33525::DEBUG::2012-12-02 15:09:54,145::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33525::DEBUG::2012-12-02 15:09:54,146::task::957::TaskManager.Task::(_decref) Task=`b60773d5-e152-4561-9822-c5a3eb900a2c`::ref 0 aborting False >Thread-33585::DEBUG::2012-12-02 15:09:55,866::task::568::TaskManager.Task::(_updateState) Task=`6b22b41a-df88-41eb-b9c6-43c21551bd60`::moving from state init -> state preparing >Thread-33585::INFO::2012-12-02 15:09:55,867::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33585::INFO::2012-12-02 15:09:55,867::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00744700431824', 'lastCheck': 1354453789.611779, 'code': 0, 'valid': True}} >Thread-33585::DEBUG::2012-12-02 15:09:55,867::task::1151::TaskManager.Task::(prepare) Task=`6b22b41a-df88-41eb-b9c6-43c21551bd60`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00744700431824', 'lastCheck': 1354453789.611779, 'code': 0, 'valid': True}} >Thread-33585::DEBUG::2012-12-02 15:09:55,867::task::568::TaskManager.Task::(_updateState) Task=`6b22b41a-df88-41eb-b9c6-43c21551bd60`::moving from state preparing -> state finished >Thread-33585::DEBUG::2012-12-02 15:09:55,867::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33585::DEBUG::2012-12-02 15:09:55,867::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33585::DEBUG::2012-12-02 15:09:55,868::task::957::TaskManager.Task::(_decref) Task=`6b22b41a-df88-41eb-b9c6-43c21551bd60`::ref 0 aborting False >Thread-33591::DEBUG::2012-12-02 15:10:04,204::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33591::DEBUG::2012-12-02 15:10:04,207::task::568::TaskManager.Task::(_updateState) Task=`bfdfe888-18ef-4857-ba66-bb2db6e40cb1`::moving from state init -> state preparing >Thread-33591::INFO::2012-12-02 15:10:04,207::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33591::INFO::2012-12-02 15:10:04,207::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33591::DEBUG::2012-12-02 15:10:04,207::task::1151::TaskManager.Task::(prepare) Task=`bfdfe888-18ef-4857-ba66-bb2db6e40cb1`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33591::DEBUG::2012-12-02 15:10:04,208::task::568::TaskManager.Task::(_updateState) Task=`bfdfe888-18ef-4857-ba66-bb2db6e40cb1`::moving from state preparing -> state finished >Thread-33591::DEBUG::2012-12-02 15:10:04,208::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33591::DEBUG::2012-12-02 15:10:04,208::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33591::DEBUG::2012-12-02 15:10:04,208::task::957::TaskManager.Task::(_decref) Task=`bfdfe888-18ef-4857-ba66-bb2db6e40cb1`::ref 0 aborting False >Thread-33592::DEBUG::2012-12-02 15:10:04,214::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33592::DEBUG::2012-12-02 15:10:04,215::task::568::TaskManager.Task::(_updateState) Task=`b09bdd95-14c6-4af6-b460-7b923cd33755`::moving from state init -> state preparing >Thread-33592::INFO::2012-12-02 15:10:04,215::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33592::DEBUG::2012-12-02 15:10:04,215::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b5e11ba9-5d61-4f16-9c40-43fbf05a68db`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33592::DEBUG::2012-12-02 15:10:04,215::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33592::DEBUG::2012-12-02 15:10:04,215::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33592::DEBUG::2012-12-02 15:10:04,216::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b5e11ba9-5d61-4f16-9c40-43fbf05a68db`::Granted request >Thread-33592::DEBUG::2012-12-02 15:10:04,216::task::794::TaskManager.Task::(resourceAcquired) Task=`b09bdd95-14c6-4af6-b460-7b923cd33755`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33592::DEBUG::2012-12-02 15:10:04,216::task::957::TaskManager.Task::(_decref) Task=`b09bdd95-14c6-4af6-b460-7b923cd33755`::ref 1 aborting False >Thread-33592::DEBUG::2012-12-02 15:10:04,222::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33592::DEBUG::2012-12-02 15:10:04,223::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33592::DEBUG::2012-12-02 15:10:04,437::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33592::WARNING::2012-12-02 15:10:04,439::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33592::DEBUG::2012-12-02 15:10:04,440::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33592::WARNING::2012-12-02 15:10:04,469::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33592::INFO::2012-12-02 15:10:04,469::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33592::DEBUG::2012-12-02 15:10:04,469::task::1151::TaskManager.Task::(prepare) Task=`b09bdd95-14c6-4af6-b460-7b923cd33755`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33592::DEBUG::2012-12-02 15:10:04,469::task::568::TaskManager.Task::(_updateState) Task=`b09bdd95-14c6-4af6-b460-7b923cd33755`::moving from state preparing -> state finished >Thread-33592::DEBUG::2012-12-02 15:10:04,470::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33592::DEBUG::2012-12-02 15:10:04,470::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33592::DEBUG::2012-12-02 15:10:04,470::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33592::DEBUG::2012-12-02 15:10:04,470::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33592::DEBUG::2012-12-02 15:10:04,470::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33592::DEBUG::2012-12-02 15:10:04,470::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33592::DEBUG::2012-12-02 15:10:04,471::task::957::TaskManager.Task::(_decref) Task=`b09bdd95-14c6-4af6-b460-7b923cd33755`::ref 0 aborting False >Thread-33652::DEBUG::2012-12-02 15:10:06,016::task::568::TaskManager.Task::(_updateState) Task=`5798c97e-a80b-48d9-8ae6-616d260df277`::moving from state init -> state preparing >Thread-33652::INFO::2012-12-02 15:10:06,017::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33652::INFO::2012-12-02 15:10:06,017::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0088529586792', 'lastCheck': 1354453799.630688, 'code': 0, 'valid': True}} >Thread-33652::DEBUG::2012-12-02 15:10:06,017::task::1151::TaskManager.Task::(prepare) Task=`5798c97e-a80b-48d9-8ae6-616d260df277`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0088529586792', 'lastCheck': 1354453799.630688, 'code': 0, 'valid': True}} >Thread-33652::DEBUG::2012-12-02 15:10:06,017::task::568::TaskManager.Task::(_updateState) Task=`5798c97e-a80b-48d9-8ae6-616d260df277`::moving from state preparing -> state finished >Thread-33652::DEBUG::2012-12-02 15:10:06,017::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33652::DEBUG::2012-12-02 15:10:06,017::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33652::DEBUG::2012-12-02 15:10:06,017::task::957::TaskManager.Task::(_decref) Task=`5798c97e-a80b-48d9-8ae6-616d260df277`::ref 0 aborting False >Thread-33658::DEBUG::2012-12-02 15:10:14,546::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33658::DEBUG::2012-12-02 15:10:14,546::task::568::TaskManager.Task::(_updateState) Task=`3d282052-fbe4-47c0-9a38-0639e69233d9`::moving from state init -> state preparing >Thread-33658::INFO::2012-12-02 15:10:14,546::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33658::INFO::2012-12-02 15:10:14,546::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33658::DEBUG::2012-12-02 15:10:14,546::task::1151::TaskManager.Task::(prepare) Task=`3d282052-fbe4-47c0-9a38-0639e69233d9`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33658::DEBUG::2012-12-02 15:10:14,547::task::568::TaskManager.Task::(_updateState) Task=`3d282052-fbe4-47c0-9a38-0639e69233d9`::moving from state preparing -> state finished >Thread-33658::DEBUG::2012-12-02 15:10:14,547::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33658::DEBUG::2012-12-02 15:10:14,547::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33658::DEBUG::2012-12-02 15:10:14,547::task::957::TaskManager.Task::(_decref) Task=`3d282052-fbe4-47c0-9a38-0639e69233d9`::ref 0 aborting False >Thread-33659::DEBUG::2012-12-02 15:10:14,553::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33659::DEBUG::2012-12-02 15:10:14,554::task::568::TaskManager.Task::(_updateState) Task=`33470d18-5a58-4924-b51d-74796b108f28`::moving from state init -> state preparing >Thread-33659::INFO::2012-12-02 15:10:14,554::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33659::DEBUG::2012-12-02 15:10:14,554::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`8850f873-8a51-4b2e-ab47-06cc599b19bd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33659::DEBUG::2012-12-02 15:10:14,554::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33659::DEBUG::2012-12-02 15:10:14,555::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33659::DEBUG::2012-12-02 15:10:14,555::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`8850f873-8a51-4b2e-ab47-06cc599b19bd`::Granted request >Thread-33659::DEBUG::2012-12-02 15:10:14,555::task::794::TaskManager.Task::(resourceAcquired) Task=`33470d18-5a58-4924-b51d-74796b108f28`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33659::DEBUG::2012-12-02 15:10:14,555::task::957::TaskManager.Task::(_decref) Task=`33470d18-5a58-4924-b51d-74796b108f28`::ref 1 aborting False >Thread-33659::DEBUG::2012-12-02 15:10:14,562::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33659::DEBUG::2012-12-02 15:10:14,562::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33659::DEBUG::2012-12-02 15:10:14,769::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33659::WARNING::2012-12-02 15:10:14,771::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33659::DEBUG::2012-12-02 15:10:14,771::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33659::WARNING::2012-12-02 15:10:14,808::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33659::INFO::2012-12-02 15:10:14,809::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33659::DEBUG::2012-12-02 15:10:14,809::task::1151::TaskManager.Task::(prepare) Task=`33470d18-5a58-4924-b51d-74796b108f28`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33659::DEBUG::2012-12-02 15:10:14,809::task::568::TaskManager.Task::(_updateState) Task=`33470d18-5a58-4924-b51d-74796b108f28`::moving from state preparing -> state finished >Thread-33659::DEBUG::2012-12-02 15:10:14,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33659::DEBUG::2012-12-02 15:10:14,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33659::DEBUG::2012-12-02 15:10:14,809::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33659::DEBUG::2012-12-02 15:10:14,810::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33659::DEBUG::2012-12-02 15:10:14,810::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33659::DEBUG::2012-12-02 15:10:14,810::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33659::DEBUG::2012-12-02 15:10:14,810::task::957::TaskManager.Task::(_decref) Task=`33470d18-5a58-4924-b51d-74796b108f28`::ref 0 aborting False >Thread-33719::DEBUG::2012-12-02 15:10:16,141::task::568::TaskManager.Task::(_updateState) Task=`9b0baf88-c624-4cf5-8380-4591d1a9c2ad`::moving from state init -> state preparing >Thread-33719::INFO::2012-12-02 15:10:16,141::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33719::INFO::2012-12-02 15:10:16,141::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0072648525238', 'lastCheck': 1354453809.6496501, 'code': 0, 'valid': True}} >Thread-33719::DEBUG::2012-12-02 15:10:16,141::task::1151::TaskManager.Task::(prepare) Task=`9b0baf88-c624-4cf5-8380-4591d1a9c2ad`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0072648525238', 'lastCheck': 1354453809.6496501, 'code': 0, 'valid': True}} >Thread-33719::DEBUG::2012-12-02 15:10:16,141::task::568::TaskManager.Task::(_updateState) Task=`9b0baf88-c624-4cf5-8380-4591d1a9c2ad`::moving from state preparing -> state finished >Thread-33719::DEBUG::2012-12-02 15:10:16,142::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33719::DEBUG::2012-12-02 15:10:16,142::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33719::DEBUG::2012-12-02 15:10:16,142::task::957::TaskManager.Task::(_decref) Task=`9b0baf88-c624-4cf5-8380-4591d1a9c2ad`::ref 0 aborting False >Thread-33725::DEBUG::2012-12-02 15:10:24,890::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33725::DEBUG::2012-12-02 15:10:24,891::task::568::TaskManager.Task::(_updateState) Task=`dcb9d63a-8d07-4e83-857d-c19c7796f3ed`::moving from state init -> state preparing >Thread-33725::INFO::2012-12-02 15:10:24,891::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33725::INFO::2012-12-02 15:10:24,891::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33725::DEBUG::2012-12-02 15:10:24,891::task::1151::TaskManager.Task::(prepare) Task=`dcb9d63a-8d07-4e83-857d-c19c7796f3ed`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33725::DEBUG::2012-12-02 15:10:24,891::task::568::TaskManager.Task::(_updateState) Task=`dcb9d63a-8d07-4e83-857d-c19c7796f3ed`::moving from state preparing -> state finished >Thread-33725::DEBUG::2012-12-02 15:10:24,891::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33725::DEBUG::2012-12-02 15:10:24,891::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33725::DEBUG::2012-12-02 15:10:24,892::task::957::TaskManager.Task::(_decref) Task=`dcb9d63a-8d07-4e83-857d-c19c7796f3ed`::ref 0 aborting False >Thread-33726::DEBUG::2012-12-02 15:10:24,898::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33726::DEBUG::2012-12-02 15:10:24,898::task::568::TaskManager.Task::(_updateState) Task=`9abdc3cb-85e0-417d-8c69-d1ab308644e8`::moving from state init -> state preparing >Thread-33726::INFO::2012-12-02 15:10:24,899::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33726::DEBUG::2012-12-02 15:10:24,899::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`38c19090-a7ba-4396-81d4-8793a8fd6ead`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33726::DEBUG::2012-12-02 15:10:24,899::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33726::DEBUG::2012-12-02 15:10:24,899::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33726::DEBUG::2012-12-02 15:10:24,899::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`38c19090-a7ba-4396-81d4-8793a8fd6ead`::Granted request >Thread-33726::DEBUG::2012-12-02 15:10:24,900::task::794::TaskManager.Task::(resourceAcquired) Task=`9abdc3cb-85e0-417d-8c69-d1ab308644e8`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33726::DEBUG::2012-12-02 15:10:24,900::task::957::TaskManager.Task::(_decref) Task=`9abdc3cb-85e0-417d-8c69-d1ab308644e8`::ref 1 aborting False >Thread-33726::DEBUG::2012-12-02 15:10:24,906::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33726::DEBUG::2012-12-02 15:10:24,906::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33726::DEBUG::2012-12-02 15:10:25,114::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33726::WARNING::2012-12-02 15:10:25,115::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33726::DEBUG::2012-12-02 15:10:25,115::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33726::WARNING::2012-12-02 15:10:25,145::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33726::INFO::2012-12-02 15:10:25,145::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33726::DEBUG::2012-12-02 15:10:25,146::task::1151::TaskManager.Task::(prepare) Task=`9abdc3cb-85e0-417d-8c69-d1ab308644e8`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33726::DEBUG::2012-12-02 15:10:25,146::task::568::TaskManager.Task::(_updateState) Task=`9abdc3cb-85e0-417d-8c69-d1ab308644e8`::moving from state preparing -> state finished >Thread-33726::DEBUG::2012-12-02 15:10:25,146::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33726::DEBUG::2012-12-02 15:10:25,146::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33726::DEBUG::2012-12-02 15:10:25,146::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33726::DEBUG::2012-12-02 15:10:25,146::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33726::DEBUG::2012-12-02 15:10:25,147::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33726::DEBUG::2012-12-02 15:10:25,147::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33726::DEBUG::2012-12-02 15:10:25,147::task::957::TaskManager.Task::(_decref) Task=`9abdc3cb-85e0-417d-8c69-d1ab308644e8`::ref 0 aborting False >Thread-33786::DEBUG::2012-12-02 15:10:26,256::task::568::TaskManager.Task::(_updateState) Task=`39196fed-afa0-4829-8467-85e1abec48e5`::moving from state init -> state preparing >Thread-33786::INFO::2012-12-02 15:10:26,256::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33786::INFO::2012-12-02 15:10:26,256::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00760412216187', 'lastCheck': 1354453819.673286, 'code': 0, 'valid': True}} >Thread-33786::DEBUG::2012-12-02 15:10:26,256::task::1151::TaskManager.Task::(prepare) Task=`39196fed-afa0-4829-8467-85e1abec48e5`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00760412216187', 'lastCheck': 1354453819.673286, 'code': 0, 'valid': True}} >Thread-33786::DEBUG::2012-12-02 15:10:26,257::task::568::TaskManager.Task::(_updateState) Task=`39196fed-afa0-4829-8467-85e1abec48e5`::moving from state preparing -> state finished >Thread-33786::DEBUG::2012-12-02 15:10:26,257::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33786::DEBUG::2012-12-02 15:10:26,257::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33786::DEBUG::2012-12-02 15:10:26,257::task::957::TaskManager.Task::(_decref) Task=`39196fed-afa0-4829-8467-85e1abec48e5`::ref 0 aborting False >Thread-33792::DEBUG::2012-12-02 15:10:35,223::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33792::DEBUG::2012-12-02 15:10:35,223::task::568::TaskManager.Task::(_updateState) Task=`2bdf261e-925b-4fb8-9331-e05f6873c2f7`::moving from state init -> state preparing >Thread-33792::INFO::2012-12-02 15:10:35,224::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33792::INFO::2012-12-02 15:10:35,224::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33792::DEBUG::2012-12-02 15:10:35,224::task::1151::TaskManager.Task::(prepare) Task=`2bdf261e-925b-4fb8-9331-e05f6873c2f7`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33792::DEBUG::2012-12-02 15:10:35,224::task::568::TaskManager.Task::(_updateState) Task=`2bdf261e-925b-4fb8-9331-e05f6873c2f7`::moving from state preparing -> state finished >Thread-33792::DEBUG::2012-12-02 15:10:35,224::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33792::DEBUG::2012-12-02 15:10:35,224::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33792::DEBUG::2012-12-02 15:10:35,224::task::957::TaskManager.Task::(_decref) Task=`2bdf261e-925b-4fb8-9331-e05f6873c2f7`::ref 0 aborting False >Thread-33793::DEBUG::2012-12-02 15:10:35,231::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33793::DEBUG::2012-12-02 15:10:35,231::task::568::TaskManager.Task::(_updateState) Task=`0c68f8eb-6b5c-4fd0-8fe0-aafade9f29d6`::moving from state init -> state preparing >Thread-33793::INFO::2012-12-02 15:10:35,231::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33793::DEBUG::2012-12-02 15:10:35,232::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`facb3227-015d-469b-b538-d0771cb5b5cc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33793::DEBUG::2012-12-02 15:10:35,232::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33793::DEBUG::2012-12-02 15:10:35,232::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33793::DEBUG::2012-12-02 15:10:35,232::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`facb3227-015d-469b-b538-d0771cb5b5cc`::Granted request >Thread-33793::DEBUG::2012-12-02 15:10:35,232::task::794::TaskManager.Task::(resourceAcquired) Task=`0c68f8eb-6b5c-4fd0-8fe0-aafade9f29d6`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33793::DEBUG::2012-12-02 15:10:35,232::task::957::TaskManager.Task::(_decref) Task=`0c68f8eb-6b5c-4fd0-8fe0-aafade9f29d6`::ref 1 aborting False >Thread-33793::DEBUG::2012-12-02 15:10:35,238::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33793::DEBUG::2012-12-02 15:10:35,238::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33793::DEBUG::2012-12-02 15:10:35,445::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33793::WARNING::2012-12-02 15:10:35,447::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33793::DEBUG::2012-12-02 15:10:35,447::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33793::WARNING::2012-12-02 15:10:35,477::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33793::INFO::2012-12-02 15:10:35,477::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33793::DEBUG::2012-12-02 15:10:35,477::task::1151::TaskManager.Task::(prepare) Task=`0c68f8eb-6b5c-4fd0-8fe0-aafade9f29d6`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33793::DEBUG::2012-12-02 15:10:35,478::task::568::TaskManager.Task::(_updateState) Task=`0c68f8eb-6b5c-4fd0-8fe0-aafade9f29d6`::moving from state preparing -> state finished >Thread-33793::DEBUG::2012-12-02 15:10:35,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33793::DEBUG::2012-12-02 15:10:35,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33793::DEBUG::2012-12-02 15:10:35,478::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33793::DEBUG::2012-12-02 15:10:35,478::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33793::DEBUG::2012-12-02 15:10:35,478::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33793::DEBUG::2012-12-02 15:10:35,479::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33793::DEBUG::2012-12-02 15:10:35,479::task::957::TaskManager.Task::(_decref) Task=`0c68f8eb-6b5c-4fd0-8fe0-aafade9f29d6`::ref 0 aborting False >Thread-33853::DEBUG::2012-12-02 15:10:36,374::task::568::TaskManager.Task::(_updateState) Task=`f7327ab3-b766-4f12-b82a-afd8654a1cdf`::moving from state init -> state preparing >Thread-33853::INFO::2012-12-02 15:10:36,374::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33853::INFO::2012-12-02 15:10:36,374::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0112411975861', 'lastCheck': 1354453829.695277, 'code': 0, 'valid': True}} >Thread-33853::DEBUG::2012-12-02 15:10:36,374::task::1151::TaskManager.Task::(prepare) Task=`f7327ab3-b766-4f12-b82a-afd8654a1cdf`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0112411975861', 'lastCheck': 1354453829.695277, 'code': 0, 'valid': True}} >Thread-33853::DEBUG::2012-12-02 15:10:36,375::task::568::TaskManager.Task::(_updateState) Task=`f7327ab3-b766-4f12-b82a-afd8654a1cdf`::moving from state preparing -> state finished >Thread-33853::DEBUG::2012-12-02 15:10:36,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33853::DEBUG::2012-12-02 15:10:36,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33853::DEBUG::2012-12-02 15:10:36,375::task::957::TaskManager.Task::(_decref) Task=`f7327ab3-b766-4f12-b82a-afd8654a1cdf`::ref 0 aborting False >Thread-33859::DEBUG::2012-12-02 15:10:45,558::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33859::DEBUG::2012-12-02 15:10:45,558::task::568::TaskManager.Task::(_updateState) Task=`c6e47f4f-5cdc-421f-89a4-c0679613b26e`::moving from state init -> state preparing >Thread-33859::INFO::2012-12-02 15:10:45,559::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33859::INFO::2012-12-02 15:10:45,559::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33859::DEBUG::2012-12-02 15:10:45,559::task::1151::TaskManager.Task::(prepare) Task=`c6e47f4f-5cdc-421f-89a4-c0679613b26e`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33859::DEBUG::2012-12-02 15:10:45,559::task::568::TaskManager.Task::(_updateState) Task=`c6e47f4f-5cdc-421f-89a4-c0679613b26e`::moving from state preparing -> state finished >Thread-33859::DEBUG::2012-12-02 15:10:45,559::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33859::DEBUG::2012-12-02 15:10:45,559::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33859::DEBUG::2012-12-02 15:10:45,560::task::957::TaskManager.Task::(_decref) Task=`c6e47f4f-5cdc-421f-89a4-c0679613b26e`::ref 0 aborting False >Thread-33860::DEBUG::2012-12-02 15:10:45,566::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33860::DEBUG::2012-12-02 15:10:45,567::task::568::TaskManager.Task::(_updateState) Task=`15015d96-7f3e-4c92-a588-2a3cfb2b8f3c`::moving from state init -> state preparing >Thread-33860::INFO::2012-12-02 15:10:45,567::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33860::DEBUG::2012-12-02 15:10:45,567::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`47fac930-803e-4749-b7e3-aa26b7a0c8e9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33860::DEBUG::2012-12-02 15:10:45,567::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33860::DEBUG::2012-12-02 15:10:45,567::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33860::DEBUG::2012-12-02 15:10:45,568::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`47fac930-803e-4749-b7e3-aa26b7a0c8e9`::Granted request >Thread-33860::DEBUG::2012-12-02 15:10:45,568::task::794::TaskManager.Task::(resourceAcquired) Task=`15015d96-7f3e-4c92-a588-2a3cfb2b8f3c`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33860::DEBUG::2012-12-02 15:10:45,568::task::957::TaskManager.Task::(_decref) Task=`15015d96-7f3e-4c92-a588-2a3cfb2b8f3c`::ref 1 aborting False >Thread-33860::DEBUG::2012-12-02 15:10:45,574::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33860::DEBUG::2012-12-02 15:10:45,575::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33860::DEBUG::2012-12-02 15:10:45,782::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33860::WARNING::2012-12-02 15:10:45,783::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33860::DEBUG::2012-12-02 15:10:45,784::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33860::WARNING::2012-12-02 15:10:45,813::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33860::INFO::2012-12-02 15:10:45,814::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33860::DEBUG::2012-12-02 15:10:45,814::task::1151::TaskManager.Task::(prepare) Task=`15015d96-7f3e-4c92-a588-2a3cfb2b8f3c`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33860::DEBUG::2012-12-02 15:10:45,814::task::568::TaskManager.Task::(_updateState) Task=`15015d96-7f3e-4c92-a588-2a3cfb2b8f3c`::moving from state preparing -> state finished >Thread-33860::DEBUG::2012-12-02 15:10:45,814::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33860::DEBUG::2012-12-02 15:10:45,814::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33860::DEBUG::2012-12-02 15:10:45,815::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33860::DEBUG::2012-12-02 15:10:45,815::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33860::DEBUG::2012-12-02 15:10:45,815::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33860::DEBUG::2012-12-02 15:10:45,815::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33860::DEBUG::2012-12-02 15:10:45,815::task::957::TaskManager.Task::(_decref) Task=`15015d96-7f3e-4c92-a588-2a3cfb2b8f3c`::ref 0 aborting False >Thread-33920::DEBUG::2012-12-02 15:10:46,485::task::568::TaskManager.Task::(_updateState) Task=`1fe57ebc-697d-41fd-9eec-91c4d2822b79`::moving from state init -> state preparing >Thread-33920::INFO::2012-12-02 15:10:46,486::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33920::INFO::2012-12-02 15:10:46,486::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00765514373779', 'lastCheck': 1354453839.713542, 'code': 0, 'valid': True}} >Thread-33920::DEBUG::2012-12-02 15:10:46,486::task::1151::TaskManager.Task::(prepare) Task=`1fe57ebc-697d-41fd-9eec-91c4d2822b79`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00765514373779', 'lastCheck': 1354453839.713542, 'code': 0, 'valid': True}} >Thread-33920::DEBUG::2012-12-02 15:10:46,486::task::568::TaskManager.Task::(_updateState) Task=`1fe57ebc-697d-41fd-9eec-91c4d2822b79`::moving from state preparing -> state finished >Thread-33920::DEBUG::2012-12-02 15:10:46,486::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33920::DEBUG::2012-12-02 15:10:46,486::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33920::DEBUG::2012-12-02 15:10:46,486::task::957::TaskManager.Task::(_decref) Task=`1fe57ebc-697d-41fd-9eec-91c4d2822b79`::ref 0 aborting False >Thread-33926::DEBUG::2012-12-02 15:10:55,892::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33926::DEBUG::2012-12-02 15:10:55,893::task::568::TaskManager.Task::(_updateState) Task=`c6af1da5-e97f-439a-8046-35bb34a1ca07`::moving from state init -> state preparing >Thread-33926::INFO::2012-12-02 15:10:55,893::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33926::INFO::2012-12-02 15:10:55,893::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33926::DEBUG::2012-12-02 15:10:55,893::task::1151::TaskManager.Task::(prepare) Task=`c6af1da5-e97f-439a-8046-35bb34a1ca07`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33926::DEBUG::2012-12-02 15:10:55,893::task::568::TaskManager.Task::(_updateState) Task=`c6af1da5-e97f-439a-8046-35bb34a1ca07`::moving from state preparing -> state finished >Thread-33926::DEBUG::2012-12-02 15:10:55,894::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33926::DEBUG::2012-12-02 15:10:55,894::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33926::DEBUG::2012-12-02 15:10:55,894::task::957::TaskManager.Task::(_decref) Task=`c6af1da5-e97f-439a-8046-35bb34a1ca07`::ref 0 aborting False >Thread-33927::DEBUG::2012-12-02 15:10:55,900::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33927::DEBUG::2012-12-02 15:10:55,901::task::568::TaskManager.Task::(_updateState) Task=`c6df7e60-c142-4163-bcc2-65dab62dee4c`::moving from state init -> state preparing >Thread-33927::INFO::2012-12-02 15:10:55,901::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33927::DEBUG::2012-12-02 15:10:55,901::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`d2037beb-3e2f-4ec9-986c-ce77a4833345`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33927::DEBUG::2012-12-02 15:10:55,901::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33927::DEBUG::2012-12-02 15:10:55,901::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33927::DEBUG::2012-12-02 15:10:55,901::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`d2037beb-3e2f-4ec9-986c-ce77a4833345`::Granted request >Thread-33927::DEBUG::2012-12-02 15:10:55,901::task::794::TaskManager.Task::(resourceAcquired) Task=`c6df7e60-c142-4163-bcc2-65dab62dee4c`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33927::DEBUG::2012-12-02 15:10:55,901::task::957::TaskManager.Task::(_decref) Task=`c6df7e60-c142-4163-bcc2-65dab62dee4c`::ref 1 aborting False >Thread-33927::DEBUG::2012-12-02 15:10:55,906::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33927::DEBUG::2012-12-02 15:10:55,907::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33927::DEBUG::2012-12-02 15:10:56,115::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33927::WARNING::2012-12-02 15:10:56,116::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33927::DEBUG::2012-12-02 15:10:56,117::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33927::WARNING::2012-12-02 15:10:56,148::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33927::INFO::2012-12-02 15:10:56,149::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33927::DEBUG::2012-12-02 15:10:56,149::task::1151::TaskManager.Task::(prepare) Task=`c6df7e60-c142-4163-bcc2-65dab62dee4c`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33927::DEBUG::2012-12-02 15:10:56,149::task::568::TaskManager.Task::(_updateState) Task=`c6df7e60-c142-4163-bcc2-65dab62dee4c`::moving from state preparing -> state finished >Thread-33927::DEBUG::2012-12-02 15:10:56,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33927::DEBUG::2012-12-02 15:10:56,149::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33927::DEBUG::2012-12-02 15:10:56,150::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33927::DEBUG::2012-12-02 15:10:56,150::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33927::DEBUG::2012-12-02 15:10:56,150::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33927::DEBUG::2012-12-02 15:10:56,150::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33927::DEBUG::2012-12-02 15:10:56,150::task::957::TaskManager.Task::(_decref) Task=`c6df7e60-c142-4163-bcc2-65dab62dee4c`::ref 0 aborting False >Thread-33987::DEBUG::2012-12-02 15:10:56,664::task::568::TaskManager.Task::(_updateState) Task=`f9cb178d-3051-4d99-a9e8-ad503cd63fa9`::moving from state init -> state preparing >Thread-33987::INFO::2012-12-02 15:10:56,665::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-33987::INFO::2012-12-02 15:10:56,665::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.011549949646', 'lastCheck': 1354453849.7354441, 'code': 0, 'valid': True}} >Thread-33987::DEBUG::2012-12-02 15:10:56,665::task::1151::TaskManager.Task::(prepare) Task=`f9cb178d-3051-4d99-a9e8-ad503cd63fa9`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.011549949646', 'lastCheck': 1354453849.7354441, 'code': 0, 'valid': True}} >Thread-33987::DEBUG::2012-12-02 15:10:56,665::task::568::TaskManager.Task::(_updateState) Task=`f9cb178d-3051-4d99-a9e8-ad503cd63fa9`::moving from state preparing -> state finished >Thread-33987::DEBUG::2012-12-02 15:10:56,665::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33987::DEBUG::2012-12-02 15:10:56,665::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33987::DEBUG::2012-12-02 15:10:56,665::task::957::TaskManager.Task::(_decref) Task=`f9cb178d-3051-4d99-a9e8-ad503cd63fa9`::ref 0 aborting False >Thread-33993::DEBUG::2012-12-02 15:11:06,227::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33993::DEBUG::2012-12-02 15:11:06,227::task::568::TaskManager.Task::(_updateState) Task=`171b86ec-4043-4fbc-a4db-06a841ee131e`::moving from state init -> state preparing >Thread-33993::INFO::2012-12-02 15:11:06,228::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33993::INFO::2012-12-02 15:11:06,228::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33993::DEBUG::2012-12-02 15:11:06,228::task::1151::TaskManager.Task::(prepare) Task=`171b86ec-4043-4fbc-a4db-06a841ee131e`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-33993::DEBUG::2012-12-02 15:11:06,228::task::568::TaskManager.Task::(_updateState) Task=`171b86ec-4043-4fbc-a4db-06a841ee131e`::moving from state preparing -> state finished >Thread-33993::DEBUG::2012-12-02 15:11:06,228::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-33993::DEBUG::2012-12-02 15:11:06,228::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33993::DEBUG::2012-12-02 15:11:06,228::task::957::TaskManager.Task::(_decref) Task=`171b86ec-4043-4fbc-a4db-06a841ee131e`::ref 0 aborting False >Thread-33994::DEBUG::2012-12-02 15:11:06,235::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-33994::DEBUG::2012-12-02 15:11:06,236::task::568::TaskManager.Task::(_updateState) Task=`5450601e-037f-44bb-a9cf-cf8cba01a059`::moving from state init -> state preparing >Thread-33994::INFO::2012-12-02 15:11:06,236::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-33994::DEBUG::2012-12-02 15:11:06,236::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`69b71100-61e7-4a6a-86f4-c67c43974070`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-33994::DEBUG::2012-12-02 15:11:06,236::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-33994::DEBUG::2012-12-02 15:11:06,236::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-33994::DEBUG::2012-12-02 15:11:06,237::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`69b71100-61e7-4a6a-86f4-c67c43974070`::Granted request >Thread-33994::DEBUG::2012-12-02 15:11:06,237::task::794::TaskManager.Task::(resourceAcquired) Task=`5450601e-037f-44bb-a9cf-cf8cba01a059`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-33994::DEBUG::2012-12-02 15:11:06,237::task::957::TaskManager.Task::(_decref) Task=`5450601e-037f-44bb-a9cf-cf8cba01a059`::ref 1 aborting False >Thread-33994::DEBUG::2012-12-02 15:11:06,243::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-33994::DEBUG::2012-12-02 15:11:06,244::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-33994::DEBUG::2012-12-02 15:11:06,449::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-33994::WARNING::2012-12-02 15:11:06,450::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-33994::DEBUG::2012-12-02 15:11:06,451::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33994::WARNING::2012-12-02 15:11:06,479::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-33994::INFO::2012-12-02 15:11:06,480::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33994::DEBUG::2012-12-02 15:11:06,480::task::1151::TaskManager.Task::(prepare) Task=`5450601e-037f-44bb-a9cf-cf8cba01a059`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-33994::DEBUG::2012-12-02 15:11:06,480::task::568::TaskManager.Task::(_updateState) Task=`5450601e-037f-44bb-a9cf-cf8cba01a059`::moving from state preparing -> state finished >Thread-33994::DEBUG::2012-12-02 15:11:06,480::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-33994::DEBUG::2012-12-02 15:11:06,480::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-33994::DEBUG::2012-12-02 15:11:06,480::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-33994::DEBUG::2012-12-02 15:11:06,481::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-33994::DEBUG::2012-12-02 15:11:06,481::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-33994::DEBUG::2012-12-02 15:11:06,481::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-33994::DEBUG::2012-12-02 15:11:06,481::task::957::TaskManager.Task::(_decref) Task=`5450601e-037f-44bb-a9cf-cf8cba01a059`::ref 0 aborting False >Thread-34054::DEBUG::2012-12-02 15:11:06,777::task::568::TaskManager.Task::(_updateState) Task=`fc616a40-c05f-49f8-9a6a-33e15277fb9f`::moving from state init -> state preparing >Thread-34054::INFO::2012-12-02 15:11:06,778::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34054::INFO::2012-12-02 15:11:06,778::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101990699768', 'lastCheck': 1354453859.7555809, 'code': 0, 'valid': True}} >Thread-34054::DEBUG::2012-12-02 15:11:06,778::task::1151::TaskManager.Task::(prepare) Task=`fc616a40-c05f-49f8-9a6a-33e15277fb9f`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0101990699768', 'lastCheck': 1354453859.7555809, 'code': 0, 'valid': True}} >Thread-34054::DEBUG::2012-12-02 15:11:06,778::task::568::TaskManager.Task::(_updateState) Task=`fc616a40-c05f-49f8-9a6a-33e15277fb9f`::moving from state preparing -> state finished >Thread-34054::DEBUG::2012-12-02 15:11:06,778::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34054::DEBUG::2012-12-02 15:11:06,778::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34054::DEBUG::2012-12-02 15:11:06,778::task::957::TaskManager.Task::(_decref) Task=`fc616a40-c05f-49f8-9a6a-33e15277fb9f`::ref 0 aborting False >Thread-34060::DEBUG::2012-12-02 15:11:16,567::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34060::DEBUG::2012-12-02 15:11:16,567::task::568::TaskManager.Task::(_updateState) Task=`9d0576cc-9191-4f86-8de6-8bfb7a97caf0`::moving from state init -> state preparing >Thread-34060::INFO::2012-12-02 15:11:16,568::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34060::INFO::2012-12-02 15:11:16,568::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34060::DEBUG::2012-12-02 15:11:16,568::task::1151::TaskManager.Task::(prepare) Task=`9d0576cc-9191-4f86-8de6-8bfb7a97caf0`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34060::DEBUG::2012-12-02 15:11:16,568::task::568::TaskManager.Task::(_updateState) Task=`9d0576cc-9191-4f86-8de6-8bfb7a97caf0`::moving from state preparing -> state finished >Thread-34060::DEBUG::2012-12-02 15:11:16,568::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34060::DEBUG::2012-12-02 15:11:16,568::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34060::DEBUG::2012-12-02 15:11:16,568::task::957::TaskManager.Task::(_decref) Task=`9d0576cc-9191-4f86-8de6-8bfb7a97caf0`::ref 0 aborting False >Thread-34061::DEBUG::2012-12-02 15:11:16,575::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34061::DEBUG::2012-12-02 15:11:16,575::task::568::TaskManager.Task::(_updateState) Task=`7ab4b484-bedc-4bde-b1dc-8842e5b95d9e`::moving from state init -> state preparing >Thread-34061::INFO::2012-12-02 15:11:16,576::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34061::DEBUG::2012-12-02 15:11:16,576::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`22487035-36b1-41b4-97a7-ea168dc61781`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34061::DEBUG::2012-12-02 15:11:16,576::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34061::DEBUG::2012-12-02 15:11:16,576::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34061::DEBUG::2012-12-02 15:11:16,576::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`22487035-36b1-41b4-97a7-ea168dc61781`::Granted request >Thread-34061::DEBUG::2012-12-02 15:11:16,577::task::794::TaskManager.Task::(resourceAcquired) Task=`7ab4b484-bedc-4bde-b1dc-8842e5b95d9e`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34061::DEBUG::2012-12-02 15:11:16,577::task::957::TaskManager.Task::(_decref) Task=`7ab4b484-bedc-4bde-b1dc-8842e5b95d9e`::ref 1 aborting False >Thread-34061::DEBUG::2012-12-02 15:11:16,583::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34061::DEBUG::2012-12-02 15:11:16,583::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34061::DEBUG::2012-12-02 15:11:16,788::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34061::WARNING::2012-12-02 15:11:16,789::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34061::DEBUG::2012-12-02 15:11:16,790::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34061::WARNING::2012-12-02 15:11:16,824::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34061::INFO::2012-12-02 15:11:16,824::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34061::DEBUG::2012-12-02 15:11:16,824::task::1151::TaskManager.Task::(prepare) Task=`7ab4b484-bedc-4bde-b1dc-8842e5b95d9e`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': '4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34061::DEBUG::2012-12-02 15:11:16,825::task::568::TaskManager.Task::(_updateState) Task=`7ab4b484-bedc-4bde-b1dc-8842e5b95d9e`::moving from state preparing -> state finished >Thread-34061::DEBUG::2012-12-02 15:11:16,825::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34061::DEBUG::2012-12-02 15:11:16,825::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34061::DEBUG::2012-12-02 15:11:16,825::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34061::DEBUG::2012-12-02 15:11:16,825::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34061::DEBUG::2012-12-02 15:11:16,825::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34061::DEBUG::2012-12-02 15:11:16,826::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34061::DEBUG::2012-12-02 15:11:16,826::task::957::TaskManager.Task::(_decref) Task=`7ab4b484-bedc-4bde-b1dc-8842e5b95d9e`::ref 0 aborting False >Thread-34121::DEBUG::2012-12-02 15:11:16,891::task::568::TaskManager.Task::(_updateState) Task=`e204bf90-3053-42ed-9b8b-d06a3fd1ab8e`::moving from state init -> state preparing >Thread-34121::INFO::2012-12-02 15:11:16,891::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34121::INFO::2012-12-02 15:11:16,892::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00693488121033', 'lastCheck': 1354453869.773699, 'code': 0, 'valid': True}} >Thread-34121::DEBUG::2012-12-02 15:11:16,892::task::1151::TaskManager.Task::(prepare) Task=`e204bf90-3053-42ed-9b8b-d06a3fd1ab8e`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00693488121033', 'lastCheck': 1354453869.773699, 'code': 0, 'valid': True}} >Thread-34121::DEBUG::2012-12-02 15:11:16,892::task::568::TaskManager.Task::(_updateState) Task=`e204bf90-3053-42ed-9b8b-d06a3fd1ab8e`::moving from state preparing -> state finished >Thread-34121::DEBUG::2012-12-02 15:11:16,892::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34121::DEBUG::2012-12-02 15:11:16,892::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34121::DEBUG::2012-12-02 15:11:16,892::task::957::TaskManager.Task::(_decref) Task=`e204bf90-3053-42ed-9b8b-d06a3fd1ab8e`::ref 0 aborting False >Thread-34124::DEBUG::2012-12-02 15:11:19,679::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34124::DEBUG::2012-12-02 15:11:19,680::task::568::TaskManager.Task::(_updateState) Task=`627343a2-a282-4824-a955-8a3c74742bd1`::moving from state init -> state preparing >Thread-34124::INFO::2012-12-02 15:11:19,680::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-34124::INFO::2012-12-02 15:11:19,680::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-34124::DEBUG::2012-12-02 15:11:19,680::task::1151::TaskManager.Task::(prepare) Task=`627343a2-a282-4824-a955-8a3c74742bd1`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-34124::DEBUG::2012-12-02 15:11:19,680::task::568::TaskManager.Task::(_updateState) Task=`627343a2-a282-4824-a955-8a3c74742bd1`::moving from state preparing -> state finished >Thread-34124::DEBUG::2012-12-02 15:11:19,680::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34124::DEBUG::2012-12-02 15:11:19,681::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34124::DEBUG::2012-12-02 15:11:19,681::task::957::TaskManager.Task::(_decref) Task=`627343a2-a282-4824-a955-8a3c74742bd1`::ref 0 aborting False >Thread-34125::DEBUG::2012-12-02 15:11:19,728::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34125::DEBUG::2012-12-02 15:11:19,728::task::568::TaskManager.Task::(_updateState) Task=`315fcb1a-f2e4-4783-b160-dbfdf5aacca8`::moving from state init -> state preparing >Thread-34125::INFO::2012-12-02 15:11:19,728::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-34125::DEBUG::2012-12-02 15:11:19,734::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510' (cwd None) >Thread-34125::DEBUG::2012-12-02 15:11:19,860::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34125::DEBUG::2012-12-02 15:11:19,861::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34125::DEBUG::2012-12-02 15:11:19,861::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34125::DEBUG::2012-12-02 15:11:19,861::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34125::DEBUG::2012-12-02 15:11:19,861::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34125::DEBUG::2012-12-02 15:11:19,861::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34125::INFO::2012-12-02 15:11:19,862::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-34125::DEBUG::2012-12-02 15:11:19,862::task::1151::TaskManager.Task::(prepare) Task=`315fcb1a-f2e4-4783-b160-dbfdf5aacca8`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-34125::DEBUG::2012-12-02 15:11:19,862::task::568::TaskManager.Task::(_updateState) Task=`315fcb1a-f2e4-4783-b160-dbfdf5aacca8`::moving from state preparing -> state finished >Thread-34125::DEBUG::2012-12-02 15:11:19,862::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34125::DEBUG::2012-12-02 15:11:19,862::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34125::DEBUG::2012-12-02 15:11:19,862::task::957::TaskManager.Task::(_decref) Task=`315fcb1a-f2e4-4783-b160-dbfdf5aacca8`::ref 0 aborting False >Thread-34126::DEBUG::2012-12-02 15:11:19,990::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34126::DEBUG::2012-12-02 15:11:19,990::task::568::TaskManager.Task::(_updateState) Task=`fad9f595-5af5-468a-b4ba-9b13a0fd503c`::moving from state init -> state preparing >Thread-34126::INFO::2012-12-02 15:11:19,991::logUtils::37::dispatcher::(wrapper) Run and protect: createStorageDomain(storageType=1, sdUUID='fafd959d-695f-4b5e-aa73-aaa2634ae743', domainName='a3', typeSpecificArg='multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', domClass=1, domVersion='3', options=None) >Thread-34126::DEBUG::2012-12-02 15:11:19,991::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-34126::DEBUG::2012-12-02 15:11:19,991::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34126::DEBUG::2012-12-02 15:11:19,991::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-34126::DEBUG::2012-12-02 15:11:19,991::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34126::DEBUG::2012-12-02 15:11:19,991::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-34126::DEBUG::2012-12-02 15:11:20,020::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-34126::DEBUG::2012-12-02 15:11:20,020::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34126::DEBUG::2012-12-02 15:11:20,652::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-34126::DEBUG::2012-12-02 15:11:20,741::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-34126::DEBUG::2012-12-02 15:11:20,741::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,742::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,742::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,742::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,742::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,742::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,742::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34126::DEBUG::2012-12-02 15:11:20,743::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34126::DEBUG::2012-12-02 15:11:20,745::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free fafd959d-695f-4b5e-aa73-aaa2634ae743' (cwd None) >Thread-34126::DEBUG::2012-12-02 15:11:20,955::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "fafd959d-695f-4b5e-aa73-aaa2634ae743" not found\n'; <rc> = 5 >Thread-34126::WARNING::2012-12-02 15:11:20,957::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "fafd959d-695f-4b5e-aa73-aaa2634ae743" not found'] >Thread-34126::DEBUG::2012-12-02 15:11:20,957::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34126::INFO::2012-12-02 15:11:20,988::nfsSD::64::Storage.StorageDomain::(create) sdUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743 domainName=a3 remotePath=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510 domClass=1 >Thread-34126::DEBUG::2012-12-02 15:11:21,208::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-34126::DEBUG::2012-12-02 15:11:21,209::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-34126::WARNING::2012-12-02 15:11:21,209::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-34126::DEBUG::2012-12-02 15:11:21,209::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-34126::DEBUG::2012-12-02 15:11:21,210::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-34126::DEBUG::2012-12-02 15:11:21,210::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'ROLE=Regular', 'SDUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=97136b77fee69c55cafb0d52fb03c618bd8dc8ac'] >Thread-34126::DEBUG::2012-12-02 15:11:21,245::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-34126::DEBUG::2012-12-02 15:11:21,258::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510/fafd959d-695f-4b5e-aa73-aaa2634ae743 >Thread-34126::DEBUG::2012-12-02 15:11:21,258::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-34126::DEBUG::2012-12-02 15:11:21,271::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'ROLE=Regular', 'SDUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=97136b77fee69c55cafb0d52fb03c618bd8dc8ac'] >Thread-34126::DEBUG::2012-12-02 15:11:21,276::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-34126::DEBUG::2012-12-02 15:11:21,276::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'fafd959d-695f-4b5e-aa73-aaa2634ae743_imageNS' >Thread-34126::DEBUG::2012-12-02 15:11:21,277::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'fafd959d-695f-4b5e-aa73-aaa2634ae743_volumeNS' >Thread-34126::DEBUG::2012-12-02 15:11:21,490::sd::389::Storage.StorageDomain::(initSPMlease) lease initialized successfully >Thread-34126::INFO::2012-12-02 15:11:21,490::logUtils::39::dispatcher::(wrapper) Run and protect: createStorageDomain, Return response: None >Thread-34126::DEBUG::2012-12-02 15:11:21,491::task::1151::TaskManager.Task::(prepare) Task=`fad9f595-5af5-468a-b4ba-9b13a0fd503c`::finished: None >Thread-34126::DEBUG::2012-12-02 15:11:21,491::task::568::TaskManager.Task::(_updateState) Task=`fad9f595-5af5-468a-b4ba-9b13a0fd503c`::moving from state preparing -> state finished >Thread-34126::DEBUG::2012-12-02 15:11:21,491::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34126::DEBUG::2012-12-02 15:11:21,491::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34126::DEBUG::2012-12-02 15:11:21,491::task::957::TaskManager.Task::(_decref) Task=`fad9f595-5af5-468a-b4ba-9b13a0fd503c`::ref 0 aborting False >Thread-34188::DEBUG::2012-12-02 15:11:21,513::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34188::DEBUG::2012-12-02 15:11:21,514::task::568::TaskManager.Task::(_updateState) Task=`920ca7ad-854f-47d8-812b-63110ffee677`::moving from state init -> state preparing >Thread-34188::INFO::2012-12-02 15:11:21,514::logUtils::37::dispatcher::(wrapper) Run and protect: getStorageDomainStats(sdUUID='fafd959d-695f-4b5e-aa73-aaa2634ae743', options=None) >Thread-34188::DEBUG::2012-12-02 15:11:21,514::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743`ReqID=`c007ee5d-e813-4103-96b1-c77057bb2d81`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34188::DEBUG::2012-12-02 15:11:21,514::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' for lock type 'shared' >Thread-34188::DEBUG::2012-12-02 15:11:21,514::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' is free. Now locking as 'shared' (1 active user) >Thread-34188::DEBUG::2012-12-02 15:11:21,515::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743`ReqID=`c007ee5d-e813-4103-96b1-c77057bb2d81`::Granted request >Thread-34188::DEBUG::2012-12-02 15:11:21,515::task::794::TaskManager.Task::(resourceAcquired) Task=`920ca7ad-854f-47d8-812b-63110ffee677`::_resourcesAcquired: Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743 (shared) >Thread-34188::DEBUG::2012-12-02 15:11:21,515::task::957::TaskManager.Task::(_decref) Task=`920ca7ad-854f-47d8-812b-63110ffee677`::ref 1 aborting False >Thread-34188::INFO::2012-12-02 15:11:21,517::logUtils::39::dispatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '45385515008', 'disktotal': '53687091200', 'mdafree': 0}} >Thread-34188::DEBUG::2012-12-02 15:11:21,517::task::1151::TaskManager.Task::(prepare) Task=`920ca7ad-854f-47d8-812b-63110ffee677`::finished: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '45385515008', 'disktotal': '53687091200', 'mdafree': 0}} >Thread-34188::DEBUG::2012-12-02 15:11:21,517::task::568::TaskManager.Task::(_updateState) Task=`920ca7ad-854f-47d8-812b-63110ffee677`::moving from state preparing -> state finished >Thread-34188::DEBUG::2012-12-02 15:11:21,517::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743': < ResourceRef 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743', isValid: 'True' obj: 'None'>} >Thread-34188::DEBUG::2012-12-02 15:11:21,517::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34188::DEBUG::2012-12-02 15:11:21,518::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' >Thread-34188::DEBUG::2012-12-02 15:11:21,518::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' (0 active users) >Thread-34188::DEBUG::2012-12-02 15:11:21,518::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' is free, finding out if anyone is waiting for it. >Thread-34188::DEBUG::2012-12-02 15:11:21,518::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743', Clearing records. >Thread-34188::DEBUG::2012-12-02 15:11:21,518::task::957::TaskManager.Task::(_decref) Task=`920ca7ad-854f-47d8-812b-63110ffee677`::ref 0 aborting False >Thread-34189::DEBUG::2012-12-02 15:11:21,664::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34189::DEBUG::2012-12-02 15:11:21,665::task::568::TaskManager.Task::(_updateState) Task=`81d6a508-9b23-4243-a0e0-ed266d4eb474`::moving from state init -> state preparing >Thread-34189::INFO::2012-12-02 15:11:21,665::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-34189::DEBUG::2012-12-02 15:11:21,666::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510' (cwd None) >Thread-34189::DEBUG::2012-12-02 15:11:21,695::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-34189::DEBUG::2012-12-02 15:11:21,695::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34189::DEBUG::2012-12-02 15:11:21,695::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-34189::DEBUG::2012-12-02 15:11:21,695::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34189::DEBUG::2012-12-02 15:11:21,695::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-34189::DEBUG::2012-12-02 15:11:21,721::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-34189::DEBUG::2012-12-02 15:11:21,721::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34189::DEBUG::2012-12-02 15:11:22,363::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-34189::DEBUG::2012-12-02 15:11:22,440::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-34189::DEBUG::2012-12-02 15:11:22,440::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34189::DEBUG::2012-12-02 15:11:22,441::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34189::DEBUG::2012-12-02 15:11:22,441::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34189::DEBUG::2012-12-02 15:11:22,441::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34189::DEBUG::2012-12-02 15:11:22,441::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34189::DEBUG::2012-12-02 15:11:22,441::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34189::DEBUG::2012-12-02 15:11:22,441::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34189::INFO::2012-12-02 15:11:22,442::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-34189::DEBUG::2012-12-02 15:11:22,442::task::1151::TaskManager.Task::(prepare) Task=`81d6a508-9b23-4243-a0e0-ed266d4eb474`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-34189::DEBUG::2012-12-02 15:11:22,442::task::568::TaskManager.Task::(_updateState) Task=`81d6a508-9b23-4243-a0e0-ed266d4eb474`::moving from state preparing -> state finished >Thread-34189::DEBUG::2012-12-02 15:11:22,442::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34189::DEBUG::2012-12-02 15:11:22,442::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34189::DEBUG::2012-12-02 15:11:22,442::task::957::TaskManager.Task::(_decref) Task=`81d6a508-9b23-4243-a0e0-ed266d4eb474`::ref 0 aborting False >Thread-34190::DEBUG::2012-12-02 15:11:22,649::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34190::DEBUG::2012-12-02 15:11:22,650::task::568::TaskManager.Task::(_updateState) Task=`67a1a51a-df5e-4d09-aaf3-766787f1ffdd`::moving from state init -> state preparing >Thread-34190::INFO::2012-12-02 15:11:22,650::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '6210b428-99d8-4d2f-898d-ef16269344ca', 'port': ''}], options=None) >Thread-34190::INFO::2012-12-02 15:11:22,650::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '6210b428-99d8-4d2f-898d-ef16269344ca'}]} >Thread-34190::DEBUG::2012-12-02 15:11:22,650::task::1151::TaskManager.Task::(prepare) Task=`67a1a51a-df5e-4d09-aaf3-766787f1ffdd`::finished: {'statuslist': [{'status': 0, 'id': '6210b428-99d8-4d2f-898d-ef16269344ca'}]} >Thread-34190::DEBUG::2012-12-02 15:11:22,650::task::568::TaskManager.Task::(_updateState) Task=`67a1a51a-df5e-4d09-aaf3-766787f1ffdd`::moving from state preparing -> state finished >Thread-34190::DEBUG::2012-12-02 15:11:22,650::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34190::DEBUG::2012-12-02 15:11:22,650::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34190::DEBUG::2012-12-02 15:11:22,651::task::957::TaskManager.Task::(_decref) Task=`67a1a51a-df5e-4d09-aaf3-766787f1ffdd`::ref 0 aborting False >Thread-34191::DEBUG::2012-12-02 15:11:22,660::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34191::DEBUG::2012-12-02 15:11:22,661::task::568::TaskManager.Task::(_updateState) Task=`2d410291-4cc5-40c3-b9a3-8d3a57436abc`::moving from state init -> state preparing >Thread-34191::INFO::2012-12-02 15:11:22,661::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '6210b428-99d8-4d2f-898d-ef16269344ca', 'port': ''}], options=None) >Thread-34191::DEBUG::2012-12-02 15:11:22,667::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510' (cwd None) >Thread-34191::DEBUG::2012-12-02 15:11:22,778::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34191::DEBUG::2012-12-02 15:11:22,779::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34191::DEBUG::2012-12-02 15:11:22,779::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34191::DEBUG::2012-12-02 15:11:22,779::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34191::DEBUG::2012-12-02 15:11:22,779::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34191::DEBUG::2012-12-02 15:11:22,779::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34191::INFO::2012-12-02 15:11:22,780::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '6210b428-99d8-4d2f-898d-ef16269344ca'}]} >Thread-34191::DEBUG::2012-12-02 15:11:22,780::task::1151::TaskManager.Task::(prepare) Task=`2d410291-4cc5-40c3-b9a3-8d3a57436abc`::finished: {'statuslist': [{'status': 0, 'id': '6210b428-99d8-4d2f-898d-ef16269344ca'}]} >Thread-34191::DEBUG::2012-12-02 15:11:22,780::task::568::TaskManager.Task::(_updateState) Task=`2d410291-4cc5-40c3-b9a3-8d3a57436abc`::moving from state preparing -> state finished >Thread-34191::DEBUG::2012-12-02 15:11:22,780::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34191::DEBUG::2012-12-02 15:11:22,780::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34191::DEBUG::2012-12-02 15:11:22,780::task::957::TaskManager.Task::(_decref) Task=`2d410291-4cc5-40c3-b9a3-8d3a57436abc`::ref 0 aborting False >Thread-34192::DEBUG::2012-12-02 15:11:22,787::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34192::DEBUG::2012-12-02 15:11:22,787::task::568::TaskManager.Task::(_updateState) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::moving from state init -> state preparing >Thread-34192::INFO::2012-12-02 15:11:22,788::logUtils::37::dispatcher::(wrapper) Run and protect: attachStorageDomain(sdUUID='fafd959d-695f-4b5e-aa73-aaa2634ae743', spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34192::DEBUG::2012-12-02 15:11:22,788::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`62a89152-f017-4477-bd17-2e0639d39833`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34192::DEBUG::2012-12-02 15:11:22,788::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-34192::DEBUG::2012-12-02 15:11:22,788::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-34192::DEBUG::2012-12-02 15:11:22,789::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`62a89152-f017-4477-bd17-2e0639d39833`::Granted request >Thread-34192::DEBUG::2012-12-02 15:11:22,789::task::794::TaskManager.Task::(resourceAcquired) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (exclusive) >Thread-34192::DEBUG::2012-12-02 15:11:22,789::task::957::TaskManager.Task::(_decref) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::ref 1 aborting False >Thread-34192::DEBUG::2012-12-02 15:11:22,789::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743`ReqID=`83397305-66cb-4fab-9d97-809e9c6ddd28`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34192::DEBUG::2012-12-02 15:11:22,789::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' for lock type 'exclusive' >Thread-34192::DEBUG::2012-12-02 15:11:22,790::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' is free. Now locking as 'exclusive' (1 active user) >Thread-34192::DEBUG::2012-12-02 15:11:22,790::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743`ReqID=`83397305-66cb-4fab-9d97-809e9c6ddd28`::Granted request >Thread-34192::DEBUG::2012-12-02 15:11:22,790::task::794::TaskManager.Task::(resourceAcquired) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::_resourcesAcquired: Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743 (exclusive) >Thread-34192::DEBUG::2012-12-02 15:11:22,790::task::957::TaskManager.Task::(_decref) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::ref 1 aborting False >Thread-34192::INFO::2012-12-02 15:11:22,790::sp::912::Storage.StoragePool::(attachSD) sdUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743 spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6 >Thread-34192::DEBUG::2012-12-02 15:11:22,796::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-34192::DEBUG::2012-12-02 15:11:22,796::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34192::DEBUG::2012-12-02 15:11:22,796::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-34192::DEBUG::2012-12-02 15:11:22,796::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34192::DEBUG::2012-12-02 15:11:22,796::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-34192::DEBUG::2012-12-02 15:11:22,822::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-34192::DEBUG::2012-12-02 15:11:22,822::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34192::DEBUG::2012-12-02 15:11:23,456::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-34192::DEBUG::2012-12-02 15:11:23,525::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-34192::DEBUG::2012-12-02 15:11:23,526::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:23,526::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:23,526::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:23,526::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:23,526::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:23,527::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:23,527::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34192::INFO::2012-12-02 15:11:23,527::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 (id: 3) >Thread-34195::DEBUG::2012-12-02 15:11:27,014::task::568::TaskManager.Task::(_updateState) Task=`1d39415d-cdf4-4641-82fc-9c4239c00c23`::moving from state init -> state preparing >Thread-34195::INFO::2012-12-02 15:11:27,015::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34195::INFO::2012-12-02 15:11:27,015::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0129630565643', 'lastCheck': 1354453879.800108, 'code': 0, 'valid': True}} >Thread-34195::DEBUG::2012-12-02 15:11:27,016::task::1151::TaskManager.Task::(prepare) Task=`1d39415d-cdf4-4641-82fc-9c4239c00c23`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0129630565643', 'lastCheck': 1354453879.800108, 'code': 0, 'valid': True}} >Thread-34195::DEBUG::2012-12-02 15:11:27,016::task::568::TaskManager.Task::(_updateState) Task=`1d39415d-cdf4-4641-82fc-9c4239c00c23`::moving from state preparing -> state finished >Thread-34195::DEBUG::2012-12-02 15:11:27,016::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34195::DEBUG::2012-12-02 15:11:27,016::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34195::DEBUG::2012-12-02 15:11:27,016::task::957::TaskManager.Task::(_decref) Task=`1d39415d-cdf4-4641-82fc-9c4239c00c23`::ref 0 aborting False >Thread-34201::DEBUG::2012-12-02 15:11:37,146::task::568::TaskManager.Task::(_updateState) Task=`f5ec3430-61cb-418b-b633-29fd4bb0a993`::moving from state init -> state preparing >Thread-34201::INFO::2012-12-02 15:11:37,147::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34201::INFO::2012-12-02 15:11:37,147::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00927186012268', 'lastCheck': 1354453889.8197711, 'code': 0, 'valid': True}} >Thread-34201::DEBUG::2012-12-02 15:11:37,147::task::1151::TaskManager.Task::(prepare) Task=`f5ec3430-61cb-418b-b633-29fd4bb0a993`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00927186012268', 'lastCheck': 1354453889.8197711, 'code': 0, 'valid': True}} >Thread-34201::DEBUG::2012-12-02 15:11:37,147::task::568::TaskManager.Task::(_updateState) Task=`f5ec3430-61cb-418b-b633-29fd4bb0a993`::moving from state preparing -> state finished >Thread-34201::DEBUG::2012-12-02 15:11:37,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34201::DEBUG::2012-12-02 15:11:37,148::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34201::DEBUG::2012-12-02 15:11:37,148::task::957::TaskManager.Task::(_decref) Task=`f5ec3430-61cb-418b-b633-29fd4bb0a993`::ref 0 aborting False >Thread-34192::DEBUG::2012-12-02 15:11:44,528::safelease::178::SANLock::(acquireHostId) Host id for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 successfully acquired (id: 3) >Thread-34192::INFO::2012-12-02 15:11:44,528::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 (id: 3) >Thread-34192::DEBUG::2012-12-02 15:11:44,866::safelease::235::SANLock::(acquire) Cluster lock for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 successfully acquired (id: 3) >Thread-34192::DEBUG::2012-12-02 15:11:44,873::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'ROLE=Regular', 'SDUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=97136b77fee69c55cafb0d52fb03c618bd8dc8ac'] >Thread-34192::DEBUG::2012-12-02 15:11:44,873::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-34192::DEBUG::2012-12-02 15:11:44,873::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-34192::DEBUG::2012-12-02 15:11:44,873::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'ROLE=Regular', 'SDUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=fa40abc1d9b71134621eec8ef4b422ee1b19a82f'] >Thread-34192::DEBUG::2012-12-02 15:11:44,901::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-34192::DEBUG::2012-12-02 15:11:44,901::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-34192::DEBUG::2012-12-02 15:11:44,901::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-34192::DEBUG::2012-12-02 15:11:44,901::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'ROLE=Master', 'SDUUID=4e891d8c-9947-468e-88ec-29e49d68e083', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=9e92236f48501ee6591c09e6d171b30eb6ec05ef'] >Thread-34192::DEBUG::2012-12-02 15:11:44,946::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-34192::INFO::2012-12-02 15:11:44,947::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510/fafd959d-695f-4b5e-aa73-aaa2634ae743 to /rhev/data-center/da40b79b-9f51-4240-9cb2-2ab10e793fb6/fafd959d-695f-4b5e-aa73-aaa2634ae743 >Thread-34192::DEBUG::2012-12-02 15:11:44,947::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-34192::DEBUG::2012-12-02 15:11:44,947::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-34192::DEBUG::2012-12-02 15:11:44,947::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-34192::DEBUG::2012-12-02 15:11:44,947::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-34192::DEBUG::2012-12-02 15:11:44,948::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=a3', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1510', 'ROLE=Regular', 'SDUUID=fafd959d-695f-4b5e-aa73-aaa2634ae743', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=82c43630a63cd18cd3fc6049e27b272cc73b3338'] >Thread-34192::DEBUG::2012-12-02 15:11:44,989::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-34192::INFO::2012-12-02 15:11:44,989::safelease::239::SANLock::(release) Releasing cluster lock for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 >Thread-34192::DEBUG::2012-12-02 15:11:45,446::safelease::250::SANLock::(release) Cluster lock for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 successfully released >Thread-34192::INFO::2012-12-02 15:11:45,447::safelease::183::SANLock::(releaseHostId) Releasing host id for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 (id: 3) >Thread-34207::DEBUG::2012-12-02 15:11:47,259::task::568::TaskManager.Task::(_updateState) Task=`ee5d5e0f-84f7-4d4d-b240-cb45617846ea`::moving from state init -> state preparing >Thread-34207::INFO::2012-12-02 15:11:47,259::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34207::INFO::2012-12-02 15:11:47,260::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00748610496521', 'lastCheck': 1354453899.8376391, 'code': 0, 'valid': True}} >Thread-34207::DEBUG::2012-12-02 15:11:47,260::task::1151::TaskManager.Task::(prepare) Task=`ee5d5e0f-84f7-4d4d-b240-cb45617846ea`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00748610496521', 'lastCheck': 1354453899.8376391, 'code': 0, 'valid': True}} >Thread-34207::DEBUG::2012-12-02 15:11:47,260::task::568::TaskManager.Task::(_updateState) Task=`ee5d5e0f-84f7-4d4d-b240-cb45617846ea`::moving from state preparing -> state finished >Thread-34207::DEBUG::2012-12-02 15:11:47,260::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34207::DEBUG::2012-12-02 15:11:47,260::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34207::DEBUG::2012-12-02 15:11:47,260::task::957::TaskManager.Task::(_decref) Task=`ee5d5e0f-84f7-4d4d-b240-cb45617846ea`::ref 0 aborting False >Thread-34192::DEBUG::2012-12-02 15:11:48,447::safelease::193::SANLock::(releaseHostId) Host id for domain fafd959d-695f-4b5e-aa73-aaa2634ae743 released successfully (id: 3) >Thread-34192::DEBUG::2012-12-02 15:11:48,447::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-34192::DEBUG::2012-12-02 15:11:48,447::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-34192::DEBUG::2012-12-02 15:11:48,448::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34192::DEBUG::2012-12-02 15:11:48,449::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34192::DEBUG::2012-12-02 15:11:48,659::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34192::WARNING::2012-12-02 15:11:48,660::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34192::DEBUG::2012-12-02 15:11:48,661::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34192::ERROR::2012-12-02 15:11:48,691::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `a3177eae-1dc9-445a-92b1-da23ec8008bc` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34192::DEBUG::2012-12-02 15:11:48,692::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-34192::INFO::2012-12-02 15:11:48,692::logUtils::39::dispatcher::(wrapper) Run and protect: attachStorageDomain, Return response: None >Thread-34192::DEBUG::2012-12-02 15:11:48,692::task::1151::TaskManager.Task::(prepare) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::finished: None >Thread-34192::DEBUG::2012-12-02 15:11:48,692::task::568::TaskManager.Task::(_updateState) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::moving from state preparing -> state finished >Thread-34192::DEBUG::2012-12-02 15:11:48,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743': < ResourceRef 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743', isValid: 'True' obj: 'None'>, 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34192::DEBUG::2012-12-02 15:11:48,692::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' (0 active users) >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743' is free, finding out if anyone is waiting for it. >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.fafd959d-695f-4b5e-aa73-aaa2634ae743', Clearing records. >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34192::DEBUG::2012-12-02 15:11:48,693::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34192::DEBUG::2012-12-02 15:11:48,694::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34192::DEBUG::2012-12-02 15:11:48,694::task::957::TaskManager.Task::(_decref) Task=`56dcc3ba-81ee-47c2-accc-4bcde99e24ed`::ref 0 aborting False >Thread-34269::DEBUG::2012-12-02 15:11:48,701::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34269::DEBUG::2012-12-02 15:11:48,701::task::568::TaskManager.Task::(_updateState) Task=`2aa0482c-c36e-45b6-870f-cd82cc98bcc3`::moving from state init -> state preparing >Thread-34269::INFO::2012-12-02 15:11:48,701::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34269::INFO::2012-12-02 15:11:48,701::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34269::DEBUG::2012-12-02 15:11:48,702::task::1151::TaskManager.Task::(prepare) Task=`2aa0482c-c36e-45b6-870f-cd82cc98bcc3`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34269::DEBUG::2012-12-02 15:11:48,702::task::568::TaskManager.Task::(_updateState) Task=`2aa0482c-c36e-45b6-870f-cd82cc98bcc3`::moving from state preparing -> state finished >Thread-34269::DEBUG::2012-12-02 15:11:48,702::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34269::DEBUG::2012-12-02 15:11:48,702::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34269::DEBUG::2012-12-02 15:11:48,702::task::957::TaskManager.Task::(_decref) Task=`2aa0482c-c36e-45b6-870f-cd82cc98bcc3`::ref 0 aborting False >Thread-34270::DEBUG::2012-12-02 15:11:48,709::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34270::DEBUG::2012-12-02 15:11:48,709::task::568::TaskManager.Task::(_updateState) Task=`7ff64ca6-a673-4761-82a0-702f79bf4614`::moving from state init -> state preparing >Thread-34270::INFO::2012-12-02 15:11:48,710::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34270::DEBUG::2012-12-02 15:11:48,710::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`515ed468-5c63-4c00-afa2-cadd01e1bfaf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34270::DEBUG::2012-12-02 15:11:48,710::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34270::DEBUG::2012-12-02 15:11:48,710::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34270::DEBUG::2012-12-02 15:11:48,710::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`515ed468-5c63-4c00-afa2-cadd01e1bfaf`::Granted request >Thread-34270::DEBUG::2012-12-02 15:11:48,711::task::794::TaskManager.Task::(resourceAcquired) Task=`7ff64ca6-a673-4761-82a0-702f79bf4614`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34270::DEBUG::2012-12-02 15:11:48,711::task::957::TaskManager.Task::(_decref) Task=`7ff64ca6-a673-4761-82a0-702f79bf4614`::ref 1 aborting False >Thread-34270::DEBUG::2012-12-02 15:11:48,717::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34270::DEBUG::2012-12-02 15:11:48,717::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34270::DEBUG::2012-12-02 15:11:48,925::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34270::WARNING::2012-12-02 15:11:48,926::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34270::DEBUG::2012-12-02 15:11:48,927::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34270::WARNING::2012-12-02 15:11:48,957::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34270::INFO::2012-12-02 15:11:48,958::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34270::DEBUG::2012-12-02 15:11:48,958::task::1151::TaskManager.Task::(prepare) Task=`7ff64ca6-a673-4761-82a0-702f79bf4614`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34270::DEBUG::2012-12-02 15:11:48,958::task::568::TaskManager.Task::(_updateState) Task=`7ff64ca6-a673-4761-82a0-702f79bf4614`::moving from state preparing -> state finished >Thread-34270::DEBUG::2012-12-02 15:11:48,958::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34270::DEBUG::2012-12-02 15:11:48,958::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34270::DEBUG::2012-12-02 15:11:48,959::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34270::DEBUG::2012-12-02 15:11:48,959::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34270::DEBUG::2012-12-02 15:11:48,959::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34270::DEBUG::2012-12-02 15:11:48,959::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34270::DEBUG::2012-12-02 15:11:48,959::task::957::TaskManager.Task::(_decref) Task=`7ff64ca6-a673-4761-82a0-702f79bf4614`::ref 0 aborting False >Thread-34335::DEBUG::2012-12-02 15:11:57,373::task::568::TaskManager.Task::(_updateState) Task=`aa1f43a1-c70e-4dc4-b7b3-2ab1938c1f4c`::moving from state init -> state preparing >Thread-34335::INFO::2012-12-02 15:11:57,373::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34335::INFO::2012-12-02 15:11:57,374::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00874805450439', 'lastCheck': 1354453909.8565681, 'code': 0, 'valid': True}} >Thread-34335::DEBUG::2012-12-02 15:11:57,374::task::1151::TaskManager.Task::(prepare) Task=`aa1f43a1-c70e-4dc4-b7b3-2ab1938c1f4c`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00874805450439', 'lastCheck': 1354453909.8565681, 'code': 0, 'valid': True}} >Thread-34335::DEBUG::2012-12-02 15:11:57,374::task::568::TaskManager.Task::(_updateState) Task=`aa1f43a1-c70e-4dc4-b7b3-2ab1938c1f4c`::moving from state preparing -> state finished >Thread-34335::DEBUG::2012-12-02 15:11:57,374::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34335::DEBUG::2012-12-02 15:11:57,374::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34335::DEBUG::2012-12-02 15:11:57,374::task::957::TaskManager.Task::(_decref) Task=`aa1f43a1-c70e-4dc4-b7b3-2ab1938c1f4c`::ref 0 aborting False >Thread-34337::DEBUG::2012-12-02 15:11:59,031::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34337::DEBUG::2012-12-02 15:11:59,031::task::568::TaskManager.Task::(_updateState) Task=`f0bf74a3-86da-4676-ae28-01b211d03ad4`::moving from state init -> state preparing >Thread-34337::INFO::2012-12-02 15:11:59,031::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34337::INFO::2012-12-02 15:11:59,031::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34337::DEBUG::2012-12-02 15:11:59,031::task::1151::TaskManager.Task::(prepare) Task=`f0bf74a3-86da-4676-ae28-01b211d03ad4`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34337::DEBUG::2012-12-02 15:11:59,032::task::568::TaskManager.Task::(_updateState) Task=`f0bf74a3-86da-4676-ae28-01b211d03ad4`::moving from state preparing -> state finished >Thread-34337::DEBUG::2012-12-02 15:11:59,032::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34337::DEBUG::2012-12-02 15:11:59,032::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34337::DEBUG::2012-12-02 15:11:59,032::task::957::TaskManager.Task::(_decref) Task=`f0bf74a3-86da-4676-ae28-01b211d03ad4`::ref 0 aborting False >Thread-34338::DEBUG::2012-12-02 15:11:59,039::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34338::DEBUG::2012-12-02 15:11:59,039::task::568::TaskManager.Task::(_updateState) Task=`48cb598b-7e43-4900-87ee-faf5e897156c`::moving from state init -> state preparing >Thread-34338::INFO::2012-12-02 15:11:59,039::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34338::DEBUG::2012-12-02 15:11:59,040::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c4f09519-e83e-475c-aeb6-8b88c0235a51`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34338::DEBUG::2012-12-02 15:11:59,040::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34338::DEBUG::2012-12-02 15:11:59,040::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34338::DEBUG::2012-12-02 15:11:59,040::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c4f09519-e83e-475c-aeb6-8b88c0235a51`::Granted request >Thread-34338::DEBUG::2012-12-02 15:11:59,040::task::794::TaskManager.Task::(resourceAcquired) Task=`48cb598b-7e43-4900-87ee-faf5e897156c`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34338::DEBUG::2012-12-02 15:11:59,040::task::957::TaskManager.Task::(_decref) Task=`48cb598b-7e43-4900-87ee-faf5e897156c`::ref 1 aborting False >Thread-34338::DEBUG::2012-12-02 15:11:59,046::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34338::DEBUG::2012-12-02 15:11:59,046::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34338::DEBUG::2012-12-02 15:11:59,254::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34338::WARNING::2012-12-02 15:11:59,256::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34338::DEBUG::2012-12-02 15:11:59,256::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34338::WARNING::2012-12-02 15:11:59,286::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34338::INFO::2012-12-02 15:11:59,287::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34338::DEBUG::2012-12-02 15:11:59,287::task::1151::TaskManager.Task::(prepare) Task=`48cb598b-7e43-4900-87ee-faf5e897156c`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34338::DEBUG::2012-12-02 15:11:59,287::task::568::TaskManager.Task::(_updateState) Task=`48cb598b-7e43-4900-87ee-faf5e897156c`::moving from state preparing -> state finished >Thread-34338::DEBUG::2012-12-02 15:11:59,287::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34338::DEBUG::2012-12-02 15:11:59,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34338::DEBUG::2012-12-02 15:11:59,287::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34338::DEBUG::2012-12-02 15:11:59,288::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34338::DEBUG::2012-12-02 15:11:59,288::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34338::DEBUG::2012-12-02 15:11:59,288::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34338::DEBUG::2012-12-02 15:11:59,288::task::957::TaskManager.Task::(_decref) Task=`48cb598b-7e43-4900-87ee-faf5e897156c`::ref 0 aborting False >Thread-34403::DEBUG::2012-12-02 15:12:07,565::task::568::TaskManager.Task::(_updateState) Task=`3a79c939-2d26-464e-a41b-31923eee7b18`::moving from state init -> state preparing >Thread-34403::INFO::2012-12-02 15:12:07,565::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34403::INFO::2012-12-02 15:12:07,565::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00955104827881', 'lastCheck': 1354453919.8763001, 'code': 0, 'valid': True}} >Thread-34403::DEBUG::2012-12-02 15:12:07,566::task::1151::TaskManager.Task::(prepare) Task=`3a79c939-2d26-464e-a41b-31923eee7b18`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00955104827881', 'lastCheck': 1354453919.8763001, 'code': 0, 'valid': True}} >Thread-34403::DEBUG::2012-12-02 15:12:07,566::task::568::TaskManager.Task::(_updateState) Task=`3a79c939-2d26-464e-a41b-31923eee7b18`::moving from state preparing -> state finished >Thread-34403::DEBUG::2012-12-02 15:12:07,566::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34403::DEBUG::2012-12-02 15:12:07,566::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34403::DEBUG::2012-12-02 15:12:07,566::task::957::TaskManager.Task::(_decref) Task=`3a79c939-2d26-464e-a41b-31923eee7b18`::ref 0 aborting False >Thread-34405::DEBUG::2012-12-02 15:12:09,374::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34405::DEBUG::2012-12-02 15:12:09,374::task::568::TaskManager.Task::(_updateState) Task=`b10c5344-b78c-4508-a4a9-5bd7acc1badd`::moving from state init -> state preparing >Thread-34405::INFO::2012-12-02 15:12:09,375::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34405::INFO::2012-12-02 15:12:09,375::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34405::DEBUG::2012-12-02 15:12:09,375::task::1151::TaskManager.Task::(prepare) Task=`b10c5344-b78c-4508-a4a9-5bd7acc1badd`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34405::DEBUG::2012-12-02 15:12:09,375::task::568::TaskManager.Task::(_updateState) Task=`b10c5344-b78c-4508-a4a9-5bd7acc1badd`::moving from state preparing -> state finished >Thread-34405::DEBUG::2012-12-02 15:12:09,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34405::DEBUG::2012-12-02 15:12:09,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34405::DEBUG::2012-12-02 15:12:09,375::task::957::TaskManager.Task::(_decref) Task=`b10c5344-b78c-4508-a4a9-5bd7acc1badd`::ref 0 aborting False >Thread-34406::DEBUG::2012-12-02 15:12:09,382::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34406::DEBUG::2012-12-02 15:12:09,382::task::568::TaskManager.Task::(_updateState) Task=`464c1f1e-4ee3-487a-b02e-a2db0727a31e`::moving from state init -> state preparing >Thread-34406::INFO::2012-12-02 15:12:09,382::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34406::DEBUG::2012-12-02 15:12:09,383::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b8987df2-5dac-4a0e-a595-145cefb87716`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34406::DEBUG::2012-12-02 15:12:09,383::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34406::DEBUG::2012-12-02 15:12:09,383::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34406::DEBUG::2012-12-02 15:12:09,383::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b8987df2-5dac-4a0e-a595-145cefb87716`::Granted request >Thread-34406::DEBUG::2012-12-02 15:12:09,383::task::794::TaskManager.Task::(resourceAcquired) Task=`464c1f1e-4ee3-487a-b02e-a2db0727a31e`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34406::DEBUG::2012-12-02 15:12:09,384::task::957::TaskManager.Task::(_decref) Task=`464c1f1e-4ee3-487a-b02e-a2db0727a31e`::ref 1 aborting False >Thread-34406::DEBUG::2012-12-02 15:12:09,388::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34406::DEBUG::2012-12-02 15:12:09,389::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34406::DEBUG::2012-12-02 15:12:09,594::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34406::WARNING::2012-12-02 15:12:09,595::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34406::DEBUG::2012-12-02 15:12:09,596::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34406::WARNING::2012-12-02 15:12:09,626::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34406::INFO::2012-12-02 15:12:09,626::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34406::DEBUG::2012-12-02 15:12:09,627::task::1151::TaskManager.Task::(prepare) Task=`464c1f1e-4ee3-487a-b02e-a2db0727a31e`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34406::DEBUG::2012-12-02 15:12:09,627::task::568::TaskManager.Task::(_updateState) Task=`464c1f1e-4ee3-487a-b02e-a2db0727a31e`::moving from state preparing -> state finished >Thread-34406::DEBUG::2012-12-02 15:12:09,627::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34406::DEBUG::2012-12-02 15:12:09,627::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34406::DEBUG::2012-12-02 15:12:09,627::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34406::DEBUG::2012-12-02 15:12:09,627::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34406::DEBUG::2012-12-02 15:12:09,628::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34406::DEBUG::2012-12-02 15:12:09,628::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34406::DEBUG::2012-12-02 15:12:09,628::task::957::TaskManager.Task::(_decref) Task=`464c1f1e-4ee3-487a-b02e-a2db0727a31e`::ref 0 aborting False >Thread-34471::DEBUG::2012-12-02 15:12:17,665::task::568::TaskManager.Task::(_updateState) Task=`c4bc7eeb-0126-4a77-a8f7-3a142f227aaa`::moving from state init -> state preparing >Thread-34471::INFO::2012-12-02 15:12:17,666::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34471::INFO::2012-12-02 15:12:17,666::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00705194473267', 'lastCheck': 1354453929.8944831, 'code': 0, 'valid': True}} >Thread-34471::DEBUG::2012-12-02 15:12:17,666::task::1151::TaskManager.Task::(prepare) Task=`c4bc7eeb-0126-4a77-a8f7-3a142f227aaa`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00705194473267', 'lastCheck': 1354453929.8944831, 'code': 0, 'valid': True}} >Thread-34471::DEBUG::2012-12-02 15:12:17,666::task::568::TaskManager.Task::(_updateState) Task=`c4bc7eeb-0126-4a77-a8f7-3a142f227aaa`::moving from state preparing -> state finished >Thread-34471::DEBUG::2012-12-02 15:12:17,666::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34471::DEBUG::2012-12-02 15:12:17,666::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34471::DEBUG::2012-12-02 15:12:17,666::task::957::TaskManager.Task::(_decref) Task=`c4bc7eeb-0126-4a77-a8f7-3a142f227aaa`::ref 0 aborting False >Thread-34473::DEBUG::2012-12-02 15:12:19,723::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34473::DEBUG::2012-12-02 15:12:19,724::task::568::TaskManager.Task::(_updateState) Task=`5d65f560-9a37-47b5-a17f-561e730ddd43`::moving from state init -> state preparing >Thread-34473::INFO::2012-12-02 15:12:19,724::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34473::INFO::2012-12-02 15:12:19,724::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34473::DEBUG::2012-12-02 15:12:19,724::task::1151::TaskManager.Task::(prepare) Task=`5d65f560-9a37-47b5-a17f-561e730ddd43`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34473::DEBUG::2012-12-02 15:12:19,724::task::568::TaskManager.Task::(_updateState) Task=`5d65f560-9a37-47b5-a17f-561e730ddd43`::moving from state preparing -> state finished >Thread-34473::DEBUG::2012-12-02 15:12:19,724::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34473::DEBUG::2012-12-02 15:12:19,724::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34473::DEBUG::2012-12-02 15:12:19,725::task::957::TaskManager.Task::(_decref) Task=`5d65f560-9a37-47b5-a17f-561e730ddd43`::ref 0 aborting False >Thread-34474::DEBUG::2012-12-02 15:12:19,731::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34474::DEBUG::2012-12-02 15:12:19,731::task::568::TaskManager.Task::(_updateState) Task=`574bd775-38a5-4d33-ae50-61335d6f05df`::moving from state init -> state preparing >Thread-34474::INFO::2012-12-02 15:12:19,731::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34474::DEBUG::2012-12-02 15:12:19,732::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`71fb16e4-189c-488b-907a-2d50f6ba23fe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34474::DEBUG::2012-12-02 15:12:19,732::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34474::DEBUG::2012-12-02 15:12:19,732::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34474::DEBUG::2012-12-02 15:12:19,732::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`71fb16e4-189c-488b-907a-2d50f6ba23fe`::Granted request >Thread-34474::DEBUG::2012-12-02 15:12:19,732::task::794::TaskManager.Task::(resourceAcquired) Task=`574bd775-38a5-4d33-ae50-61335d6f05df`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34474::DEBUG::2012-12-02 15:12:19,732::task::957::TaskManager.Task::(_decref) Task=`574bd775-38a5-4d33-ae50-61335d6f05df`::ref 1 aborting False >Thread-34474::DEBUG::2012-12-02 15:12:19,737::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34474::DEBUG::2012-12-02 15:12:19,738::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34474::DEBUG::2012-12-02 15:12:19,943::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34474::WARNING::2012-12-02 15:12:19,945::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34474::DEBUG::2012-12-02 15:12:19,945::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34474::WARNING::2012-12-02 15:12:19,979::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34474::INFO::2012-12-02 15:12:19,980::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34474::DEBUG::2012-12-02 15:12:19,981::task::1151::TaskManager.Task::(prepare) Task=`574bd775-38a5-4d33-ae50-61335d6f05df`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34474::DEBUG::2012-12-02 15:12:19,982::task::568::TaskManager.Task::(_updateState) Task=`574bd775-38a5-4d33-ae50-61335d6f05df`::moving from state preparing -> state finished >Thread-34474::DEBUG::2012-12-02 15:12:19,983::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34474::DEBUG::2012-12-02 15:12:19,985::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34474::DEBUG::2012-12-02 15:12:19,987::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34474::DEBUG::2012-12-02 15:12:19,987::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34474::DEBUG::2012-12-02 15:12:19,987::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34474::DEBUG::2012-12-02 15:12:19,987::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34474::DEBUG::2012-12-02 15:12:19,987::task::957::TaskManager.Task::(_decref) Task=`574bd775-38a5-4d33-ae50-61335d6f05df`::ref 0 aborting False >Thread-34539::DEBUG::2012-12-02 15:12:27,787::task::568::TaskManager.Task::(_updateState) Task=`a0a59b55-2910-4deb-b83b-5ec566b41dec`::moving from state init -> state preparing >Thread-34539::INFO::2012-12-02 15:12:27,787::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34539::INFO::2012-12-02 15:12:27,787::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00969004631042', 'lastCheck': 1354453940.0024431, 'code': 0, 'valid': True}} >Thread-34539::DEBUG::2012-12-02 15:12:27,788::task::1151::TaskManager.Task::(prepare) Task=`a0a59b55-2910-4deb-b83b-5ec566b41dec`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00969004631042', 'lastCheck': 1354453940.0024431, 'code': 0, 'valid': True}} >Thread-34539::DEBUG::2012-12-02 15:12:27,788::task::568::TaskManager.Task::(_updateState) Task=`a0a59b55-2910-4deb-b83b-5ec566b41dec`::moving from state preparing -> state finished >Thread-34539::DEBUG::2012-12-02 15:12:27,788::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34539::DEBUG::2012-12-02 15:12:27,788::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34539::DEBUG::2012-12-02 15:12:27,788::task::957::TaskManager.Task::(_decref) Task=`a0a59b55-2910-4deb-b83b-5ec566b41dec`::ref 0 aborting False >Thread-34542::DEBUG::2012-12-02 15:12:30,089::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34542::DEBUG::2012-12-02 15:12:30,089::task::568::TaskManager.Task::(_updateState) Task=`a51c6382-8b64-4319-b2a2-853bcfbd8a8f`::moving from state init -> state preparing >Thread-34542::INFO::2012-12-02 15:12:30,090::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34542::INFO::2012-12-02 15:12:30,090::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34542::DEBUG::2012-12-02 15:12:30,090::task::1151::TaskManager.Task::(prepare) Task=`a51c6382-8b64-4319-b2a2-853bcfbd8a8f`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34542::DEBUG::2012-12-02 15:12:30,090::task::568::TaskManager.Task::(_updateState) Task=`a51c6382-8b64-4319-b2a2-853bcfbd8a8f`::moving from state preparing -> state finished >Thread-34542::DEBUG::2012-12-02 15:12:30,090::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34542::DEBUG::2012-12-02 15:12:30,090::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34542::DEBUG::2012-12-02 15:12:30,090::task::957::TaskManager.Task::(_decref) Task=`a51c6382-8b64-4319-b2a2-853bcfbd8a8f`::ref 0 aborting False >Thread-34543::DEBUG::2012-12-02 15:12:30,095::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34543::DEBUG::2012-12-02 15:12:30,095::task::568::TaskManager.Task::(_updateState) Task=`5c86078f-ca46-423b-a6d6-1b9daa34dd65`::moving from state init -> state preparing >Thread-34543::INFO::2012-12-02 15:12:30,095::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34543::DEBUG::2012-12-02 15:12:30,095::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`47616a7c-e2ee-453f-99ed-29938e08dc8f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34543::DEBUG::2012-12-02 15:12:30,095::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34543::DEBUG::2012-12-02 15:12:30,096::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34543::DEBUG::2012-12-02 15:12:30,096::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`47616a7c-e2ee-453f-99ed-29938e08dc8f`::Granted request >Thread-34543::DEBUG::2012-12-02 15:12:30,096::task::794::TaskManager.Task::(resourceAcquired) Task=`5c86078f-ca46-423b-a6d6-1b9daa34dd65`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34543::DEBUG::2012-12-02 15:12:30,096::task::957::TaskManager.Task::(_decref) Task=`5c86078f-ca46-423b-a6d6-1b9daa34dd65`::ref 1 aborting False >Thread-34543::DEBUG::2012-12-02 15:12:30,102::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34543::DEBUG::2012-12-02 15:12:30,103::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34543::DEBUG::2012-12-02 15:12:30,307::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34543::WARNING::2012-12-02 15:12:30,309::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34543::DEBUG::2012-12-02 15:12:30,309::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34543::WARNING::2012-12-02 15:12:30,338::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34543::INFO::2012-12-02 15:12:30,339::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34543::DEBUG::2012-12-02 15:12:30,339::task::1151::TaskManager.Task::(prepare) Task=`5c86078f-ca46-423b-a6d6-1b9daa34dd65`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34543::DEBUG::2012-12-02 15:12:30,339::task::568::TaskManager.Task::(_updateState) Task=`5c86078f-ca46-423b-a6d6-1b9daa34dd65`::moving from state preparing -> state finished >Thread-34543::DEBUG::2012-12-02 15:12:30,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34543::DEBUG::2012-12-02 15:12:30,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34543::DEBUG::2012-12-02 15:12:30,340::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34543::DEBUG::2012-12-02 15:12:30,340::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34543::DEBUG::2012-12-02 15:12:30,340::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34543::DEBUG::2012-12-02 15:12:30,340::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34543::DEBUG::2012-12-02 15:12:30,340::task::957::TaskManager.Task::(_decref) Task=`5c86078f-ca46-423b-a6d6-1b9daa34dd65`::ref 0 aborting False >Thread-34607::DEBUG::2012-12-02 15:12:37,890::task::568::TaskManager.Task::(_updateState) Task=`1c3a522c-1a55-41c1-be3f-951550d45921`::moving from state init -> state preparing >Thread-34607::INFO::2012-12-02 15:12:37,890::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34607::INFO::2012-12-02 15:12:37,890::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0121610164642', 'lastCheck': 1354453950.0254099, 'code': 0, 'valid': True}} >Thread-34607::DEBUG::2012-12-02 15:12:37,890::task::1151::TaskManager.Task::(prepare) Task=`1c3a522c-1a55-41c1-be3f-951550d45921`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0121610164642', 'lastCheck': 1354453950.0254099, 'code': 0, 'valid': True}} >Thread-34607::DEBUG::2012-12-02 15:12:37,890::task::568::TaskManager.Task::(_updateState) Task=`1c3a522c-1a55-41c1-be3f-951550d45921`::moving from state preparing -> state finished >Thread-34607::DEBUG::2012-12-02 15:12:37,891::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34607::DEBUG::2012-12-02 15:12:37,891::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34607::DEBUG::2012-12-02 15:12:37,891::task::957::TaskManager.Task::(_decref) Task=`1c3a522c-1a55-41c1-be3f-951550d45921`::ref 0 aborting False >Thread-34610::DEBUG::2012-12-02 15:12:40,451::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34610::DEBUG::2012-12-02 15:12:40,451::task::568::TaskManager.Task::(_updateState) Task=`7c3b8d89-4220-4eca-abaf-582f65c89d2d`::moving from state init -> state preparing >Thread-34610::INFO::2012-12-02 15:12:40,451::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34610::INFO::2012-12-02 15:12:40,451::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34610::DEBUG::2012-12-02 15:12:40,451::task::1151::TaskManager.Task::(prepare) Task=`7c3b8d89-4220-4eca-abaf-582f65c89d2d`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34610::DEBUG::2012-12-02 15:12:40,451::task::568::TaskManager.Task::(_updateState) Task=`7c3b8d89-4220-4eca-abaf-582f65c89d2d`::moving from state preparing -> state finished >Thread-34610::DEBUG::2012-12-02 15:12:40,452::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34610::DEBUG::2012-12-02 15:12:40,452::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34610::DEBUG::2012-12-02 15:12:40,452::task::957::TaskManager.Task::(_decref) Task=`7c3b8d89-4220-4eca-abaf-582f65c89d2d`::ref 0 aborting False >Thread-34611::DEBUG::2012-12-02 15:12:40,458::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34611::DEBUG::2012-12-02 15:12:40,459::task::568::TaskManager.Task::(_updateState) Task=`63de028f-dc7a-4de2-b87e-547602a89e1d`::moving from state init -> state preparing >Thread-34611::INFO::2012-12-02 15:12:40,459::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34611::DEBUG::2012-12-02 15:12:40,459::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`2dde694c-71de-4aa7-be8b-79baa1b505e7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34611::DEBUG::2012-12-02 15:12:40,459::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34611::DEBUG::2012-12-02 15:12:40,460::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34611::DEBUG::2012-12-02 15:12:40,460::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`2dde694c-71de-4aa7-be8b-79baa1b505e7`::Granted request >Thread-34611::DEBUG::2012-12-02 15:12:40,460::task::794::TaskManager.Task::(resourceAcquired) Task=`63de028f-dc7a-4de2-b87e-547602a89e1d`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34611::DEBUG::2012-12-02 15:12:40,460::task::957::TaskManager.Task::(_decref) Task=`63de028f-dc7a-4de2-b87e-547602a89e1d`::ref 1 aborting False >Thread-34611::DEBUG::2012-12-02 15:12:40,466::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34611::DEBUG::2012-12-02 15:12:40,466::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34611::DEBUG::2012-12-02 15:12:40,669::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34611::WARNING::2012-12-02 15:12:40,670::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34611::DEBUG::2012-12-02 15:12:40,671::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34611::WARNING::2012-12-02 15:12:40,700::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34611::INFO::2012-12-02 15:12:40,700::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34611::DEBUG::2012-12-02 15:12:40,700::task::1151::TaskManager.Task::(prepare) Task=`63de028f-dc7a-4de2-b87e-547602a89e1d`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34611::DEBUG::2012-12-02 15:12:40,701::task::568::TaskManager.Task::(_updateState) Task=`63de028f-dc7a-4de2-b87e-547602a89e1d`::moving from state preparing -> state finished >Thread-34611::DEBUG::2012-12-02 15:12:40,701::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34611::DEBUG::2012-12-02 15:12:40,701::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34611::DEBUG::2012-12-02 15:12:40,701::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34611::DEBUG::2012-12-02 15:12:40,701::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34611::DEBUG::2012-12-02 15:12:40,701::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34611::DEBUG::2012-12-02 15:12:40,702::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34611::DEBUG::2012-12-02 15:12:40,702::task::957::TaskManager.Task::(_decref) Task=`63de028f-dc7a-4de2-b87e-547602a89e1d`::ref 0 aborting False >Thread-34675::DEBUG::2012-12-02 15:12:48,012::task::568::TaskManager.Task::(_updateState) Task=`8e163314-3c93-4fa8-86b1-e099c5d2bd90`::moving from state init -> state preparing >Thread-34675::INFO::2012-12-02 15:12:48,012::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34675::INFO::2012-12-02 15:12:48,012::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0110599994659', 'lastCheck': 1354453960.0472641, 'code': 0, 'valid': True}} >Thread-34675::DEBUG::2012-12-02 15:12:48,012::task::1151::TaskManager.Task::(prepare) Task=`8e163314-3c93-4fa8-86b1-e099c5d2bd90`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0110599994659', 'lastCheck': 1354453960.0472641, 'code': 0, 'valid': True}} >Thread-34675::DEBUG::2012-12-02 15:12:48,012::task::568::TaskManager.Task::(_updateState) Task=`8e163314-3c93-4fa8-86b1-e099c5d2bd90`::moving from state preparing -> state finished >Thread-34675::DEBUG::2012-12-02 15:12:48,012::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34675::DEBUG::2012-12-02 15:12:48,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34675::DEBUG::2012-12-02 15:12:48,013::task::957::TaskManager.Task::(_decref) Task=`8e163314-3c93-4fa8-86b1-e099c5d2bd90`::ref 0 aborting False >Thread-34678::DEBUG::2012-12-02 15:12:50,818::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34678::DEBUG::2012-12-02 15:12:50,818::task::568::TaskManager.Task::(_updateState) Task=`0769b291-0508-4dc8-8508-ea5aef17a4ae`::moving from state init -> state preparing >Thread-34678::INFO::2012-12-02 15:12:50,818::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34678::INFO::2012-12-02 15:12:50,818::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34678::DEBUG::2012-12-02 15:12:50,818::task::1151::TaskManager.Task::(prepare) Task=`0769b291-0508-4dc8-8508-ea5aef17a4ae`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34678::DEBUG::2012-12-02 15:12:50,819::task::568::TaskManager.Task::(_updateState) Task=`0769b291-0508-4dc8-8508-ea5aef17a4ae`::moving from state preparing -> state finished >Thread-34678::DEBUG::2012-12-02 15:12:50,819::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34678::DEBUG::2012-12-02 15:12:50,819::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34678::DEBUG::2012-12-02 15:12:50,819::task::957::TaskManager.Task::(_decref) Task=`0769b291-0508-4dc8-8508-ea5aef17a4ae`::ref 0 aborting False >Thread-34679::DEBUG::2012-12-02 15:12:50,825::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34679::DEBUG::2012-12-02 15:12:50,826::task::568::TaskManager.Task::(_updateState) Task=`41171004-7280-4a32-9e69-4fd6b7e52f87`::moving from state init -> state preparing >Thread-34679::INFO::2012-12-02 15:12:50,826::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34679::DEBUG::2012-12-02 15:12:50,826::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`9f294fff-4016-42df-ba9e-e7ec60d8a429`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34679::DEBUG::2012-12-02 15:12:50,826::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34679::DEBUG::2012-12-02 15:12:50,827::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34679::DEBUG::2012-12-02 15:12:50,827::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`9f294fff-4016-42df-ba9e-e7ec60d8a429`::Granted request >Thread-34679::DEBUG::2012-12-02 15:12:50,827::task::794::TaskManager.Task::(resourceAcquired) Task=`41171004-7280-4a32-9e69-4fd6b7e52f87`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34679::DEBUG::2012-12-02 15:12:50,827::task::957::TaskManager.Task::(_decref) Task=`41171004-7280-4a32-9e69-4fd6b7e52f87`::ref 1 aborting False >Thread-34679::DEBUG::2012-12-02 15:12:50,833::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34679::DEBUG::2012-12-02 15:12:50,834::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34679::DEBUG::2012-12-02 15:12:51,038::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34679::WARNING::2012-12-02 15:12:51,040::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34679::DEBUG::2012-12-02 15:12:51,040::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34679::WARNING::2012-12-02 15:12:51,070::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34679::INFO::2012-12-02 15:12:51,070::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34679::DEBUG::2012-12-02 15:12:51,070::task::1151::TaskManager.Task::(prepare) Task=`41171004-7280-4a32-9e69-4fd6b7e52f87`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34679::DEBUG::2012-12-02 15:12:51,070::task::568::TaskManager.Task::(_updateState) Task=`41171004-7280-4a32-9e69-4fd6b7e52f87`::moving from state preparing -> state finished >Thread-34679::DEBUG::2012-12-02 15:12:51,071::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34679::DEBUG::2012-12-02 15:12:51,071::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34679::DEBUG::2012-12-02 15:12:51,071::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34679::DEBUG::2012-12-02 15:12:51,071::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34679::DEBUG::2012-12-02 15:12:51,071::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34679::DEBUG::2012-12-02 15:12:51,071::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34679::DEBUG::2012-12-02 15:12:51,072::task::957::TaskManager.Task::(_decref) Task=`41171004-7280-4a32-9e69-4fd6b7e52f87`::ref 0 aborting False >Thread-34743::DEBUG::2012-12-02 15:12:58,138::task::568::TaskManager.Task::(_updateState) Task=`dbea8951-b42b-4fd2-b14a-f11daff1e0cc`::moving from state init -> state preparing >Thread-34743::INFO::2012-12-02 15:12:58,139::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34743::INFO::2012-12-02 15:12:58,139::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00387191772461', 'lastCheck': 1354453970.0616441, 'code': 0, 'valid': True}} >Thread-34743::DEBUG::2012-12-02 15:12:58,139::task::1151::TaskManager.Task::(prepare) Task=`dbea8951-b42b-4fd2-b14a-f11daff1e0cc`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.00387191772461', 'lastCheck': 1354453970.0616441, 'code': 0, 'valid': True}} >Thread-34743::DEBUG::2012-12-02 15:12:58,139::task::568::TaskManager.Task::(_updateState) Task=`dbea8951-b42b-4fd2-b14a-f11daff1e0cc`::moving from state preparing -> state finished >Thread-34743::DEBUG::2012-12-02 15:12:58,139::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34743::DEBUG::2012-12-02 15:12:58,139::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34743::DEBUG::2012-12-02 15:12:58,140::task::957::TaskManager.Task::(_decref) Task=`dbea8951-b42b-4fd2-b14a-f11daff1e0cc`::ref 0 aborting False >Thread-34746::DEBUG::2012-12-02 15:13:01,177::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34746::DEBUG::2012-12-02 15:13:01,178::task::568::TaskManager.Task::(_updateState) Task=`bfcd4e97-34b5-4a27-afce-0279edd14043`::moving from state init -> state preparing >Thread-34746::INFO::2012-12-02 15:13:01,178::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34746::INFO::2012-12-02 15:13:01,178::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34746::DEBUG::2012-12-02 15:13:01,178::task::1151::TaskManager.Task::(prepare) Task=`bfcd4e97-34b5-4a27-afce-0279edd14043`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-34746::DEBUG::2012-12-02 15:13:01,178::task::568::TaskManager.Task::(_updateState) Task=`bfcd4e97-34b5-4a27-afce-0279edd14043`::moving from state preparing -> state finished >Thread-34746::DEBUG::2012-12-02 15:13:01,179::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34746::DEBUG::2012-12-02 15:13:01,179::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34746::DEBUG::2012-12-02 15:13:01,179::task::957::TaskManager.Task::(_decref) Task=`bfcd4e97-34b5-4a27-afce-0279edd14043`::ref 0 aborting False >Thread-34747::DEBUG::2012-12-02 15:13:01,185::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34747::DEBUG::2012-12-02 15:13:01,186::task::568::TaskManager.Task::(_updateState) Task=`47672d90-20ad-4c33-b818-39906882c597`::moving from state init -> state preparing >Thread-34747::INFO::2012-12-02 15:13:01,186::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34747::DEBUG::2012-12-02 15:13:01,186::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`afabb847-0a1d-4986-8803-fffce0f06a3e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-34747::DEBUG::2012-12-02 15:13:01,186::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'shared' >Thread-34747::DEBUG::2012-12-02 15:13:01,187::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'shared' (1 active user) >Thread-34747::DEBUG::2012-12-02 15:13:01,187::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`afabb847-0a1d-4986-8803-fffce0f06a3e`::Granted request >Thread-34747::DEBUG::2012-12-02 15:13:01,187::task::794::TaskManager.Task::(resourceAcquired) Task=`47672d90-20ad-4c33-b818-39906882c597`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (shared) >Thread-34747::DEBUG::2012-12-02 15:13:01,187::task::957::TaskManager.Task::(_decref) Task=`47672d90-20ad-4c33-b818-39906882c597`::ref 1 aborting False >Thread-34747::DEBUG::2012-12-02 15:13:01,193::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34747::DEBUG::2012-12-02 15:13:01,194::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a3177eae-1dc9-445a-92b1-da23ec8008bc' (cwd None) >Thread-34747::DEBUG::2012-12-02 15:13:01,399::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found\n'; <rc> = 5 >Thread-34747::WARNING::2012-12-02 15:13:01,401::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a3177eae-1dc9-445a-92b1-da23ec8008bc" not found'] >Thread-34747::DEBUG::2012-12-02 15:13:01,401::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34747::WARNING::2012-12-02 15:13:01,431::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a3177eae-1dc9-445a-92b1-da23ec8008bc',) >Thread-34747::INFO::2012-12-02 15:13:01,432::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34747::DEBUG::2012-12-02 15:13:01,432::task::1151::TaskManager.Task::(prepare) Task=`47672d90-20ad-4c33-b818-39906882c597`::finished: {'info': {'spm_id': 3, 'master_uuid': '4e891d8c-9947-468e-88ec-29e49d68e083', 'name': 'dc', 'version': '3', 'domains': 'fafd959d-695f-4b5e-aa73-aaa2634ae743:Attached,4e891d8c-9947-468e-88ec-29e49d68e083:Active,a3177eae-1dc9-445a-92b1-da23ec8008bc:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'fafd959d-695f-4b5e-aa73-aaa2634ae743': {'status': 'Attached'}, '4e891d8c-9947-468e-88ec-29e49d68e083': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a3177eae-1dc9-445a-92b1-da23ec8008bc': {'status': 'Active'}}} >Thread-34747::DEBUG::2012-12-02 15:13:01,432::task::568::TaskManager.Task::(_updateState) Task=`47672d90-20ad-4c33-b818-39906882c597`::moving from state preparing -> state finished >Thread-34747::DEBUG::2012-12-02 15:13:01,432::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-34747::DEBUG::2012-12-02 15:13:01,432::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34747::DEBUG::2012-12-02 15:13:01,432::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-34747::DEBUG::2012-12-02 15:13:01,433::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-34747::DEBUG::2012-12-02 15:13:01,433::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-34747::DEBUG::2012-12-02 15:13:01,433::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-34747::DEBUG::2012-12-02 15:13:01,433::task::957::TaskManager.Task::(_decref) Task=`47672d90-20ad-4c33-b818-39906882c597`::ref 0 aborting False >Thread-34811::DEBUG::2012-12-02 15:13:08,257::task::568::TaskManager.Task::(_updateState) Task=`08ffc9ab-ee06-4d55-8e60-ee56f6453c39`::moving from state init -> state preparing >Thread-34811::INFO::2012-12-02 15:13:08,257::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-34811::INFO::2012-12-02 15:13:08,257::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-34811::DEBUG::2012-12-02 15:13:08,257::task::1151::TaskManager.Task::(prepare) Task=`08ffc9ab-ee06-4d55-8e60-ee56f6453c39`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-34811::DEBUG::2012-12-02 15:13:08,257::task::568::TaskManager.Task::(_updateState) Task=`08ffc9ab-ee06-4d55-8e60-ee56f6453c39`::moving from state preparing -> state finished >Thread-34811::DEBUG::2012-12-02 15:13:08,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34811::DEBUG::2012-12-02 15:13:08,258::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34811::DEBUG::2012-12-02 15:13:08,258::task::957::TaskManager.Task::(_decref) Task=`08ffc9ab-ee06-4d55-8e60-ee56f6453c39`::ref 0 aborting False >Thread-32462::DEBUG::2012-12-02 15:13:10,078::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 4e891d8c-9947-468e-88ec-29e49d68e083 >Thread-32462::DEBUG::2012-12-02 15:13:10,078::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32462::DEBUG::2012-12-02 15:13:10,078::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-32462::DEBUG::2012-12-02 15:13:10,285::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-32462::WARNING::2012-12-02 15:13:10,287::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-32462::DEBUG::2012-12-02 15:13:10,287::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32462::DEBUG::2012-12-02 15:13:10,317::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-32462::DEBUG::2012-12-02 15:13:10,317::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-32462::DEBUG::2012-12-02 15:13:10,524::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-32462::WARNING::2012-12-02 15:13:10,526::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-32462::DEBUG::2012-12-02 15:13:10,526::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34934::DEBUG::2012-12-02 15:13:11,538::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34934::DEBUG::2012-12-02 15:13:11,538::task::568::TaskManager.Task::(_updateState) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::moving from state init -> state preparing >Thread-34934::INFO::2012-12-02 15:13:11,538::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34934::DEBUG::2012-12-02 15:13:11,539::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34934::DEBUG::2012-12-02 15:13:11,539::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-34934::DEBUG::2012-12-02 15:13:11,746::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-34934::WARNING::2012-12-02 15:13:11,748::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-34934::DEBUG::2012-12-02 15:13:11,748::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34934::ERROR::2012-12-02 15:13:11,777::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-34934::ERROR::2012-12-02 15:13:11,778::task::833::TaskManager.Task::(_setError) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-34934::DEBUG::2012-12-02 15:13:11,778::task::852::TaskManager.Task::(_run) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::Task._run: aebf5da8-f0ce-4dae-bc66-03a94e0a3094 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-34934::DEBUG::2012-12-02 15:13:11,778::task::1177::TaskManager.Task::(stop) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::stopping in state preparing (force False) >Thread-34934::DEBUG::2012-12-02 15:13:11,778::task::957::TaskManager.Task::(_decref) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::ref 1 aborting True >Thread-34934::INFO::2012-12-02 15:13:11,779::task::1134::TaskManager.Task::(prepare) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-34934::DEBUG::2012-12-02 15:13:11,779::task::1139::TaskManager.Task::(prepare) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::Prepare: aborted: Error validating master storage domain >Thread-34934::DEBUG::2012-12-02 15:13:11,779::task::957::TaskManager.Task::(_decref) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::ref 0 aborting True >Thread-34934::DEBUG::2012-12-02 15:13:11,779::task::892::TaskManager.Task::(_doAbort) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::Task._doAbort: force False >Thread-34934::DEBUG::2012-12-02 15:13:11,779::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34934::DEBUG::2012-12-02 15:13:11,779::task::568::TaskManager.Task::(_updateState) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::moving from state preparing -> state aborting >Thread-34934::DEBUG::2012-12-02 15:13:11,779::task::523::TaskManager.Task::(__state_aborting) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::_aborting: recover policy none >Thread-34934::DEBUG::2012-12-02 15:13:11,779::task::568::TaskManager.Task::(_updateState) Task=`aebf5da8-f0ce-4dae-bc66-03a94e0a3094`::moving from state aborting -> state failed >Thread-34934::DEBUG::2012-12-02 15:13:11,780::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34934::DEBUG::2012-12-02 15:13:11,780::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34934::ERROR::2012-12-02 15:13:11,780::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-34995::DEBUG::2012-12-02 15:13:11,870::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34995::DEBUG::2012-12-02 15:13:11,871::task::568::TaskManager.Task::(_updateState) Task=`c9b64775-cf0d-47b2-97cf-df0fb1cc5d49`::moving from state init -> state preparing >Thread-34995::INFO::2012-12-02 15:13:11,871::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-34995::DEBUG::2012-12-02 15:13:11,871::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-34995::DEBUG::2012-12-02 15:13:11,871::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-34995::INFO::2012-12-02 15:13:11,871::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-34995::DEBUG::2012-12-02 15:13:11,871::task::1151::TaskManager.Task::(prepare) Task=`c9b64775-cf0d-47b2-97cf-df0fb1cc5d49`::finished: {'allTasksStatus': {}} >Thread-34995::DEBUG::2012-12-02 15:13:11,872::task::568::TaskManager.Task::(_updateState) Task=`c9b64775-cf0d-47b2-97cf-df0fb1cc5d49`::moving from state preparing -> state finished >Thread-34995::DEBUG::2012-12-02 15:13:11,872::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34995::DEBUG::2012-12-02 15:13:11,872::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34995::DEBUG::2012-12-02 15:13:11,872::task::957::TaskManager.Task::(_decref) Task=`c9b64775-cf0d-47b2-97cf-df0fb1cc5d49`::ref 0 aborting False >Thread-34996::DEBUG::2012-12-02 15:13:11,899::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-34996::DEBUG::2012-12-02 15:13:11,899::task::568::TaskManager.Task::(_updateState) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::moving from state init -> state preparing >Thread-34996::INFO::2012-12-02 15:13:11,899::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-34996::DEBUG::2012-12-02 15:13:11,900::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-34996::DEBUG::2012-12-02 15:13:11,900::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-34996::DEBUG::2012-12-02 15:13:12,107::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-34996::WARNING::2012-12-02 15:13:12,108::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-34996::DEBUG::2012-12-02 15:13:12,109::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34996::ERROR::2012-12-02 15:13:12,137::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-34996::ERROR::2012-12-02 15:13:12,138::task::833::TaskManager.Task::(_setError) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-34996::DEBUG::2012-12-02 15:13:12,138::task::852::TaskManager.Task::(_run) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::Task._run: 2c484740-96d7-4b33-a90e-e27c4f836329 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-34996::DEBUG::2012-12-02 15:13:12,138::task::1177::TaskManager.Task::(stop) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::stopping in state preparing (force False) >Thread-34996::DEBUG::2012-12-02 15:13:12,138::task::957::TaskManager.Task::(_decref) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::ref 1 aborting True >Thread-34996::INFO::2012-12-02 15:13:12,138::task::1134::TaskManager.Task::(prepare) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-34996::DEBUG::2012-12-02 15:13:12,139::task::1139::TaskManager.Task::(prepare) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::Prepare: aborted: Error validating master storage domain >Thread-34996::DEBUG::2012-12-02 15:13:12,139::task::957::TaskManager.Task::(_decref) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::ref 0 aborting True >Thread-34996::DEBUG::2012-12-02 15:13:12,139::task::892::TaskManager.Task::(_doAbort) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::Task._doAbort: force False >Thread-34996::DEBUG::2012-12-02 15:13:12,139::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34996::DEBUG::2012-12-02 15:13:12,139::task::568::TaskManager.Task::(_updateState) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::moving from state preparing -> state aborting >Thread-34996::DEBUG::2012-12-02 15:13:12,139::task::523::TaskManager.Task::(__state_aborting) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::_aborting: recover policy none >Thread-34996::DEBUG::2012-12-02 15:13:12,139::task::568::TaskManager.Task::(_updateState) Task=`2c484740-96d7-4b33-a90e-e27c4f836329`::moving from state aborting -> state failed >Thread-34996::DEBUG::2012-12-02 15:13:12,139::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-34996::DEBUG::2012-12-02 15:13:12,140::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-34996::ERROR::2012-12-02 15:13:12,140::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-35060::DEBUG::2012-12-02 15:13:18,372::task::568::TaskManager.Task::(_updateState) Task=`14d8c168-6353-4e19-8c81-340aed40afce`::moving from state init -> state preparing >Thread-35060::INFO::2012-12-02 15:13:18,372::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-35060::INFO::2012-12-02 15:13:18,372::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35060::DEBUG::2012-12-02 15:13:18,373::task::1151::TaskManager.Task::(prepare) Task=`14d8c168-6353-4e19-8c81-340aed40afce`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35060::DEBUG::2012-12-02 15:13:18,373::task::568::TaskManager.Task::(_updateState) Task=`14d8c168-6353-4e19-8c81-340aed40afce`::moving from state preparing -> state finished >Thread-35060::DEBUG::2012-12-02 15:13:18,373::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35060::DEBUG::2012-12-02 15:13:18,373::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35060::DEBUG::2012-12-02 15:13:18,373::task::957::TaskManager.Task::(_decref) Task=`14d8c168-6353-4e19-8c81-340aed40afce`::ref 0 aborting False >Thread-35063::DEBUG::2012-12-02 15:13:22,168::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-35063::DEBUG::2012-12-02 15:13:22,169::task::568::TaskManager.Task::(_updateState) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::moving from state init -> state preparing >Thread-35063::INFO::2012-12-02 15:13:22,169::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-35063::DEBUG::2012-12-02 15:13:22,169::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-35063::DEBUG::2012-12-02 15:13:22,170::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-35063::DEBUG::2012-12-02 15:13:22,374::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-35063::WARNING::2012-12-02 15:13:22,376::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-35063::DEBUG::2012-12-02 15:13:22,376::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35063::ERROR::2012-12-02 15:13:22,411::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-35063::ERROR::2012-12-02 15:13:22,411::task::833::TaskManager.Task::(_setError) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-35063::DEBUG::2012-12-02 15:13:22,411::task::852::TaskManager.Task::(_run) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::Task._run: 255b2a3b-39c1-4a72-aed6-9fb21a057506 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-35063::DEBUG::2012-12-02 15:13:22,411::task::1177::TaskManager.Task::(stop) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::stopping in state preparing (force False) >Thread-35063::DEBUG::2012-12-02 15:13:22,412::task::957::TaskManager.Task::(_decref) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::ref 1 aborting True >Thread-35063::INFO::2012-12-02 15:13:22,412::task::1134::TaskManager.Task::(prepare) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-35063::DEBUG::2012-12-02 15:13:22,412::task::1139::TaskManager.Task::(prepare) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::Prepare: aborted: Error validating master storage domain >Thread-35063::DEBUG::2012-12-02 15:13:22,412::task::957::TaskManager.Task::(_decref) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::ref 0 aborting True >Thread-35063::DEBUG::2012-12-02 15:13:22,412::task::892::TaskManager.Task::(_doAbort) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::Task._doAbort: force False >Thread-35063::DEBUG::2012-12-02 15:13:22,412::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35063::DEBUG::2012-12-02 15:13:22,412::task::568::TaskManager.Task::(_updateState) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::moving from state preparing -> state aborting >Thread-35063::DEBUG::2012-12-02 15:13:22,413::task::523::TaskManager.Task::(__state_aborting) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::_aborting: recover policy none >Thread-35063::DEBUG::2012-12-02 15:13:22,413::task::568::TaskManager.Task::(_updateState) Task=`255b2a3b-39c1-4a72-aed6-9fb21a057506`::moving from state aborting -> state failed >Thread-35063::DEBUG::2012-12-02 15:13:22,413::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35063::DEBUG::2012-12-02 15:13:22,413::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35063::ERROR::2012-12-02 15:13:22,413::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-35127::DEBUG::2012-12-02 15:13:28,508::task::568::TaskManager.Task::(_updateState) Task=`34760015-2b93-4aee-8816-59d01c967784`::moving from state init -> state preparing >Thread-35127::INFO::2012-12-02 15:13:28,508::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-35127::INFO::2012-12-02 15:13:28,508::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35127::DEBUG::2012-12-02 15:13:28,508::task::1151::TaskManager.Task::(prepare) Task=`34760015-2b93-4aee-8816-59d01c967784`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35127::DEBUG::2012-12-02 15:13:28,509::task::568::TaskManager.Task::(_updateState) Task=`34760015-2b93-4aee-8816-59d01c967784`::moving from state preparing -> state finished >Thread-35127::DEBUG::2012-12-02 15:13:28,509::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35127::DEBUG::2012-12-02 15:13:28,509::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35127::DEBUG::2012-12-02 15:13:28,509::task::957::TaskManager.Task::(_decref) Task=`34760015-2b93-4aee-8816-59d01c967784`::ref 0 aborting False >Thread-35130::DEBUG::2012-12-02 15:13:32,445::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-35130::DEBUG::2012-12-02 15:13:32,446::task::568::TaskManager.Task::(_updateState) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::moving from state init -> state preparing >Thread-35130::INFO::2012-12-02 15:13:32,446::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-35130::DEBUG::2012-12-02 15:13:32,446::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-35130::DEBUG::2012-12-02 15:13:32,447::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-35130::DEBUG::2012-12-02 15:13:32,652::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-35130::WARNING::2012-12-02 15:13:32,654::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-35130::DEBUG::2012-12-02 15:13:32,654::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35130::ERROR::2012-12-02 15:13:32,684::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-35130::ERROR::2012-12-02 15:13:32,685::task::833::TaskManager.Task::(_setError) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-35130::DEBUG::2012-12-02 15:13:32,685::task::852::TaskManager.Task::(_run) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::Task._run: 7c7eb6a3-8fd5-48e9-8b18-705eefc7a981 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-35130::DEBUG::2012-12-02 15:13:32,685::task::1177::TaskManager.Task::(stop) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::stopping in state preparing (force False) >Thread-35130::DEBUG::2012-12-02 15:13:32,685::task::957::TaskManager.Task::(_decref) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::ref 1 aborting True >Thread-35130::INFO::2012-12-02 15:13:32,685::task::1134::TaskManager.Task::(prepare) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-35130::DEBUG::2012-12-02 15:13:32,685::task::1139::TaskManager.Task::(prepare) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::Prepare: aborted: Error validating master storage domain >Thread-35130::DEBUG::2012-12-02 15:13:32,685::task::957::TaskManager.Task::(_decref) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::ref 0 aborting True >Thread-35130::DEBUG::2012-12-02 15:13:32,686::task::892::TaskManager.Task::(_doAbort) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::Task._doAbort: force False >Thread-35130::DEBUG::2012-12-02 15:13:32,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35130::DEBUG::2012-12-02 15:13:32,686::task::568::TaskManager.Task::(_updateState) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::moving from state preparing -> state aborting >Thread-35130::DEBUG::2012-12-02 15:13:32,686::task::523::TaskManager.Task::(__state_aborting) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::_aborting: recover policy none >Thread-35130::DEBUG::2012-12-02 15:13:32,686::task::568::TaskManager.Task::(_updateState) Task=`7c7eb6a3-8fd5-48e9-8b18-705eefc7a981`::moving from state aborting -> state failed >Thread-35130::DEBUG::2012-12-02 15:13:32,686::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35130::DEBUG::2012-12-02 15:13:32,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35130::ERROR::2012-12-02 15:13:32,686::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-35194::DEBUG::2012-12-02 15:13:38,621::task::568::TaskManager.Task::(_updateState) Task=`efcfb1fd-b096-41bd-b27e-0add47c364ad`::moving from state init -> state preparing >Thread-35194::INFO::2012-12-02 15:13:38,621::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-35194::INFO::2012-12-02 15:13:38,621::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35194::DEBUG::2012-12-02 15:13:38,622::task::1151::TaskManager.Task::(prepare) Task=`efcfb1fd-b096-41bd-b27e-0add47c364ad`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35194::DEBUG::2012-12-02 15:13:38,622::task::568::TaskManager.Task::(_updateState) Task=`efcfb1fd-b096-41bd-b27e-0add47c364ad`::moving from state preparing -> state finished >Thread-35194::DEBUG::2012-12-02 15:13:38,622::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35194::DEBUG::2012-12-02 15:13:38,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35194::DEBUG::2012-12-02 15:13:38,622::task::957::TaskManager.Task::(_decref) Task=`efcfb1fd-b096-41bd-b27e-0add47c364ad`::ref 0 aborting False >Thread-35197::DEBUG::2012-12-02 15:13:42,705::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-35197::DEBUG::2012-12-02 15:13:42,706::task::568::TaskManager.Task::(_updateState) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::moving from state init -> state preparing >Thread-35197::INFO::2012-12-02 15:13:42,706::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-35197::DEBUG::2012-12-02 15:13:42,706::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-35197::DEBUG::2012-12-02 15:13:42,706::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-35197::DEBUG::2012-12-02 15:13:42,914::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-35197::WARNING::2012-12-02 15:13:42,916::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-35197::DEBUG::2012-12-02 15:13:42,916::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35197::ERROR::2012-12-02 15:13:42,946::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-35197::ERROR::2012-12-02 15:13:42,946::task::833::TaskManager.Task::(_setError) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-35197::DEBUG::2012-12-02 15:13:42,946::task::852::TaskManager.Task::(_run) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::Task._run: 263ca435-229e-4b28-9d5e-4c0cb0286c28 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-35197::DEBUG::2012-12-02 15:13:42,946::task::1177::TaskManager.Task::(stop) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::stopping in state preparing (force False) >Thread-35197::DEBUG::2012-12-02 15:13:42,947::task::957::TaskManager.Task::(_decref) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::ref 1 aborting True >Thread-35197::INFO::2012-12-02 15:13:42,947::task::1134::TaskManager.Task::(prepare) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-35197::DEBUG::2012-12-02 15:13:42,947::task::1139::TaskManager.Task::(prepare) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::Prepare: aborted: Error validating master storage domain >Thread-35197::DEBUG::2012-12-02 15:13:42,947::task::957::TaskManager.Task::(_decref) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::ref 0 aborting True >Thread-35197::DEBUG::2012-12-02 15:13:42,947::task::892::TaskManager.Task::(_doAbort) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::Task._doAbort: force False >Thread-35197::DEBUG::2012-12-02 15:13:42,947::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35197::DEBUG::2012-12-02 15:13:42,947::task::568::TaskManager.Task::(_updateState) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::moving from state preparing -> state aborting >Thread-35197::DEBUG::2012-12-02 15:13:42,948::task::523::TaskManager.Task::(__state_aborting) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::_aborting: recover policy none >Thread-35197::DEBUG::2012-12-02 15:13:42,948::task::568::TaskManager.Task::(_updateState) Task=`263ca435-229e-4b28-9d5e-4c0cb0286c28`::moving from state aborting -> state failed >Thread-35197::DEBUG::2012-12-02 15:13:42,948::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35197::DEBUG::2012-12-02 15:13:42,948::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35197::ERROR::2012-12-02 15:13:42,948::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-35261::DEBUG::2012-12-02 15:13:48,744::task::568::TaskManager.Task::(_updateState) Task=`e68e60b5-6a56-4b78-b717-e999fd63ee28`::moving from state init -> state preparing >Thread-35261::INFO::2012-12-02 15:13:48,744::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-35261::INFO::2012-12-02 15:13:48,744::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35261::DEBUG::2012-12-02 15:13:48,744::task::1151::TaskManager.Task::(prepare) Task=`e68e60b5-6a56-4b78-b717-e999fd63ee28`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35261::DEBUG::2012-12-02 15:13:48,744::task::568::TaskManager.Task::(_updateState) Task=`e68e60b5-6a56-4b78-b717-e999fd63ee28`::moving from state preparing -> state finished >Thread-35261::DEBUG::2012-12-02 15:13:48,744::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35261::DEBUG::2012-12-02 15:13:48,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35261::DEBUG::2012-12-02 15:13:48,745::task::957::TaskManager.Task::(_decref) Task=`e68e60b5-6a56-4b78-b717-e999fd63ee28`::ref 0 aborting False >Thread-35265::DEBUG::2012-12-02 15:13:52,978::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-35265::DEBUG::2012-12-02 15:13:52,979::task::568::TaskManager.Task::(_updateState) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::moving from state init -> state preparing >Thread-35265::INFO::2012-12-02 15:13:52,979::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-35265::DEBUG::2012-12-02 15:13:52,979::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-35265::DEBUG::2012-12-02 15:13:52,980::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-35265::DEBUG::2012-12-02 15:13:53,185::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-35265::WARNING::2012-12-02 15:13:53,187::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-35265::DEBUG::2012-12-02 15:13:53,187::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35265::ERROR::2012-12-02 15:13:53,216::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-35265::ERROR::2012-12-02 15:13:53,217::task::833::TaskManager.Task::(_setError) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-35265::DEBUG::2012-12-02 15:13:53,217::task::852::TaskManager.Task::(_run) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::Task._run: 16b948ab-2f25-4098-9ae9-df9dadfc2eb6 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-35265::DEBUG::2012-12-02 15:13:53,217::task::1177::TaskManager.Task::(stop) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::stopping in state preparing (force False) >Thread-35265::DEBUG::2012-12-02 15:13:53,217::task::957::TaskManager.Task::(_decref) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::ref 1 aborting True >Thread-35265::INFO::2012-12-02 15:13:53,217::task::1134::TaskManager.Task::(prepare) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-35265::DEBUG::2012-12-02 15:13:53,218::task::1139::TaskManager.Task::(prepare) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::Prepare: aborted: Error validating master storage domain >Thread-35265::DEBUG::2012-12-02 15:13:53,218::task::957::TaskManager.Task::(_decref) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::ref 0 aborting True >Thread-35265::DEBUG::2012-12-02 15:13:53,218::task::892::TaskManager.Task::(_doAbort) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::Task._doAbort: force False >Thread-35265::DEBUG::2012-12-02 15:13:53,218::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35265::DEBUG::2012-12-02 15:13:53,218::task::568::TaskManager.Task::(_updateState) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::moving from state preparing -> state aborting >Thread-35265::DEBUG::2012-12-02 15:13:53,218::task::523::TaskManager.Task::(__state_aborting) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::_aborting: recover policy none >Thread-35265::DEBUG::2012-12-02 15:13:53,218::task::568::TaskManager.Task::(_updateState) Task=`16b948ab-2f25-4098-9ae9-df9dadfc2eb6`::moving from state aborting -> state failed >Thread-35265::DEBUG::2012-12-02 15:13:53,218::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35265::DEBUG::2012-12-02 15:13:53,219::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35265::ERROR::2012-12-02 15:13:53,219::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-35328::DEBUG::2012-12-02 15:13:58,840::task::568::TaskManager.Task::(_updateState) Task=`ef740a54-f6db-49ff-bf9e-30209f9046ba`::moving from state init -> state preparing >Thread-35328::INFO::2012-12-02 15:13:58,841::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-35328::INFO::2012-12-02 15:13:58,841::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35328::DEBUG::2012-12-02 15:13:58,841::task::1151::TaskManager.Task::(prepare) Task=`ef740a54-f6db-49ff-bf9e-30209f9046ba`::finished: {'4e891d8c-9947-468e-88ec-29e49d68e083': {'delay': '0.0062141418457', 'lastCheck': 1354453980.0778921, 'code': 0, 'valid': True}} >Thread-35328::DEBUG::2012-12-02 15:13:58,841::task::568::TaskManager.Task::(_updateState) Task=`ef740a54-f6db-49ff-bf9e-30209f9046ba`::moving from state preparing -> state finished >Thread-35328::DEBUG::2012-12-02 15:13:58,841::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35328::DEBUG::2012-12-02 15:13:58,841::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35328::DEBUG::2012-12-02 15:13:58,842::task::957::TaskManager.Task::(_decref) Task=`ef740a54-f6db-49ff-bf9e-30209f9046ba`::ref 0 aborting False >Thread-35332::DEBUG::2012-12-02 15:14:03,244::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-35332::DEBUG::2012-12-02 15:14:03,245::task::568::TaskManager.Task::(_updateState) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::moving from state init -> state preparing >Thread-35332::INFO::2012-12-02 15:14:03,245::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-35332::DEBUG::2012-12-02 15:14:03,245::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-35332::DEBUG::2012-12-02 15:14:03,245::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-35332::DEBUG::2012-12-02 15:14:03,451::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-35332::WARNING::2012-12-02 15:14:03,452::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-35332::DEBUG::2012-12-02 15:14:03,452::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35332::ERROR::2012-12-02 15:14:03,483::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('4e891d8c-9947-468e-88ec-29e49d68e083',) >Thread-35332::ERROR::2012-12-02 15:14:03,483::task::833::TaskManager.Task::(_setError) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-35332::DEBUG::2012-12-02 15:14:03,484::task::852::TaskManager.Task::(_run) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::Task._run: 34f0619a-b793-49b0-84ca-0c52f5f8c3eb ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-35332::DEBUG::2012-12-02 15:14:03,484::task::1177::TaskManager.Task::(stop) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::stopping in state preparing (force False) >Thread-35332::DEBUG::2012-12-02 15:14:03,484::task::957::TaskManager.Task::(_decref) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::ref 1 aborting True >Thread-35332::INFO::2012-12-02 15:14:03,484::task::1134::TaskManager.Task::(prepare) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-35332::DEBUG::2012-12-02 15:14:03,484::task::1139::TaskManager.Task::(prepare) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::Prepare: aborted: Error validating master storage domain >Thread-35332::DEBUG::2012-12-02 15:14:03,484::task::957::TaskManager.Task::(_decref) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::ref 0 aborting True >Thread-35332::DEBUG::2012-12-02 15:14:03,484::task::892::TaskManager.Task::(_doAbort) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::Task._doAbort: force False >Thread-35332::DEBUG::2012-12-02 15:14:03,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35332::DEBUG::2012-12-02 15:14:03,485::task::568::TaskManager.Task::(_updateState) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::moving from state preparing -> state aborting >Thread-35332::DEBUG::2012-12-02 15:14:03,485::task::523::TaskManager.Task::(__state_aborting) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::_aborting: recover policy none >Thread-35332::DEBUG::2012-12-02 15:14:03,485::task::568::TaskManager.Task::(_updateState) Task=`34f0619a-b793-49b0-84ca-0c52f5f8c3eb`::moving from state aborting -> state failed >Thread-35332::DEBUG::2012-12-02 15:14:03,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-35332::DEBUG::2012-12-02 15:14:03,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-35332::ERROR::2012-12-02 15:14:03,485::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >MainThread::INFO::2012-12-02 15:14:06,937::vmChannels::135::vds::(stop) VM channels listener was stopped. >MainThread::DEBUG::2012-12-02 15:14:06,938::task::568::TaskManager.Task::(_updateState) Task=`4625d4d6-1f82-4dff-9a06-157cd26ce889`::moving from state init -> state preparing >MainThread::DEBUG::2012-12-02 15:14:06,938::clientIF::198::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently >MainThread::INFO::2012-12-02 15:14:06,938::logUtils::37::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2012-12-02 15:14:06,939::storageServer::617::ConnectionMonitor::(_monitorConnections) Monitoring stopped >MainThread::DEBUG::2012-12-02 15:14:06,947::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >MainThread::DEBUG::2012-12-02 15:14:06,999::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks >MainThread::INFO::2012-12-02 15:14:06,999::logUtils::39::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None >MainThread::DEBUG::2012-12-02 15:14:06,999::task::1151::TaskManager.Task::(prepare) Task=`4625d4d6-1f82-4dff-9a06-157cd26ce889`::finished: None >MainThread::DEBUG::2012-12-02 15:14:06,999::task::568::TaskManager.Task::(_updateState) Task=`4625d4d6-1f82-4dff-9a06-157cd26ce889`::moving from state preparing -> state finished >MainThread::DEBUG::2012-12-02 15:14:06,999::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2012-12-02 15:14:06,999::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2012-12-02 15:14:06,999::task::957::TaskManager.Task::(_decref) Task=`4625d4d6-1f82-4dff-9a06-157cd26ce889`::ref 0 aborting False >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::76::vds::(run) VDSM main thread ended. Waiting for 13 other threads... >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <_MainThread(MainThread, started 139740808939264)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <Thread(libvirtEventLoop, started daemon 139740730083072)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(Thread-2, started daemon 139740679059200)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(983edf27-03aa-409f-a9e0-7d7652f9f4c4, started daemon 139740689549056)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(Thread-4, started daemon 139740451960576)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <KsmMonitorThread(KsmMonitor, started daemon 139739940239104)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(03768cbd-23ec-42bc-a704-af83f38ea3ed, started daemon 139740441470720)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(7c46563b-f50c-410c-b631-8a10e75835fc, started daemon 139740420491008)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(e143b971-cde5-4f18-92eb-a3750573504d, started daemon 139740430980864)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(a10d4e41-3aaf-44fe-8e96-96b8c3caeace, started daemon 139740410001152)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(a1638662-19ff-4644-9c5b-5db7283d2365, started daemon 139739982198528)> >MainThread::INFO::2012-12-02 15:14:07,000::vdsm::79::vds::(run) <WorkerThread(Thread-3, started daemon 139740668569344)> >MainThread::INFO::2012-12-02 15:14:07,000::vmChannels::135::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2012-12-02 15:14:07,001::vdsm::79::vds::(run) <Listener(VM Channels Listener, started daemon 139739512436480)> >MainThread::INFO::2012-12-02 15:14:07,001::vdsm::79::vds::(run) <WorkerThread(Thread-9, started daemon 139740399511296)> >MainThread::INFO::2012-12-02 15:14:07,063::vdsm::71::vds::(run) I am the actual vdsm 4.10-0.239 >MainThread::DEBUG::2012-12-02 15:14:07,200::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2012-12-02 15:14:07,200::threadPool::45::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::DEBUG::2012-12-02 15:14:07,207::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >MainThread::DEBUG::2012-12-02 15:14:07,230::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/kill -9 5531' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:07,238::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 15:14:07,238::supervdsm::91::SuperVdsmProxy::(_launchSupervdsm) Launching Super Vdsm >MainThread::DEBUG::2012-12-02 15:14:07,238::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /usr/bin/python /usr/share/vdsm/supervdsmServer.py 8ebfa8cd-e13b-4b15-9f22-186b6c933925 18638' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:07,357::supervdsmServer::306::SuperVdsm.Server::(main) Making sure I'm root >MainThread::DEBUG::2012-12-02 15:14:07,357::supervdsmServer::310::SuperVdsm.Server::(main) Parsing cmd args >MainThread::DEBUG::2012-12-02 15:14:07,357::supervdsmServer::313::SuperVdsm.Server::(main) Creating PID file >MainThread::DEBUG::2012-12-02 15:14:07,357::supervdsmServer::317::SuperVdsm.Server::(main) Cleaning old socket >MainThread::DEBUG::2012-12-02 15:14:07,357::supervdsmServer::321::SuperVdsm.Server::(main) Setting up keep alive thread >MainThread::DEBUG::2012-12-02 15:14:07,357::supervdsmServer::327::SuperVdsm.Server::(main) Creating remote object manager >MainThread::DEBUG::2012-12-02 15:14:07,358::supervdsmServer::338::SuperVdsm.Server::(main) Started serving super vdsm object >MainThread::DEBUG::2012-12-02 15:14:09,243::supervdsm::113::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2012-12-02 15:14:09,247::supervdsm::84::SuperVdsmProxy::(__init__) Connected to Super Vdsm >MainThread::DEBUG::2012-12-02 15:14:09,248::multipath::106::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2012-12-02 15:14:09,248::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:09,319::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 15:14:09,320::lvm::334::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2012-12-02 15:14:09,321::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:09,629::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >MainThread::DEBUG::2012-12-02 15:14:09,629::lvm::359::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >MainThread::DEBUG::2012-12-02 15:14:09,629::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2012-12-02 15:14:09,630::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:10,015::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >MainThread::DEBUG::2012-12-02 15:14:10,019::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >MainThread::DEBUG::2012-12-02 15:14:10,019::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:10,360::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >Thread-12::DEBUG::2012-12-02 15:14:10,376::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >MainThread::INFO::2012-12-02 15:14:10,377::dispatcher::95::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >Thread-12::DEBUG::2012-12-02 15:14:10,377::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-12::DEBUG::2012-12-02 15:14:10,378::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-12::DEBUG::2012-12-02 15:14:10,378::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-12::DEBUG::2012-12-02 15:14:10,379::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:10,385::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/pgrep -xf ksmd' (cwd None) >MainThread::DEBUG::2012-12-02 15:14:10,396::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::INFO::2012-12-02 15:14:10,397::vmChannels::139::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2012-12-02 15:14:10,398::vmChannels::127::vds::(run) Starting VM channels listener thread. >MainThread::WARNING::2012-12-02 15:14:10,401::clientIF::169::vds::(_prepareBindings) Unable to load the rest server module. Please make sure it is installed. >Thread-12::DEBUG::2012-12-02 15:14:10,406::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-12::DEBUG::2012-12-02 15:14:10,407::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-14::DEBUG::2012-12-02 15:14:11,033::BindingXMLRPC::872::vds::(wrapper) client [10.35.1.163]::call getCapabilities with () {} flowID [183d383b] >Thread-12::DEBUG::2012-12-02 15:14:11,051::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-14::DEBUG::2012-12-02 15:14:11,127::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-12::DEBUG::2012-12-02 15:14:11,131::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-12::DEBUG::2012-12-02 15:14:11,132::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 15:14:11,134::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 15:14:11,134::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 15:14:11,134::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 15:14:11,135::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 15:14:11,135::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 15:14:11,135::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-12::DEBUG::2012-12-02 15:14:11,135::hsm::355::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >Thread-14::DEBUG::2012-12-02 15:14:11,139::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-14::DEBUG::2012-12-02 15:14:11,141::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-12::DEBUG::2012-12-02 15:14:11,142::hsm::387::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >Thread-12::DEBUG::2012-12-02 15:14:11,146::hsm::388::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_101', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0941', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0851', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sdds1', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sdds2', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_export1', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_data3', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510'] >Thread-12::DEBUG::2012-12-02 15:14:11,146::hsm::390::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >Thread-12::DEBUG::2012-12-02 15:14:11,149::hsm::433::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >Thread-14::DEBUG::2012-12-02 15:14:11,154::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-14::DEBUG::2012-12-02 15:14:11,154::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-14::DEBUG::2012-12-02 15:14:11,164::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 15:14:11,164::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-14::DEBUG::2012-12-02 15:14:11,175::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 15:14:11,175::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-14::DEBUG::2012-12-02 15:14:11,187::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 15:14:11,399::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '0.0.0.0'}} >Thread-16::DEBUG::2012-12-02 15:14:11,675::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-16::DEBUG::2012-12-02 15:14:11,676::task::568::TaskManager.Task::(_updateState) Task=`af48b8da-86d0-4e6c-a8bc-85c4f43bd84f`::moving from state init -> state preparing >Thread-16::INFO::2012-12-02 15:14:11,676::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/1447', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175', 'port': ''}], options=None) >Thread-16::DEBUG::2012-12-02 15:14:11,682::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2012-12-02 15:14:11,682::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2012-12-02 15:14:11,683::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2012-12-02 15:14:11,683::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2012-12-02 15:14:11,683::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2012-12-02 15:14:11,683::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::INFO::2012-12-02 15:14:11,683::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-16::DEBUG::2012-12-02 15:14:11,683::task::1151::TaskManager.Task::(prepare) Task=`af48b8da-86d0-4e6c-a8bc-85c4f43bd84f`::finished: {'statuslist': [{'status': 0, 'id': 'a2b7b0c8-b680-45db-a7c1-e0d6bf4c5175'}]} >Thread-16::DEBUG::2012-12-02 15:14:11,683::task::568::TaskManager.Task::(_updateState) Task=`af48b8da-86d0-4e6c-a8bc-85c4f43bd84f`::moving from state preparing -> state finished >Thread-16::DEBUG::2012-12-02 15:14:11,684::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-16::DEBUG::2012-12-02 15:14:11,684::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-16::DEBUG::2012-12-02 15:14:11,684::task::957::TaskManager.Task::(_decref) Task=`af48b8da-86d0-4e6c-a8bc-85c4f43bd84f`::ref 0 aborting False >Thread-17::DEBUG::2012-12-02 15:14:11,740::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-17::DEBUG::2012-12-02 15:14:11,741::task::568::TaskManager.Task::(_updateState) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::moving from state init -> state preparing >Thread-17::INFO::2012-12-02 15:14:11,741::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-17::DEBUG::2012-12-02 15:14:11,741::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c85b233b-5860-45ac-961f-58da0b1edd6e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-17::DEBUG::2012-12-02 15:14:11,741::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-17::DEBUG::2012-12-02 15:14:11,742::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-17::DEBUG::2012-12-02 15:14:11,742::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`c85b233b-5860-45ac-961f-58da0b1edd6e`::Granted request >Thread-17::INFO::2012-12-02 15:14:11,742::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-17::DEBUG::2012-12-02 15:14:11,743::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:11,743::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:11,743::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:11,743::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:11,743::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:11,743::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:11,743::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-17::DEBUG::2012-12-02 15:14:11,744::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-17::DEBUG::2012-12-02 15:14:11,744::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-17::DEBUG::2012-12-02 15:14:11,744::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-17::DEBUG::2012-12-02 15:14:11,744::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-17::DEBUG::2012-12-02 15:14:11,774::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-17::DEBUG::2012-12-02 15:14:11,774::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-17::DEBUG::2012-12-02 15:14:12,408::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-17::DEBUG::2012-12-02 15:14:12,511::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-17::DEBUG::2012-12-02 15:14:12,511::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,511::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,512::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,512::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,512::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,512::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,512::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-17::DEBUG::2012-12-02 15:14:12,512::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,514::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-17::DEBUG::2012-12-02 15:14:12,723::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-17::WARNING::2012-12-02 15:14:12,725::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-17::DEBUG::2012-12-02 15:14:12,725::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 15:14:12,834::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-17::DEBUG::2012-12-02 15:14:12,834::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-17::DEBUG::2012-12-02 15:14:12,835::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-17::DEBUG::2012-12-02 15:14:12,835::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-17::ERROR::2012-12-02 15:14:12,835::task::833::TaskManager.Task::(_setError) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-17::DEBUG::2012-12-02 15:14:12,837::task::852::TaskManager.Task::(_run) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::Task._run: 43a48400-a3a1-4c74-a717-2c3aedbbd69d ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-17::DEBUG::2012-12-02 15:14:12,837::task::1177::TaskManager.Task::(stop) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::stopping in state preparing (force False) >Thread-17::DEBUG::2012-12-02 15:14:12,837::task::957::TaskManager.Task::(_decref) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::ref 1 aborting True >Thread-17::INFO::2012-12-02 15:14:12,837::task::1134::TaskManager.Task::(prepare) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-17::DEBUG::2012-12-02 15:14:12,837::task::1139::TaskManager.Task::(prepare) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::Prepare: aborted: Cannot find master domain >Thread-17::DEBUG::2012-12-02 15:14:12,838::task::957::TaskManager.Task::(_decref) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::ref 0 aborting True >Thread-17::DEBUG::2012-12-02 15:14:12,838::task::892::TaskManager.Task::(_doAbort) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::Task._doAbort: force False >Thread-17::DEBUG::2012-12-02 15:14:12,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-17::DEBUG::2012-12-02 15:14:12,838::task::568::TaskManager.Task::(_updateState) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::moving from state preparing -> state aborting >Thread-17::DEBUG::2012-12-02 15:14:12,838::task::523::TaskManager.Task::(__state_aborting) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::_aborting: recover policy none >Thread-17::DEBUG::2012-12-02 15:14:12,838::task::568::TaskManager.Task::(_updateState) Task=`43a48400-a3a1-4c74-a717-2c3aedbbd69d`::moving from state aborting -> state failed >Thread-17::DEBUG::2012-12-02 15:14:12,838::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-17::DEBUG::2012-12-02 15:14:12,839::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-17::ERROR::2012-12-02 15:14:12,839::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-78::DEBUG::2012-12-02 15:14:13,517::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-78::DEBUG::2012-12-02 15:14:13,517::task::568::TaskManager.Task::(_updateState) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::moving from state init -> state preparing >Thread-78::INFO::2012-12-02 15:14:13,517::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-78::ERROR::2012-12-02 15:14:13,517::task::833::TaskManager.Task::(_setError) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-78::DEBUG::2012-12-02 15:14:13,518::task::852::TaskManager.Task::(_run) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::Task._run: 5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-78::DEBUG::2012-12-02 15:14:13,518::task::1177::TaskManager.Task::(stop) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::stopping in state preparing (force False) >Thread-78::DEBUG::2012-12-02 15:14:13,518::task::957::TaskManager.Task::(_decref) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::ref 1 aborting True >Thread-78::INFO::2012-12-02 15:14:13,518::task::1134::TaskManager.Task::(prepare) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-78::DEBUG::2012-12-02 15:14:13,518::task::1139::TaskManager.Task::(prepare) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::Prepare: aborted: Unknown pool id, pool not connected >Thread-78::DEBUG::2012-12-02 15:14:13,518::task::957::TaskManager.Task::(_decref) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::ref 0 aborting True >Thread-78::DEBUG::2012-12-02 15:14:13,519::task::892::TaskManager.Task::(_doAbort) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::Task._doAbort: force False >Thread-78::DEBUG::2012-12-02 15:14:13,519::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-78::DEBUG::2012-12-02 15:14:13,519::task::568::TaskManager.Task::(_updateState) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::moving from state preparing -> state aborting >Thread-78::DEBUG::2012-12-02 15:14:13,519::task::523::TaskManager.Task::(__state_aborting) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::_aborting: recover policy none >Thread-78::DEBUG::2012-12-02 15:14:13,519::task::568::TaskManager.Task::(_updateState) Task=`5e6f4339-4c2e-41f7-ba04-ab23a6bfc59d`::moving from state aborting -> state failed >Thread-78::DEBUG::2012-12-02 15:14:13,519::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-78::DEBUG::2012-12-02 15:14:13,519::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-78::ERROR::2012-12-02 15:14:13,519::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-79::DEBUG::2012-12-02 15:14:13,589::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-79::DEBUG::2012-12-02 15:14:13,590::task::568::TaskManager.Task::(_updateState) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::moving from state init -> state preparing >Thread-79::INFO::2012-12-02 15:14:13,590::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-79::DEBUG::2012-12-02 15:14:13,590::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`cfb4d9bb-b54d-4b4c-b776-ff304777c450`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-79::DEBUG::2012-12-02 15:14:13,590::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-79::DEBUG::2012-12-02 15:14:13,591::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-79::DEBUG::2012-12-02 15:14:13,591::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`cfb4d9bb-b54d-4b4c-b776-ff304777c450`::Granted request >Thread-79::INFO::2012-12-02 15:14:13,591::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-79::DEBUG::2012-12-02 15:14:13,591::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:13,592::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:13,592::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:13,592::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:13,592::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:13,592::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:13,592::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-79::DEBUG::2012-12-02 15:14:13,592::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-79::DEBUG::2012-12-02 15:14:13,593::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-79::DEBUG::2012-12-02 15:14:13,593::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-79::DEBUG::2012-12-02 15:14:13,593::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-79::DEBUG::2012-12-02 15:14:13,617::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-79::DEBUG::2012-12-02 15:14:13,618::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-79::DEBUG::2012-12-02 15:14:14,250::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-79::DEBUG::2012-12-02 15:14:14,315::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-79::DEBUG::2012-12-02 15:14:14,316::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,316::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,316::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,316::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,317::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,317::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,317::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-79::DEBUG::2012-12-02 15:14:14,317::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,319::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-79::DEBUG::2012-12-02 15:14:14,525::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-79::WARNING::2012-12-02 15:14:14,526::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-79::DEBUG::2012-12-02 15:14:14,527::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-79::DEBUG::2012-12-02 15:14:14,563::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-79::DEBUG::2012-12-02 15:14:14,563::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-79::DEBUG::2012-12-02 15:14:14,563::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-79::DEBUG::2012-12-02 15:14:14,563::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-79::ERROR::2012-12-02 15:14:14,563::task::833::TaskManager.Task::(_setError) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-79::DEBUG::2012-12-02 15:14:14,564::task::852::TaskManager.Task::(_run) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::Task._run: 0a39c7b4-8772-462a-bc34-295255becfb4 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-79::DEBUG::2012-12-02 15:14:14,564::task::1177::TaskManager.Task::(stop) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::stopping in state preparing (force False) >Thread-79::DEBUG::2012-12-02 15:14:14,564::task::957::TaskManager.Task::(_decref) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::ref 1 aborting True >Thread-79::INFO::2012-12-02 15:14:14,564::task::1134::TaskManager.Task::(prepare) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-79::DEBUG::2012-12-02 15:14:14,564::task::1139::TaskManager.Task::(prepare) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::Prepare: aborted: Cannot find master domain >Thread-79::DEBUG::2012-12-02 15:14:14,564::task::957::TaskManager.Task::(_decref) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::ref 0 aborting True >Thread-79::DEBUG::2012-12-02 15:14:14,565::task::892::TaskManager.Task::(_doAbort) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::Task._doAbort: force False >Thread-79::DEBUG::2012-12-02 15:14:14,565::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-79::DEBUG::2012-12-02 15:14:14,565::task::568::TaskManager.Task::(_updateState) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::moving from state preparing -> state aborting >Thread-79::DEBUG::2012-12-02 15:14:14,565::task::523::TaskManager.Task::(__state_aborting) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::_aborting: recover policy none >Thread-79::DEBUG::2012-12-02 15:14:14,565::task::568::TaskManager.Task::(_updateState) Task=`0a39c7b4-8772-462a-bc34-295255becfb4`::moving from state aborting -> state failed >Thread-79::DEBUG::2012-12-02 15:14:14,565::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-79::DEBUG::2012-12-02 15:14:14,565::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-79::ERROR::2012-12-02 15:14:14,565::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-142::DEBUG::2012-12-02 15:14:17,750::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-142::DEBUG::2012-12-02 15:14:17,750::task::568::TaskManager.Task::(_updateState) Task=`fb10d486-160e-49cd-b50e-1f94fa05495e`::moving from state init -> state preparing >Thread-142::INFO::2012-12-02 15:14:17,751::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-142::DEBUG::2012-12-02 15:14:17,751::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-142::DEBUG::2012-12-02 15:14:17,751::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-142::INFO::2012-12-02 15:14:17,751::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-142::DEBUG::2012-12-02 15:14:17,751::task::1151::TaskManager.Task::(prepare) Task=`fb10d486-160e-49cd-b50e-1f94fa05495e`::finished: {'allTasksStatus': {}} >Thread-142::DEBUG::2012-12-02 15:14:17,752::task::568::TaskManager.Task::(_updateState) Task=`fb10d486-160e-49cd-b50e-1f94fa05495e`::moving from state preparing -> state finished >Thread-142::DEBUG::2012-12-02 15:14:17,752::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-142::DEBUG::2012-12-02 15:14:17,752::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-142::DEBUG::2012-12-02 15:14:17,752::task::957::TaskManager.Task::(_decref) Task=`fb10d486-160e-49cd-b50e-1f94fa05495e`::ref 0 aborting False >Thread-143::DEBUG::2012-12-02 15:14:17,765::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-143::DEBUG::2012-12-02 15:14:17,765::task::568::TaskManager.Task::(_updateState) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::moving from state init -> state preparing >Thread-143::INFO::2012-12-02 15:14:17,765::logUtils::37::dispatcher::(wrapper) Run and protect: spmStop(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-143::DEBUG::2012-12-02 15:14:17,766::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`29735285-50b4-47d0-a4a5-44856a17003a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-143::DEBUG::2012-12-02 15:14:17,766::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-143::DEBUG::2012-12-02 15:14:17,766::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-143::DEBUG::2012-12-02 15:14:17,766::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`29735285-50b4-47d0-a4a5-44856a17003a`::Granted request >Thread-143::DEBUG::2012-12-02 15:14:17,766::task::794::TaskManager.Task::(resourceAcquired) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::_resourcesAcquired: Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6 (exclusive) >Thread-143::DEBUG::2012-12-02 15:14:17,767::task::957::TaskManager.Task::(_decref) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::ref 1 aborting False >Thread-143::ERROR::2012-12-02 15:14:17,767::task::833::TaskManager.Task::(_setError) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 516, in spmStop > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-143::DEBUG::2012-12-02 15:14:17,767::task::852::TaskManager.Task::(_run) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::Task._run: 46e68b7a-1b1b-4949-a096-436af64e97dc ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-143::DEBUG::2012-12-02 15:14:17,767::task::1177::TaskManager.Task::(stop) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::stopping in state preparing (force False) >Thread-143::DEBUG::2012-12-02 15:14:17,767::task::957::TaskManager.Task::(_decref) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::ref 1 aborting True >Thread-143::INFO::2012-12-02 15:14:17,767::task::1134::TaskManager.Task::(prepare) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-143::DEBUG::2012-12-02 15:14:17,767::task::1139::TaskManager.Task::(prepare) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::Prepare: aborted: Unknown pool id, pool not connected >Thread-143::DEBUG::2012-12-02 15:14:17,768::task::957::TaskManager.Task::(_decref) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::ref 0 aborting True >Thread-143::DEBUG::2012-12-02 15:14:17,768::task::892::TaskManager.Task::(_doAbort) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::Task._doAbort: force False >Thread-143::DEBUG::2012-12-02 15:14:17,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-143::DEBUG::2012-12-02 15:14:17,768::task::568::TaskManager.Task::(_updateState) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::moving from state preparing -> state aborting >Thread-143::DEBUG::2012-12-02 15:14:17,768::task::523::TaskManager.Task::(__state_aborting) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::_aborting: recover policy none >Thread-143::DEBUG::2012-12-02 15:14:17,768::task::568::TaskManager.Task::(_updateState) Task=`46e68b7a-1b1b-4949-a096-436af64e97dc`::moving from state aborting -> state failed >Thread-143::DEBUG::2012-12-02 15:14:17,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6': < ResourceRef 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', isValid: 'True' obj: 'None'>} >Thread-143::DEBUG::2012-12-02 15:14:17,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-143::DEBUG::2012-12-02 15:14:17,769::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-143::DEBUG::2012-12-02 15:14:17,769::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-143::DEBUG::2012-12-02 15:14:17,769::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-143::DEBUG::2012-12-02 15:14:17,769::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-143::ERROR::2012-12-02 15:14:17,769::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-145::DEBUG::2012-12-02 15:14:20,966::task::568::TaskManager.Task::(_updateState) Task=`3afef219-4ffa-4ba1-b5fb-9a17a42e697d`::moving from state init -> state preparing >Thread-145::INFO::2012-12-02 15:14:20,966::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-145::INFO::2012-12-02 15:14:20,966::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-145::DEBUG::2012-12-02 15:14:20,966::task::1151::TaskManager.Task::(prepare) Task=`3afef219-4ffa-4ba1-b5fb-9a17a42e697d`::finished: {} >Thread-145::DEBUG::2012-12-02 15:14:20,966::task::568::TaskManager.Task::(_updateState) Task=`3afef219-4ffa-4ba1-b5fb-9a17a42e697d`::moving from state preparing -> state finished >Thread-145::DEBUG::2012-12-02 15:14:20,967::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-145::DEBUG::2012-12-02 15:14:20,967::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-145::DEBUG::2012-12-02 15:14:20,967::task::957::TaskManager.Task::(_decref) Task=`3afef219-4ffa-4ba1-b5fb-9a17a42e697d`::ref 0 aborting False >Thread-151::DEBUG::2012-12-02 15:14:31,076::task::568::TaskManager.Task::(_updateState) Task=`d2870912-4e10-4272-b730-510ce3512f34`::moving from state init -> state preparing >Thread-151::INFO::2012-12-02 15:14:31,076::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-151::INFO::2012-12-02 15:14:31,076::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-151::DEBUG::2012-12-02 15:14:31,076::task::1151::TaskManager.Task::(prepare) Task=`d2870912-4e10-4272-b730-510ce3512f34`::finished: {} >Thread-151::DEBUG::2012-12-02 15:14:31,077::task::568::TaskManager.Task::(_updateState) Task=`d2870912-4e10-4272-b730-510ce3512f34`::moving from state preparing -> state finished >Thread-151::DEBUG::2012-12-02 15:14:31,077::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-151::DEBUG::2012-12-02 15:14:31,077::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-151::DEBUG::2012-12-02 15:14:31,077::task::957::TaskManager.Task::(_decref) Task=`d2870912-4e10-4272-b730-510ce3512f34`::ref 0 aborting False >Thread-157::DEBUG::2012-12-02 15:14:41,197::task::568::TaskManager.Task::(_updateState) Task=`2072e084-eb2a-4aa3-b64b-d6849534d247`::moving from state init -> state preparing >Thread-157::INFO::2012-12-02 15:14:41,197::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-157::INFO::2012-12-02 15:14:41,198::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-157::DEBUG::2012-12-02 15:14:41,198::task::1151::TaskManager.Task::(prepare) Task=`2072e084-eb2a-4aa3-b64b-d6849534d247`::finished: {} >Thread-157::DEBUG::2012-12-02 15:14:41,198::task::568::TaskManager.Task::(_updateState) Task=`2072e084-eb2a-4aa3-b64b-d6849534d247`::moving from state preparing -> state finished >Thread-157::DEBUG::2012-12-02 15:14:41,198::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-157::DEBUG::2012-12-02 15:14:41,198::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-157::DEBUG::2012-12-02 15:14:41,198::task::957::TaskManager.Task::(_decref) Task=`2072e084-eb2a-4aa3-b64b-d6849534d247`::ref 0 aborting False >Thread-161::DEBUG::2012-12-02 15:14:46,815::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-161::DEBUG::2012-12-02 15:14:46,815::task::568::TaskManager.Task::(_updateState) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::moving from state init -> state preparing >Thread-161::INFO::2012-12-02 15:14:46,816::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-161::ERROR::2012-12-02 15:14:46,816::task::833::TaskManager.Task::(_setError) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-161::DEBUG::2012-12-02 15:14:46,816::task::852::TaskManager.Task::(_run) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::Task._run: 2c4bef54-c537-48b4-b353-d4ae4afe7c84 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-161::DEBUG::2012-12-02 15:14:46,816::task::1177::TaskManager.Task::(stop) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::stopping in state preparing (force False) >Thread-161::DEBUG::2012-12-02 15:14:46,816::task::957::TaskManager.Task::(_decref) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::ref 1 aborting True >Thread-161::INFO::2012-12-02 15:14:46,816::task::1134::TaskManager.Task::(prepare) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-161::DEBUG::2012-12-02 15:14:46,817::task::1139::TaskManager.Task::(prepare) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::Prepare: aborted: Unknown pool id, pool not connected >Thread-161::DEBUG::2012-12-02 15:14:46,817::task::957::TaskManager.Task::(_decref) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::ref 0 aborting True >Thread-161::DEBUG::2012-12-02 15:14:46,817::task::892::TaskManager.Task::(_doAbort) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::Task._doAbort: force False >Thread-161::DEBUG::2012-12-02 15:14:46,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-161::DEBUG::2012-12-02 15:14:46,817::task::568::TaskManager.Task::(_updateState) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::moving from state preparing -> state aborting >Thread-161::DEBUG::2012-12-02 15:14:46,817::task::523::TaskManager.Task::(__state_aborting) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::_aborting: recover policy none >Thread-161::DEBUG::2012-12-02 15:14:46,817::task::568::TaskManager.Task::(_updateState) Task=`2c4bef54-c537-48b4-b353-d4ae4afe7c84`::moving from state aborting -> state failed >Thread-161::DEBUG::2012-12-02 15:14:46,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-161::DEBUG::2012-12-02 15:14:46,818::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-161::ERROR::2012-12-02 15:14:46,818::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-162::DEBUG::2012-12-02 15:14:46,856::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-162::DEBUG::2012-12-02 15:14:46,856::task::568::TaskManager.Task::(_updateState) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::moving from state init -> state preparing >Thread-162::INFO::2012-12-02 15:14:46,856::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-162::DEBUG::2012-12-02 15:14:46,857::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b4bff0e8-d146-4d6a-aba4-6eeebc35a000`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-162::DEBUG::2012-12-02 15:14:46,857::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-162::DEBUG::2012-12-02 15:14:46,857::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-162::DEBUG::2012-12-02 15:14:46,857::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b4bff0e8-d146-4d6a-aba4-6eeebc35a000`::Granted request >Thread-162::INFO::2012-12-02 15:14:46,857::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-162::DEBUG::2012-12-02 15:14:46,858::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:46,858::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:46,858::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:46,858::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:46,858::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:46,859::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:46,859::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-162::DEBUG::2012-12-02 15:14:46,859::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-162::DEBUG::2012-12-02 15:14:46,859::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-162::DEBUG::2012-12-02 15:14:46,859::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-162::DEBUG::2012-12-02 15:14:46,859::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-162::DEBUG::2012-12-02 15:14:46,886::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-162::DEBUG::2012-12-02 15:14:46,886::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-162::DEBUG::2012-12-02 15:14:47,518::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-162::DEBUG::2012-12-02 15:14:47,589::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,590::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-162::DEBUG::2012-12-02 15:14:47,590::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,592::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-162::DEBUG::2012-12-02 15:14:47,800::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-162::WARNING::2012-12-02 15:14:47,802::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-162::DEBUG::2012-12-02 15:14:47,802::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-162::DEBUG::2012-12-02 15:14:47,836::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-162::DEBUG::2012-12-02 15:14:47,836::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-162::DEBUG::2012-12-02 15:14:47,836::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-162::DEBUG::2012-12-02 15:14:47,836::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-162::ERROR::2012-12-02 15:14:47,836::task::833::TaskManager.Task::(_setError) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-162::DEBUG::2012-12-02 15:14:47,837::task::852::TaskManager.Task::(_run) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::Task._run: 17265c0c-4154-4e0f-aaff-84cc4a734630 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-162::DEBUG::2012-12-02 15:14:47,837::task::1177::TaskManager.Task::(stop) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::stopping in state preparing (force False) >Thread-162::DEBUG::2012-12-02 15:14:47,837::task::957::TaskManager.Task::(_decref) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::ref 1 aborting True >Thread-162::INFO::2012-12-02 15:14:47,837::task::1134::TaskManager.Task::(prepare) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-162::DEBUG::2012-12-02 15:14:47,837::task::1139::TaskManager.Task::(prepare) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::Prepare: aborted: Cannot find master domain >Thread-162::DEBUG::2012-12-02 15:14:47,838::task::957::TaskManager.Task::(_decref) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::ref 0 aborting True >Thread-162::DEBUG::2012-12-02 15:14:47,838::task::892::TaskManager.Task::(_doAbort) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::Task._doAbort: force False >Thread-162::DEBUG::2012-12-02 15:14:47,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-162::DEBUG::2012-12-02 15:14:47,838::task::568::TaskManager.Task::(_updateState) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::moving from state preparing -> state aborting >Thread-162::DEBUG::2012-12-02 15:14:47,838::task::523::TaskManager.Task::(__state_aborting) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::_aborting: recover policy none >Thread-162::DEBUG::2012-12-02 15:14:47,838::task::568::TaskManager.Task::(_updateState) Task=`17265c0c-4154-4e0f-aaff-84cc4a734630`::moving from state aborting -> state failed >Thread-162::DEBUG::2012-12-02 15:14:47,838::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-162::DEBUG::2012-12-02 15:14:47,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-162::ERROR::2012-12-02 15:14:47,839::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-225::DEBUG::2012-12-02 15:14:51,316::task::568::TaskManager.Task::(_updateState) Task=`5f55d1a2-ec91-47e0-a2b1-64f8ee6e21c2`::moving from state init -> state preparing >Thread-225::INFO::2012-12-02 15:14:51,316::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-225::INFO::2012-12-02 15:14:51,317::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-225::DEBUG::2012-12-02 15:14:51,317::task::1151::TaskManager.Task::(prepare) Task=`5f55d1a2-ec91-47e0-a2b1-64f8ee6e21c2`::finished: {} >Thread-225::DEBUG::2012-12-02 15:14:51,317::task::568::TaskManager.Task::(_updateState) Task=`5f55d1a2-ec91-47e0-a2b1-64f8ee6e21c2`::moving from state preparing -> state finished >Thread-225::DEBUG::2012-12-02 15:14:51,317::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-225::DEBUG::2012-12-02 15:14:51,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-225::DEBUG::2012-12-02 15:14:51,317::task::957::TaskManager.Task::(_decref) Task=`5f55d1a2-ec91-47e0-a2b1-64f8ee6e21c2`::ref 0 aborting False >Thread-231::DEBUG::2012-12-02 15:15:01,102::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-231::DEBUG::2012-12-02 15:15:01,102::task::568::TaskManager.Task::(_updateState) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::moving from state init -> state preparing >Thread-231::INFO::2012-12-02 15:15:01,102::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-231::ERROR::2012-12-02 15:15:01,102::task::833::TaskManager.Task::(_setError) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-231::DEBUG::2012-12-02 15:15:01,103::task::852::TaskManager.Task::(_run) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::Task._run: c162385d-d943-449d-bb21-0d47bd42a11a ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-231::DEBUG::2012-12-02 15:15:01,103::task::1177::TaskManager.Task::(stop) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::stopping in state preparing (force False) >Thread-231::DEBUG::2012-12-02 15:15:01,103::task::957::TaskManager.Task::(_decref) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::ref 1 aborting True >Thread-231::INFO::2012-12-02 15:15:01,104::task::1134::TaskManager.Task::(prepare) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-231::DEBUG::2012-12-02 15:15:01,104::task::1139::TaskManager.Task::(prepare) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::Prepare: aborted: Unknown pool id, pool not connected >Thread-231::DEBUG::2012-12-02 15:15:01,104::task::957::TaskManager.Task::(_decref) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::ref 0 aborting True >Thread-231::DEBUG::2012-12-02 15:15:01,104::task::892::TaskManager.Task::(_doAbort) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::Task._doAbort: force False >Thread-231::DEBUG::2012-12-02 15:15:01,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-231::DEBUG::2012-12-02 15:15:01,104::task::568::TaskManager.Task::(_updateState) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::moving from state preparing -> state aborting >Thread-231::DEBUG::2012-12-02 15:15:01,104::task::523::TaskManager.Task::(__state_aborting) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::_aborting: recover policy none >Thread-231::DEBUG::2012-12-02 15:15:01,104::task::568::TaskManager.Task::(_updateState) Task=`c162385d-d943-449d-bb21-0d47bd42a11a`::moving from state aborting -> state failed >Thread-231::DEBUG::2012-12-02 15:15:01,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-231::DEBUG::2012-12-02 15:15:01,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-231::ERROR::2012-12-02 15:15:01,105::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-232::DEBUG::2012-12-02 15:15:01,114::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-232::DEBUG::2012-12-02 15:15:01,114::task::568::TaskManager.Task::(_updateState) Task=`92810b5a-1472-4ea0-b756-02d08f453803`::moving from state init -> state preparing >Thread-232::INFO::2012-12-02 15:15:01,114::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-232::DEBUG::2012-12-02 15:15:01,114::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-232::DEBUG::2012-12-02 15:15:01,114::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-232::INFO::2012-12-02 15:15:01,115::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-232::DEBUG::2012-12-02 15:15:01,115::task::1151::TaskManager.Task::(prepare) Task=`92810b5a-1472-4ea0-b756-02d08f453803`::finished: {'allTasksStatus': {}} >Thread-232::DEBUG::2012-12-02 15:15:01,115::task::568::TaskManager.Task::(_updateState) Task=`92810b5a-1472-4ea0-b756-02d08f453803`::moving from state preparing -> state finished >Thread-232::DEBUG::2012-12-02 15:15:01,115::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-232::DEBUG::2012-12-02 15:15:01,115::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-232::DEBUG::2012-12-02 15:15:01,115::task::957::TaskManager.Task::(_decref) Task=`92810b5a-1472-4ea0-b756-02d08f453803`::ref 0 aborting False >Thread-233::DEBUG::2012-12-02 15:15:01,126::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-233::DEBUG::2012-12-02 15:15:01,126::task::568::TaskManager.Task::(_updateState) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::moving from state init -> state preparing >Thread-233::INFO::2012-12-02 15:15:01,126::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-233::ERROR::2012-12-02 15:15:01,126::task::833::TaskManager.Task::(_setError) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-233::DEBUG::2012-12-02 15:15:01,127::task::852::TaskManager.Task::(_run) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::Task._run: 77695fa4-6c8c-4e7f-920a-34b16fbe051f ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-233::DEBUG::2012-12-02 15:15:01,127::task::1177::TaskManager.Task::(stop) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::stopping in state preparing (force False) >Thread-233::DEBUG::2012-12-02 15:15:01,127::task::957::TaskManager.Task::(_decref) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::ref 1 aborting True >Thread-233::INFO::2012-12-02 15:15:01,127::task::1134::TaskManager.Task::(prepare) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-233::DEBUG::2012-12-02 15:15:01,127::task::1139::TaskManager.Task::(prepare) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::Prepare: aborted: Unknown pool id, pool not connected >Thread-233::DEBUG::2012-12-02 15:15:01,127::task::957::TaskManager.Task::(_decref) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::ref 0 aborting True >Thread-233::DEBUG::2012-12-02 15:15:01,128::task::892::TaskManager.Task::(_doAbort) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::Task._doAbort: force False >Thread-233::DEBUG::2012-12-02 15:15:01,128::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-233::DEBUG::2012-12-02 15:15:01,128::task::568::TaskManager.Task::(_updateState) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::moving from state preparing -> state aborting >Thread-233::DEBUG::2012-12-02 15:15:01,128::task::523::TaskManager.Task::(__state_aborting) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::_aborting: recover policy none >Thread-233::DEBUG::2012-12-02 15:15:01,128::task::568::TaskManager.Task::(_updateState) Task=`77695fa4-6c8c-4e7f-920a-34b16fbe051f`::moving from state aborting -> state failed >Thread-233::DEBUG::2012-12-02 15:15:01,128::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-233::DEBUG::2012-12-02 15:15:01,128::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-233::ERROR::2012-12-02 15:15:01,128::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-234::DEBUG::2012-12-02 15:15:01,194::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-234::DEBUG::2012-12-02 15:15:01,194::task::568::TaskManager.Task::(_updateState) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::moving from state init -> state preparing >Thread-234::INFO::2012-12-02 15:15:01,194::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-234::DEBUG::2012-12-02 15:15:01,195::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`ae064af7-f993-4647-b251-e099b2ed92e8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-234::DEBUG::2012-12-02 15:15:01,195::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-234::DEBUG::2012-12-02 15:15:01,195::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-234::DEBUG::2012-12-02 15:15:01,195::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`ae064af7-f993-4647-b251-e099b2ed92e8`::Granted request >Thread-234::INFO::2012-12-02 15:15:01,195::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-234::DEBUG::2012-12-02 15:15:01,195::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,196::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,196::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,196::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,196::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,196::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,196::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-234::DEBUG::2012-12-02 15:15:01,196::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-234::DEBUG::2012-12-02 15:15:01,196::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-234::DEBUG::2012-12-02 15:15:01,196::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-234::DEBUG::2012-12-02 15:15:01,196::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-234::DEBUG::2012-12-02 15:15:01,220::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-234::DEBUG::2012-12-02 15:15:01,221::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-235::DEBUG::2012-12-02 15:15:01,428::task::568::TaskManager.Task::(_updateState) Task=`9c0e308f-9a98-40f7-bc17-efbb903ed8d1`::moving from state init -> state preparing >Thread-235::INFO::2012-12-02 15:15:01,428::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-235::INFO::2012-12-02 15:15:01,428::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-235::DEBUG::2012-12-02 15:15:01,428::task::1151::TaskManager.Task::(prepare) Task=`9c0e308f-9a98-40f7-bc17-efbb903ed8d1`::finished: {} >Thread-235::DEBUG::2012-12-02 15:15:01,428::task::568::TaskManager.Task::(_updateState) Task=`9c0e308f-9a98-40f7-bc17-efbb903ed8d1`::moving from state preparing -> state finished >Thread-235::DEBUG::2012-12-02 15:15:01,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-235::DEBUG::2012-12-02 15:15:01,428::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-235::DEBUG::2012-12-02 15:15:01,429::task::957::TaskManager.Task::(_decref) Task=`9c0e308f-9a98-40f7-bc17-efbb903ed8d1`::ref 0 aborting False >Thread-234::DEBUG::2012-12-02 15:15:01,853::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-234::DEBUG::2012-12-02 15:15:01,926::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-234::DEBUG::2012-12-02 15:15:01,926::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,927::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,927::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,927::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,927::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,927::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,927::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-234::DEBUG::2012-12-02 15:15:01,928::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:01,929::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-234::DEBUG::2012-12-02 15:15:02,136::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-234::WARNING::2012-12-02 15:15:02,137::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-234::DEBUG::2012-12-02 15:15:02,138::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-234::DEBUG::2012-12-02 15:15:02,166::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-234::DEBUG::2012-12-02 15:15:02,167::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-234::DEBUG::2012-12-02 15:15:02,167::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-234::DEBUG::2012-12-02 15:15:02,167::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-234::ERROR::2012-12-02 15:15:02,167::task::833::TaskManager.Task::(_setError) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-234::DEBUG::2012-12-02 15:15:02,167::task::852::TaskManager.Task::(_run) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::Task._run: 2fb36e84-cd3a-4603-8032-a8298e711d55 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-234::DEBUG::2012-12-02 15:15:02,168::task::1177::TaskManager.Task::(stop) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::stopping in state preparing (force False) >Thread-234::DEBUG::2012-12-02 15:15:02,168::task::957::TaskManager.Task::(_decref) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::ref 1 aborting True >Thread-234::INFO::2012-12-02 15:15:02,168::task::1134::TaskManager.Task::(prepare) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-234::DEBUG::2012-12-02 15:15:02,168::task::1139::TaskManager.Task::(prepare) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::Prepare: aborted: Cannot find master domain >Thread-234::DEBUG::2012-12-02 15:15:02,168::task::957::TaskManager.Task::(_decref) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::ref 0 aborting True >Thread-234::DEBUG::2012-12-02 15:15:02,168::task::892::TaskManager.Task::(_doAbort) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::Task._doAbort: force False >Thread-234::DEBUG::2012-12-02 15:15:02,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-234::DEBUG::2012-12-02 15:15:02,169::task::568::TaskManager.Task::(_updateState) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::moving from state preparing -> state aborting >Thread-234::DEBUG::2012-12-02 15:15:02,169::task::523::TaskManager.Task::(__state_aborting) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::_aborting: recover policy none >Thread-234::DEBUG::2012-12-02 15:15:02,169::task::568::TaskManager.Task::(_updateState) Task=`2fb36e84-cd3a-4603-8032-a8298e711d55`::moving from state aborting -> state failed >Thread-234::DEBUG::2012-12-02 15:15:02,169::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-234::DEBUG::2012-12-02 15:15:02,169::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-234::ERROR::2012-12-02 15:15:02,169::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-301::DEBUG::2012-12-02 15:15:11,559::task::568::TaskManager.Task::(_updateState) Task=`9b26af66-0b53-42ba-895b-cf69b3b70e5d`::moving from state init -> state preparing >Thread-301::INFO::2012-12-02 15:15:11,559::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-301::INFO::2012-12-02 15:15:11,559::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-301::DEBUG::2012-12-02 15:15:11,560::task::1151::TaskManager.Task::(prepare) Task=`9b26af66-0b53-42ba-895b-cf69b3b70e5d`::finished: {} >Thread-301::DEBUG::2012-12-02 15:15:11,560::task::568::TaskManager.Task::(_updateState) Task=`9b26af66-0b53-42ba-895b-cf69b3b70e5d`::moving from state preparing -> state finished >Thread-301::DEBUG::2012-12-02 15:15:11,560::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-301::DEBUG::2012-12-02 15:15:11,560::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-301::DEBUG::2012-12-02 15:15:11,560::task::957::TaskManager.Task::(_decref) Task=`9b26af66-0b53-42ba-895b-cf69b3b70e5d`::ref 0 aborting False >Thread-307::DEBUG::2012-12-02 15:15:21,676::task::568::TaskManager.Task::(_updateState) Task=`51f6a2e3-6d4d-4f6f-890e-d41694ac48c0`::moving from state init -> state preparing >Thread-307::INFO::2012-12-02 15:15:21,676::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-307::INFO::2012-12-02 15:15:21,676::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-307::DEBUG::2012-12-02 15:15:21,676::task::1151::TaskManager.Task::(prepare) Task=`51f6a2e3-6d4d-4f6f-890e-d41694ac48c0`::finished: {} >Thread-307::DEBUG::2012-12-02 15:15:21,676::task::568::TaskManager.Task::(_updateState) Task=`51f6a2e3-6d4d-4f6f-890e-d41694ac48c0`::moving from state preparing -> state finished >Thread-307::DEBUG::2012-12-02 15:15:21,676::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-307::DEBUG::2012-12-02 15:15:21,676::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-307::DEBUG::2012-12-02 15:15:21,677::task::957::TaskManager.Task::(_decref) Task=`51f6a2e3-6d4d-4f6f-890e-d41694ac48c0`::ref 0 aborting False >Thread-313::DEBUG::2012-12-02 15:15:31,781::task::568::TaskManager.Task::(_updateState) Task=`427d54e5-14d8-427e-a488-e165c89c6535`::moving from state init -> state preparing >Thread-313::INFO::2012-12-02 15:15:31,781::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-313::INFO::2012-12-02 15:15:31,781::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-313::DEBUG::2012-12-02 15:15:31,781::task::1151::TaskManager.Task::(prepare) Task=`427d54e5-14d8-427e-a488-e165c89c6535`::finished: {} >Thread-313::DEBUG::2012-12-02 15:15:31,781::task::568::TaskManager.Task::(_updateState) Task=`427d54e5-14d8-427e-a488-e165c89c6535`::moving from state preparing -> state finished >Thread-313::DEBUG::2012-12-02 15:15:31,782::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-313::DEBUG::2012-12-02 15:15:31,782::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-313::DEBUG::2012-12-02 15:15:31,782::task::957::TaskManager.Task::(_decref) Task=`427d54e5-14d8-427e-a488-e165c89c6535`::ref 0 aborting False >Thread-315::DEBUG::2012-12-02 15:15:33,589::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-315::DEBUG::2012-12-02 15:15:33,589::task::568::TaskManager.Task::(_updateState) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::moving from state init -> state preparing >Thread-315::INFO::2012-12-02 15:15:33,589::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-315::ERROR::2012-12-02 15:15:33,590::task::833::TaskManager.Task::(_setError) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-315::DEBUG::2012-12-02 15:15:33,590::task::852::TaskManager.Task::(_run) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::Task._run: 3eaf2ef2-efa0-47e9-82d9-c6febbba715d ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-315::DEBUG::2012-12-02 15:15:33,590::task::1177::TaskManager.Task::(stop) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::stopping in state preparing (force False) >Thread-315::DEBUG::2012-12-02 15:15:33,590::task::957::TaskManager.Task::(_decref) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::ref 1 aborting True >Thread-315::INFO::2012-12-02 15:15:33,590::task::1134::TaskManager.Task::(prepare) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-315::DEBUG::2012-12-02 15:15:33,590::task::1139::TaskManager.Task::(prepare) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::Prepare: aborted: Unknown pool id, pool not connected >Thread-315::DEBUG::2012-12-02 15:15:33,590::task::957::TaskManager.Task::(_decref) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::ref 0 aborting True >Thread-315::DEBUG::2012-12-02 15:15:33,591::task::892::TaskManager.Task::(_doAbort) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::Task._doAbort: force False >Thread-315::DEBUG::2012-12-02 15:15:33,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-315::DEBUG::2012-12-02 15:15:33,591::task::568::TaskManager.Task::(_updateState) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::moving from state preparing -> state aborting >Thread-315::DEBUG::2012-12-02 15:15:33,591::task::523::TaskManager.Task::(__state_aborting) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::_aborting: recover policy none >Thread-315::DEBUG::2012-12-02 15:15:33,591::task::568::TaskManager.Task::(_updateState) Task=`3eaf2ef2-efa0-47e9-82d9-c6febbba715d`::moving from state aborting -> state failed >Thread-315::DEBUG::2012-12-02 15:15:33,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-315::DEBUG::2012-12-02 15:15:33,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-315::ERROR::2012-12-02 15:15:33,591::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-316::DEBUG::2012-12-02 15:15:33,600::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-316::DEBUG::2012-12-02 15:15:33,601::task::568::TaskManager.Task::(_updateState) Task=`bf14e6b6-928b-4942-b036-e6ef2e74b192`::moving from state init -> state preparing >Thread-316::INFO::2012-12-02 15:15:33,601::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-316::DEBUG::2012-12-02 15:15:33,601::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-316::DEBUG::2012-12-02 15:15:33,601::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-316::INFO::2012-12-02 15:15:33,601::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-316::DEBUG::2012-12-02 15:15:33,601::task::1151::TaskManager.Task::(prepare) Task=`bf14e6b6-928b-4942-b036-e6ef2e74b192`::finished: {'allTasksStatus': {}} >Thread-316::DEBUG::2012-12-02 15:15:33,601::task::568::TaskManager.Task::(_updateState) Task=`bf14e6b6-928b-4942-b036-e6ef2e74b192`::moving from state preparing -> state finished >Thread-316::DEBUG::2012-12-02 15:15:33,602::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-316::DEBUG::2012-12-02 15:15:33,602::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-316::DEBUG::2012-12-02 15:15:33,602::task::957::TaskManager.Task::(_decref) Task=`bf14e6b6-928b-4942-b036-e6ef2e74b192`::ref 0 aborting False >Thread-317::DEBUG::2012-12-02 15:15:33,612::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-317::DEBUG::2012-12-02 15:15:33,612::task::568::TaskManager.Task::(_updateState) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::moving from state init -> state preparing >Thread-317::INFO::2012-12-02 15:15:33,612::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-317::ERROR::2012-12-02 15:15:33,612::task::833::TaskManager.Task::(_setError) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-317::DEBUG::2012-12-02 15:15:33,613::task::852::TaskManager.Task::(_run) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::Task._run: 9cb66a91-2bc1-487f-a7e6-7d4c42944a2b ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-317::DEBUG::2012-12-02 15:15:33,613::task::1177::TaskManager.Task::(stop) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::stopping in state preparing (force False) >Thread-317::DEBUG::2012-12-02 15:15:33,613::task::957::TaskManager.Task::(_decref) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::ref 1 aborting True >Thread-317::INFO::2012-12-02 15:15:33,613::task::1134::TaskManager.Task::(prepare) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-317::DEBUG::2012-12-02 15:15:33,613::task::1139::TaskManager.Task::(prepare) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::Prepare: aborted: Unknown pool id, pool not connected >Thread-317::DEBUG::2012-12-02 15:15:33,613::task::957::TaskManager.Task::(_decref) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::ref 0 aborting True >Thread-317::DEBUG::2012-12-02 15:15:33,613::task::892::TaskManager.Task::(_doAbort) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::Task._doAbort: force False >Thread-317::DEBUG::2012-12-02 15:15:33,614::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-317::DEBUG::2012-12-02 15:15:33,614::task::568::TaskManager.Task::(_updateState) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::moving from state preparing -> state aborting >Thread-317::DEBUG::2012-12-02 15:15:33,614::task::523::TaskManager.Task::(__state_aborting) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::_aborting: recover policy none >Thread-317::DEBUG::2012-12-02 15:15:33,614::task::568::TaskManager.Task::(_updateState) Task=`9cb66a91-2bc1-487f-a7e6-7d4c42944a2b`::moving from state aborting -> state failed >Thread-317::DEBUG::2012-12-02 15:15:33,614::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-317::DEBUG::2012-12-02 15:15:33,614::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-317::ERROR::2012-12-02 15:15:33,614::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-318::DEBUG::2012-12-02 15:15:33,705::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-318::DEBUG::2012-12-02 15:15:33,705::task::568::TaskManager.Task::(_updateState) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::moving from state init -> state preparing >Thread-318::INFO::2012-12-02 15:15:33,706::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-318::DEBUG::2012-12-02 15:15:33,706::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`743e239a-f75b-4bf1-a003-49e89878007e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-318::DEBUG::2012-12-02 15:15:33,706::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-318::DEBUG::2012-12-02 15:15:33,706::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-318::DEBUG::2012-12-02 15:15:33,706::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`743e239a-f75b-4bf1-a003-49e89878007e`::Granted request >Thread-318::INFO::2012-12-02 15:15:33,707::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-318::DEBUG::2012-12-02 15:15:33,707::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:33,707::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:33,707::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:33,707::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:33,708::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:33,708::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:33,708::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-318::DEBUG::2012-12-02 15:15:33,708::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-318::DEBUG::2012-12-02 15:15:33,708::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-318::DEBUG::2012-12-02 15:15:33,708::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-318::DEBUG::2012-12-02 15:15:33,708::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-318::DEBUG::2012-12-02 15:15:33,733::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-318::DEBUG::2012-12-02 15:15:33,733::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-318::DEBUG::2012-12-02 15:15:34,366::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-318::DEBUG::2012-12-02 15:15:34,431::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-318::DEBUG::2012-12-02 15:15:34,431::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,431::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,431::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,432::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,432::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,432::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,432::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-318::DEBUG::2012-12-02 15:15:34,432::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,434::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-318::DEBUG::2012-12-02 15:15:34,640::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-318::WARNING::2012-12-02 15:15:34,642::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-318::DEBUG::2012-12-02 15:15:34,642::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-318::DEBUG::2012-12-02 15:15:34,671::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-318::DEBUG::2012-12-02 15:15:34,671::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-318::DEBUG::2012-12-02 15:15:34,672::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-318::DEBUG::2012-12-02 15:15:34,672::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-318::ERROR::2012-12-02 15:15:34,672::task::833::TaskManager.Task::(_setError) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-318::DEBUG::2012-12-02 15:15:34,672::task::852::TaskManager.Task::(_run) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::Task._run: f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-318::DEBUG::2012-12-02 15:15:34,672::task::1177::TaskManager.Task::(stop) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::stopping in state preparing (force False) >Thread-318::DEBUG::2012-12-02 15:15:34,673::task::957::TaskManager.Task::(_decref) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::ref 1 aborting True >Thread-318::INFO::2012-12-02 15:15:34,673::task::1134::TaskManager.Task::(prepare) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-318::DEBUG::2012-12-02 15:15:34,673::task::1139::TaskManager.Task::(prepare) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::Prepare: aborted: Cannot find master domain >Thread-318::DEBUG::2012-12-02 15:15:34,673::task::957::TaskManager.Task::(_decref) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::ref 0 aborting True >Thread-318::DEBUG::2012-12-02 15:15:34,673::task::892::TaskManager.Task::(_doAbort) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::Task._doAbort: force False >Thread-318::DEBUG::2012-12-02 15:15:34,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-318::DEBUG::2012-12-02 15:15:34,673::task::568::TaskManager.Task::(_updateState) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::moving from state preparing -> state aborting >Thread-318::DEBUG::2012-12-02 15:15:34,674::task::523::TaskManager.Task::(__state_aborting) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::_aborting: recover policy none >Thread-318::DEBUG::2012-12-02 15:15:34,674::task::568::TaskManager.Task::(_updateState) Task=`f2a6ae9b-82b6-48f9-a1cb-a51accdfe40c`::moving from state aborting -> state failed >Thread-318::DEBUG::2012-12-02 15:15:34,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-318::DEBUG::2012-12-02 15:15:34,674::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-318::ERROR::2012-12-02 15:15:34,674::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-383::DEBUG::2012-12-02 15:15:41,897::task::568::TaskManager.Task::(_updateState) Task=`391c277f-809e-4d2d-ac0a-35f7084c82b1`::moving from state init -> state preparing >Thread-383::INFO::2012-12-02 15:15:41,897::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-383::INFO::2012-12-02 15:15:41,897::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-383::DEBUG::2012-12-02 15:15:41,898::task::1151::TaskManager.Task::(prepare) Task=`391c277f-809e-4d2d-ac0a-35f7084c82b1`::finished: {} >Thread-383::DEBUG::2012-12-02 15:15:41,898::task::568::TaskManager.Task::(_updateState) Task=`391c277f-809e-4d2d-ac0a-35f7084c82b1`::moving from state preparing -> state finished >Thread-383::DEBUG::2012-12-02 15:15:41,898::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-383::DEBUG::2012-12-02 15:15:41,898::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-383::DEBUG::2012-12-02 15:15:41,898::task::957::TaskManager.Task::(_decref) Task=`391c277f-809e-4d2d-ac0a-35f7084c82b1`::ref 0 aborting False >Thread-386::DEBUG::2012-12-02 15:15:44,891::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-386::DEBUG::2012-12-02 15:15:44,891::task::568::TaskManager.Task::(_updateState) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::moving from state init -> state preparing >Thread-386::INFO::2012-12-02 15:15:44,891::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-386::ERROR::2012-12-02 15:15:44,891::task::833::TaskManager.Task::(_setError) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-386::DEBUG::2012-12-02 15:15:44,892::task::852::TaskManager.Task::(_run) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::Task._run: ffdd52e3-c7f3-4a93-955a-efffd3adb4d6 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-386::DEBUG::2012-12-02 15:15:44,892::task::1177::TaskManager.Task::(stop) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::stopping in state preparing (force False) >Thread-386::DEBUG::2012-12-02 15:15:44,892::task::957::TaskManager.Task::(_decref) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::ref 1 aborting True >Thread-386::INFO::2012-12-02 15:15:44,892::task::1134::TaskManager.Task::(prepare) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-386::DEBUG::2012-12-02 15:15:44,892::task::1139::TaskManager.Task::(prepare) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::Prepare: aborted: Unknown pool id, pool not connected >Thread-386::DEBUG::2012-12-02 15:15:44,892::task::957::TaskManager.Task::(_decref) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::ref 0 aborting True >Thread-386::DEBUG::2012-12-02 15:15:44,892::task::892::TaskManager.Task::(_doAbort) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::Task._doAbort: force False >Thread-386::DEBUG::2012-12-02 15:15:44,893::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-386::DEBUG::2012-12-02 15:15:44,893::task::568::TaskManager.Task::(_updateState) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::moving from state preparing -> state aborting >Thread-386::DEBUG::2012-12-02 15:15:44,893::task::523::TaskManager.Task::(__state_aborting) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::_aborting: recover policy none >Thread-386::DEBUG::2012-12-02 15:15:44,893::task::568::TaskManager.Task::(_updateState) Task=`ffdd52e3-c7f3-4a93-955a-efffd3adb4d6`::moving from state aborting -> state failed >Thread-386::DEBUG::2012-12-02 15:15:44,893::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-386::DEBUG::2012-12-02 15:15:44,893::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-386::ERROR::2012-12-02 15:15:44,893::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-387::DEBUG::2012-12-02 15:15:44,902::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-387::DEBUG::2012-12-02 15:15:44,902::task::568::TaskManager.Task::(_updateState) Task=`1e85a89b-ee0f-4b00-9a8c-518ab324b171`::moving from state init -> state preparing >Thread-387::INFO::2012-12-02 15:15:44,902::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-387::DEBUG::2012-12-02 15:15:44,902::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-387::DEBUG::2012-12-02 15:15:44,903::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-387::INFO::2012-12-02 15:15:44,903::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-387::DEBUG::2012-12-02 15:15:44,903::task::1151::TaskManager.Task::(prepare) Task=`1e85a89b-ee0f-4b00-9a8c-518ab324b171`::finished: {'allTasksStatus': {}} >Thread-387::DEBUG::2012-12-02 15:15:44,903::task::568::TaskManager.Task::(_updateState) Task=`1e85a89b-ee0f-4b00-9a8c-518ab324b171`::moving from state preparing -> state finished >Thread-387::DEBUG::2012-12-02 15:15:44,903::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-387::DEBUG::2012-12-02 15:15:44,903::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-387::DEBUG::2012-12-02 15:15:44,903::task::957::TaskManager.Task::(_decref) Task=`1e85a89b-ee0f-4b00-9a8c-518ab324b171`::ref 0 aborting False >Thread-388::DEBUG::2012-12-02 15:15:44,913::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-388::DEBUG::2012-12-02 15:15:44,913::task::568::TaskManager.Task::(_updateState) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::moving from state init -> state preparing >Thread-388::INFO::2012-12-02 15:15:44,914::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-388::ERROR::2012-12-02 15:15:44,914::task::833::TaskManager.Task::(_setError) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-388::DEBUG::2012-12-02 15:15:44,914::task::852::TaskManager.Task::(_run) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::Task._run: c02b8167-0684-4573-b4be-7cc0b28a2b27 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-388::DEBUG::2012-12-02 15:15:44,914::task::1177::TaskManager.Task::(stop) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::stopping in state preparing (force False) >Thread-388::DEBUG::2012-12-02 15:15:44,914::task::957::TaskManager.Task::(_decref) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::ref 1 aborting True >Thread-388::INFO::2012-12-02 15:15:44,914::task::1134::TaskManager.Task::(prepare) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-388::DEBUG::2012-12-02 15:15:44,915::task::1139::TaskManager.Task::(prepare) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::Prepare: aborted: Unknown pool id, pool not connected >Thread-388::DEBUG::2012-12-02 15:15:44,915::task::957::TaskManager.Task::(_decref) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::ref 0 aborting True >Thread-388::DEBUG::2012-12-02 15:15:44,915::task::892::TaskManager.Task::(_doAbort) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::Task._doAbort: force False >Thread-388::DEBUG::2012-12-02 15:15:44,915::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-388::DEBUG::2012-12-02 15:15:44,915::task::568::TaskManager.Task::(_updateState) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::moving from state preparing -> state aborting >Thread-388::DEBUG::2012-12-02 15:15:44,915::task::523::TaskManager.Task::(__state_aborting) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::_aborting: recover policy none >Thread-388::DEBUG::2012-12-02 15:15:44,915::task::568::TaskManager.Task::(_updateState) Task=`c02b8167-0684-4573-b4be-7cc0b28a2b27`::moving from state aborting -> state failed >Thread-388::DEBUG::2012-12-02 15:15:44,915::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-388::DEBUG::2012-12-02 15:15:44,915::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-388::ERROR::2012-12-02 15:15:44,916::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-389::DEBUG::2012-12-02 15:15:44,980::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-389::DEBUG::2012-12-02 15:15:44,981::task::568::TaskManager.Task::(_updateState) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::moving from state init -> state preparing >Thread-389::INFO::2012-12-02 15:15:44,981::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-389::DEBUG::2012-12-02 15:15:44,981::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b239bd55-ad85-4c74-acad-b9e2bac475e2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-389::DEBUG::2012-12-02 15:15:44,981::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-389::DEBUG::2012-12-02 15:15:44,981::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-389::DEBUG::2012-12-02 15:15:44,982::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b239bd55-ad85-4c74-acad-b9e2bac475e2`::Granted request >Thread-389::INFO::2012-12-02 15:15:44,982::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-389::DEBUG::2012-12-02 15:15:44,982::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:44,982::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:44,983::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:44,983::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:44,983::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:44,983::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:44,983::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-389::DEBUG::2012-12-02 15:15:44,983::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-389::DEBUG::2012-12-02 15:15:44,983::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-389::DEBUG::2012-12-02 15:15:44,984::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-389::DEBUG::2012-12-02 15:15:44,984::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-389::DEBUG::2012-12-02 15:15:45,010::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-389::DEBUG::2012-12-02 15:15:45,010::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-389::DEBUG::2012-12-02 15:15:45,643::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-389::DEBUG::2012-12-02 15:15:45,694::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-389::DEBUG::2012-12-02 15:15:45,695::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,695::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,695::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,695::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,696::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,696::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,696::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-389::DEBUG::2012-12-02 15:15:45,696::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,698::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-389::DEBUG::2012-12-02 15:15:45,903::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-389::WARNING::2012-12-02 15:15:45,905::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-389::DEBUG::2012-12-02 15:15:45,905::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-389::DEBUG::2012-12-02 15:15:45,934::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-389::DEBUG::2012-12-02 15:15:45,934::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-389::DEBUG::2012-12-02 15:15:45,934::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-389::DEBUG::2012-12-02 15:15:45,934::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-389::ERROR::2012-12-02 15:15:45,935::task::833::TaskManager.Task::(_setError) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-389::DEBUG::2012-12-02 15:15:45,935::task::852::TaskManager.Task::(_run) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::Task._run: 0c1e8102-ebe4-488c-b0af-b86a66b2776c ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-389::DEBUG::2012-12-02 15:15:45,935::task::1177::TaskManager.Task::(stop) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::stopping in state preparing (force False) >Thread-389::DEBUG::2012-12-02 15:15:45,935::task::957::TaskManager.Task::(_decref) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::ref 1 aborting True >Thread-389::INFO::2012-12-02 15:15:45,935::task::1134::TaskManager.Task::(prepare) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-389::DEBUG::2012-12-02 15:15:45,936::task::1139::TaskManager.Task::(prepare) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::Prepare: aborted: Cannot find master domain >Thread-389::DEBUG::2012-12-02 15:15:45,936::task::957::TaskManager.Task::(_decref) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::ref 0 aborting True >Thread-389::DEBUG::2012-12-02 15:15:45,936::task::892::TaskManager.Task::(_doAbort) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::Task._doAbort: force False >Thread-389::DEBUG::2012-12-02 15:15:45,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-389::DEBUG::2012-12-02 15:15:45,936::task::568::TaskManager.Task::(_updateState) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::moving from state preparing -> state aborting >Thread-389::DEBUG::2012-12-02 15:15:45,936::task::523::TaskManager.Task::(__state_aborting) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::_aborting: recover policy none >Thread-389::DEBUG::2012-12-02 15:15:45,936::task::568::TaskManager.Task::(_updateState) Task=`0c1e8102-ebe4-488c-b0af-b86a66b2776c`::moving from state aborting -> state failed >Thread-389::DEBUG::2012-12-02 15:15:45,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-389::DEBUG::2012-12-02 15:15:45,937::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-389::ERROR::2012-12-02 15:15:45,937::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-453::DEBUG::2012-12-02 15:15:52,063::task::568::TaskManager.Task::(_updateState) Task=`ac15b666-d180-43cd-9f2a-bb3298a1bc7b`::moving from state init -> state preparing >Thread-453::INFO::2012-12-02 15:15:52,064::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-453::INFO::2012-12-02 15:15:52,064::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-453::DEBUG::2012-12-02 15:15:52,064::task::1151::TaskManager.Task::(prepare) Task=`ac15b666-d180-43cd-9f2a-bb3298a1bc7b`::finished: {} >Thread-453::DEBUG::2012-12-02 15:15:52,064::task::568::TaskManager.Task::(_updateState) Task=`ac15b666-d180-43cd-9f2a-bb3298a1bc7b`::moving from state preparing -> state finished >Thread-453::DEBUG::2012-12-02 15:15:52,064::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-453::DEBUG::2012-12-02 15:15:52,064::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-453::DEBUG::2012-12-02 15:15:52,065::task::957::TaskManager.Task::(_decref) Task=`ac15b666-d180-43cd-9f2a-bb3298a1bc7b`::ref 0 aborting False >Thread-456::DEBUG::2012-12-02 15:15:56,166::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-456::DEBUG::2012-12-02 15:15:56,167::task::568::TaskManager.Task::(_updateState) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::moving from state init -> state preparing >Thread-456::INFO::2012-12-02 15:15:56,167::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-456::ERROR::2012-12-02 15:15:56,167::task::833::TaskManager.Task::(_setError) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-456::DEBUG::2012-12-02 15:15:56,167::task::852::TaskManager.Task::(_run) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::Task._run: 52b43c8f-c4ad-4ef5-9493-f898c9747d25 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-456::DEBUG::2012-12-02 15:15:56,167::task::1177::TaskManager.Task::(stop) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::stopping in state preparing (force False) >Thread-456::DEBUG::2012-12-02 15:15:56,167::task::957::TaskManager.Task::(_decref) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::ref 1 aborting True >Thread-456::INFO::2012-12-02 15:15:56,168::task::1134::TaskManager.Task::(prepare) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-456::DEBUG::2012-12-02 15:15:56,168::task::1139::TaskManager.Task::(prepare) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::Prepare: aborted: Unknown pool id, pool not connected >Thread-456::DEBUG::2012-12-02 15:15:56,168::task::957::TaskManager.Task::(_decref) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::ref 0 aborting True >Thread-456::DEBUG::2012-12-02 15:15:56,168::task::892::TaskManager.Task::(_doAbort) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::Task._doAbort: force False >Thread-456::DEBUG::2012-12-02 15:15:56,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-456::DEBUG::2012-12-02 15:15:56,168::task::568::TaskManager.Task::(_updateState) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::moving from state preparing -> state aborting >Thread-456::DEBUG::2012-12-02 15:15:56,168::task::523::TaskManager.Task::(__state_aborting) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::_aborting: recover policy none >Thread-456::DEBUG::2012-12-02 15:15:56,168::task::568::TaskManager.Task::(_updateState) Task=`52b43c8f-c4ad-4ef5-9493-f898c9747d25`::moving from state aborting -> state failed >Thread-456::DEBUG::2012-12-02 15:15:56,169::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-456::DEBUG::2012-12-02 15:15:56,169::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-456::ERROR::2012-12-02 15:15:56,169::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-458::DEBUG::2012-12-02 15:15:56,178::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-458::DEBUG::2012-12-02 15:15:56,178::task::568::TaskManager.Task::(_updateState) Task=`4624d850-40ee-4c43-b4b3-cc4732e61c8f`::moving from state init -> state preparing >Thread-458::INFO::2012-12-02 15:15:56,178::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-458::DEBUG::2012-12-02 15:15:56,178::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-458::DEBUG::2012-12-02 15:15:56,178::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-458::INFO::2012-12-02 15:15:56,179::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-458::DEBUG::2012-12-02 15:15:56,179::task::1151::TaskManager.Task::(prepare) Task=`4624d850-40ee-4c43-b4b3-cc4732e61c8f`::finished: {'allTasksStatus': {}} >Thread-458::DEBUG::2012-12-02 15:15:56,179::task::568::TaskManager.Task::(_updateState) Task=`4624d850-40ee-4c43-b4b3-cc4732e61c8f`::moving from state preparing -> state finished >Thread-458::DEBUG::2012-12-02 15:15:56,179::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-458::DEBUG::2012-12-02 15:15:56,179::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-458::DEBUG::2012-12-02 15:15:56,179::task::957::TaskManager.Task::(_decref) Task=`4624d850-40ee-4c43-b4b3-cc4732e61c8f`::ref 0 aborting False >Thread-459::DEBUG::2012-12-02 15:15:56,189::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-459::DEBUG::2012-12-02 15:15:56,189::task::568::TaskManager.Task::(_updateState) Task=`826663c8-c428-4426-81b6-c972b33c27af`::moving from state init -> state preparing >Thread-459::INFO::2012-12-02 15:15:56,190::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-459::ERROR::2012-12-02 15:15:56,190::task::833::TaskManager.Task::(_setError) Task=`826663c8-c428-4426-81b6-c972b33c27af`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-459::DEBUG::2012-12-02 15:15:56,190::task::852::TaskManager.Task::(_run) Task=`826663c8-c428-4426-81b6-c972b33c27af`::Task._run: 826663c8-c428-4426-81b6-c972b33c27af ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-459::DEBUG::2012-12-02 15:15:56,190::task::1177::TaskManager.Task::(stop) Task=`826663c8-c428-4426-81b6-c972b33c27af`::stopping in state preparing (force False) >Thread-459::DEBUG::2012-12-02 15:15:56,190::task::957::TaskManager.Task::(_decref) Task=`826663c8-c428-4426-81b6-c972b33c27af`::ref 1 aborting True >Thread-459::INFO::2012-12-02 15:15:56,190::task::1134::TaskManager.Task::(prepare) Task=`826663c8-c428-4426-81b6-c972b33c27af`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-459::DEBUG::2012-12-02 15:15:56,190::task::1139::TaskManager.Task::(prepare) Task=`826663c8-c428-4426-81b6-c972b33c27af`::Prepare: aborted: Unknown pool id, pool not connected >Thread-459::DEBUG::2012-12-02 15:15:56,191::task::957::TaskManager.Task::(_decref) Task=`826663c8-c428-4426-81b6-c972b33c27af`::ref 0 aborting True >Thread-459::DEBUG::2012-12-02 15:15:56,191::task::892::TaskManager.Task::(_doAbort) Task=`826663c8-c428-4426-81b6-c972b33c27af`::Task._doAbort: force False >Thread-459::DEBUG::2012-12-02 15:15:56,191::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-459::DEBUG::2012-12-02 15:15:56,191::task::568::TaskManager.Task::(_updateState) Task=`826663c8-c428-4426-81b6-c972b33c27af`::moving from state preparing -> state aborting >Thread-459::DEBUG::2012-12-02 15:15:56,191::task::523::TaskManager.Task::(__state_aborting) Task=`826663c8-c428-4426-81b6-c972b33c27af`::_aborting: recover policy none >Thread-459::DEBUG::2012-12-02 15:15:56,191::task::568::TaskManager.Task::(_updateState) Task=`826663c8-c428-4426-81b6-c972b33c27af`::moving from state aborting -> state failed >Thread-459::DEBUG::2012-12-02 15:15:56,191::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-459::DEBUG::2012-12-02 15:15:56,191::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-459::ERROR::2012-12-02 15:15:56,192::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-460::DEBUG::2012-12-02 15:15:56,254::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-460::DEBUG::2012-12-02 15:15:56,255::task::568::TaskManager.Task::(_updateState) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::moving from state init -> state preparing >Thread-460::INFO::2012-12-02 15:15:56,255::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-460::DEBUG::2012-12-02 15:15:56,255::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`2cc7fcf1-6d30-4c87-9ac3-7bc9125ebf8c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-460::DEBUG::2012-12-02 15:15:56,255::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-460::DEBUG::2012-12-02 15:15:56,256::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-460::DEBUG::2012-12-02 15:15:56,256::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`2cc7fcf1-6d30-4c87-9ac3-7bc9125ebf8c`::Granted request >Thread-460::INFO::2012-12-02 15:15:56,256::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-460::DEBUG::2012-12-02 15:15:56,256::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,257::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,257::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,257::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,257::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,257::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,257::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-460::DEBUG::2012-12-02 15:15:56,257::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-460::DEBUG::2012-12-02 15:15:56,258::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-460::DEBUG::2012-12-02 15:15:56,258::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-460::DEBUG::2012-12-02 15:15:56,258::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-460::DEBUG::2012-12-02 15:15:56,283::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-460::DEBUG::2012-12-02 15:15:56,283::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-460::DEBUG::2012-12-02 15:15:56,916::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-460::DEBUG::2012-12-02 15:15:56,987::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-460::DEBUG::2012-12-02 15:15:56,988::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,988::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,988::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,988::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,988::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,989::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,989::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-460::DEBUG::2012-12-02 15:15:56,989::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-460::DEBUG::2012-12-02 15:15:56,991::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-460::DEBUG::2012-12-02 15:15:57,196::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-460::WARNING::2012-12-02 15:15:57,198::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-460::DEBUG::2012-12-02 15:15:57,198::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-511::WARNING::2012-12-02 15:15:57,225::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-513::WARNING::2012-12-02 15:15:57,227::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-514::WARNING::2012-12-02 15:15:57,227::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-515::WARNING::2012-12-02 15:15:57,228::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-516::WARNING::2012-12-02 15:15:57,228::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-517::WARNING::2012-12-02 15:15:57,229::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-518::WARNING::2012-12-02 15:15:57,230::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-519::WARNING::2012-12-02 15:15:57,231::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-520::WARNING::2012-12-02 15:15:57,231::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-460::DEBUG::2012-12-02 15:15:57,257::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-460::DEBUG::2012-12-02 15:15:57,257::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-460::DEBUG::2012-12-02 15:15:57,257::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-460::DEBUG::2012-12-02 15:15:57,257::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-460::ERROR::2012-12-02 15:15:57,257::task::833::TaskManager.Task::(_setError) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-460::DEBUG::2012-12-02 15:15:57,257::task::852::TaskManager.Task::(_run) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::Task._run: 4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::1177::TaskManager.Task::(stop) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::stopping in state preparing (force False) >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::957::TaskManager.Task::(_decref) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::ref 1 aborting True >Thread-460::INFO::2012-12-02 15:15:57,258::task::1134::TaskManager.Task::(prepare) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::1139::TaskManager.Task::(prepare) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::Prepare: aborted: Cannot find master domain >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::957::TaskManager.Task::(_decref) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::ref 0 aborting True >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::892::TaskManager.Task::(_doAbort) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::Task._doAbort: force False >Thread-460::DEBUG::2012-12-02 15:15:57,258::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::568::TaskManager.Task::(_updateState) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::moving from state preparing -> state aborting >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::523::TaskManager.Task::(__state_aborting) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::_aborting: recover policy none >Thread-460::DEBUG::2012-12-02 15:15:57,258::task::568::TaskManager.Task::(_updateState) Task=`4a851c9f-e9c4-4efb-8d5b-26cd9db5a04c`::moving from state aborting -> state failed >Thread-460::DEBUG::2012-12-02 15:15:57,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-460::DEBUG::2012-12-02 15:15:57,259::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-460::ERROR::2012-12-02 15:15:57,259::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-523::DEBUG::2012-12-02 15:16:02,199::task::568::TaskManager.Task::(_updateState) Task=`5f25a4a4-6810-4eb6-8c89-234712578615`::moving from state init -> state preparing >Thread-523::INFO::2012-12-02 15:16:02,199::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-523::INFO::2012-12-02 15:16:02,199::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-523::DEBUG::2012-12-02 15:16:02,199::task::1151::TaskManager.Task::(prepare) Task=`5f25a4a4-6810-4eb6-8c89-234712578615`::finished: {} >Thread-523::DEBUG::2012-12-02 15:16:02,199::task::568::TaskManager.Task::(_updateState) Task=`5f25a4a4-6810-4eb6-8c89-234712578615`::moving from state preparing -> state finished >Thread-523::DEBUG::2012-12-02 15:16:02,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-523::DEBUG::2012-12-02 15:16:02,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-523::DEBUG::2012-12-02 15:16:02,199::task::957::TaskManager.Task::(_decref) Task=`5f25a4a4-6810-4eb6-8c89-234712578615`::ref 0 aborting False >Thread-527::DEBUG::2012-12-02 15:16:07,483::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-527::DEBUG::2012-12-02 15:16:07,484::task::568::TaskManager.Task::(_updateState) Task=`7406463a-9f70-4947-8816-a99d7750a770`::moving from state init -> state preparing >Thread-527::INFO::2012-12-02 15:16:07,484::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-527::ERROR::2012-12-02 15:16:07,484::task::833::TaskManager.Task::(_setError) Task=`7406463a-9f70-4947-8816-a99d7750a770`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-527::DEBUG::2012-12-02 15:16:07,484::task::852::TaskManager.Task::(_run) Task=`7406463a-9f70-4947-8816-a99d7750a770`::Task._run: 7406463a-9f70-4947-8816-a99d7750a770 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-527::DEBUG::2012-12-02 15:16:07,484::task::1177::TaskManager.Task::(stop) Task=`7406463a-9f70-4947-8816-a99d7750a770`::stopping in state preparing (force False) >Thread-527::DEBUG::2012-12-02 15:16:07,484::task::957::TaskManager.Task::(_decref) Task=`7406463a-9f70-4947-8816-a99d7750a770`::ref 1 aborting True >Thread-527::INFO::2012-12-02 15:16:07,485::task::1134::TaskManager.Task::(prepare) Task=`7406463a-9f70-4947-8816-a99d7750a770`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-527::DEBUG::2012-12-02 15:16:07,485::task::1139::TaskManager.Task::(prepare) Task=`7406463a-9f70-4947-8816-a99d7750a770`::Prepare: aborted: Unknown pool id, pool not connected >Thread-527::DEBUG::2012-12-02 15:16:07,485::task::957::TaskManager.Task::(_decref) Task=`7406463a-9f70-4947-8816-a99d7750a770`::ref 0 aborting True >Thread-527::DEBUG::2012-12-02 15:16:07,485::task::892::TaskManager.Task::(_doAbort) Task=`7406463a-9f70-4947-8816-a99d7750a770`::Task._doAbort: force False >Thread-527::DEBUG::2012-12-02 15:16:07,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-527::DEBUG::2012-12-02 15:16:07,485::task::568::TaskManager.Task::(_updateState) Task=`7406463a-9f70-4947-8816-a99d7750a770`::moving from state preparing -> state aborting >Thread-527::DEBUG::2012-12-02 15:16:07,485::task::523::TaskManager.Task::(__state_aborting) Task=`7406463a-9f70-4947-8816-a99d7750a770`::_aborting: recover policy none >Thread-527::DEBUG::2012-12-02 15:16:07,485::task::568::TaskManager.Task::(_updateState) Task=`7406463a-9f70-4947-8816-a99d7750a770`::moving from state aborting -> state failed >Thread-527::DEBUG::2012-12-02 15:16:07,486::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-527::DEBUG::2012-12-02 15:16:07,486::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-527::ERROR::2012-12-02 15:16:07,486::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-528::DEBUG::2012-12-02 15:16:07,495::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-528::DEBUG::2012-12-02 15:16:07,495::task::568::TaskManager.Task::(_updateState) Task=`26063b39-d222-4f83-90c0-2c95071e12a0`::moving from state init -> state preparing >Thread-528::INFO::2012-12-02 15:16:07,495::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-528::DEBUG::2012-12-02 15:16:07,495::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-528::DEBUG::2012-12-02 15:16:07,495::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-528::INFO::2012-12-02 15:16:07,495::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-528::DEBUG::2012-12-02 15:16:07,496::task::1151::TaskManager.Task::(prepare) Task=`26063b39-d222-4f83-90c0-2c95071e12a0`::finished: {'allTasksStatus': {}} >Thread-528::DEBUG::2012-12-02 15:16:07,496::task::568::TaskManager.Task::(_updateState) Task=`26063b39-d222-4f83-90c0-2c95071e12a0`::moving from state preparing -> state finished >Thread-528::DEBUG::2012-12-02 15:16:07,496::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-528::DEBUG::2012-12-02 15:16:07,496::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-528::DEBUG::2012-12-02 15:16:07,496::task::957::TaskManager.Task::(_decref) Task=`26063b39-d222-4f83-90c0-2c95071e12a0`::ref 0 aborting False >Thread-529::DEBUG::2012-12-02 15:16:07,506::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-529::DEBUG::2012-12-02 15:16:07,507::task::568::TaskManager.Task::(_updateState) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::moving from state init -> state preparing >Thread-529::INFO::2012-12-02 15:16:07,507::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-529::ERROR::2012-12-02 15:16:07,507::task::833::TaskManager.Task::(_setError) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-529::DEBUG::2012-12-02 15:16:07,507::task::852::TaskManager.Task::(_run) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::Task._run: 8a293973-8bc8-4d35-b46c-8058fa7c2ad1 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-529::DEBUG::2012-12-02 15:16:07,507::task::1177::TaskManager.Task::(stop) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::stopping in state preparing (force False) >Thread-529::DEBUG::2012-12-02 15:16:07,507::task::957::TaskManager.Task::(_decref) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::ref 1 aborting True >Thread-529::INFO::2012-12-02 15:16:07,508::task::1134::TaskManager.Task::(prepare) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-529::DEBUG::2012-12-02 15:16:07,508::task::1139::TaskManager.Task::(prepare) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::Prepare: aborted: Unknown pool id, pool not connected >Thread-529::DEBUG::2012-12-02 15:16:07,508::task::957::TaskManager.Task::(_decref) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::ref 0 aborting True >Thread-529::DEBUG::2012-12-02 15:16:07,508::task::892::TaskManager.Task::(_doAbort) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::Task._doAbort: force False >Thread-529::DEBUG::2012-12-02 15:16:07,508::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-529::DEBUG::2012-12-02 15:16:07,508::task::568::TaskManager.Task::(_updateState) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::moving from state preparing -> state aborting >Thread-529::DEBUG::2012-12-02 15:16:07,508::task::523::TaskManager.Task::(__state_aborting) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::_aborting: recover policy none >Thread-529::DEBUG::2012-12-02 15:16:07,508::task::568::TaskManager.Task::(_updateState) Task=`8a293973-8bc8-4d35-b46c-8058fa7c2ad1`::moving from state aborting -> state failed >Thread-529::DEBUG::2012-12-02 15:16:07,508::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-529::DEBUG::2012-12-02 15:16:07,509::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-529::ERROR::2012-12-02 15:16:07,509::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-530::DEBUG::2012-12-02 15:16:07,580::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-530::DEBUG::2012-12-02 15:16:07,581::task::568::TaskManager.Task::(_updateState) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::moving from state init -> state preparing >Thread-530::INFO::2012-12-02 15:16:07,581::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-530::DEBUG::2012-12-02 15:16:07,581::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b93ca234-a459-4b2b-b528-f1f64185018d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-530::DEBUG::2012-12-02 15:16:07,581::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-530::DEBUG::2012-12-02 15:16:07,581::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-530::DEBUG::2012-12-02 15:16:07,582::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`b93ca234-a459-4b2b-b528-f1f64185018d`::Granted request >Thread-530::INFO::2012-12-02 15:16:07,582::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-530::DEBUG::2012-12-02 15:16:07,582::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:07,582::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:07,583::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:07,583::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:07,583::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:07,583::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:07,583::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-530::DEBUG::2012-12-02 15:16:07,583::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-530::DEBUG::2012-12-02 15:16:07,583::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-530::DEBUG::2012-12-02 15:16:07,584::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-530::DEBUG::2012-12-02 15:16:07,584::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-530::DEBUG::2012-12-02 15:16:07,608::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-530::DEBUG::2012-12-02 15:16:07,608::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-530::DEBUG::2012-12-02 15:16:08,255::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-530::DEBUG::2012-12-02 15:16:08,322::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-530::DEBUG::2012-12-02 15:16:08,322::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,322::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,322::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,323::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,323::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,323::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,323::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-530::DEBUG::2012-12-02 15:16:08,323::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,325::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-530::DEBUG::2012-12-02 15:16:08,530::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-530::WARNING::2012-12-02 15:16:08,532::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-530::DEBUG::2012-12-02 15:16:08,532::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-530::DEBUG::2012-12-02 15:16:08,563::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-530::DEBUG::2012-12-02 15:16:08,563::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-530::DEBUG::2012-12-02 15:16:08,563::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-530::DEBUG::2012-12-02 15:16:08,564::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-530::ERROR::2012-12-02 15:16:08,564::task::833::TaskManager.Task::(_setError) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-530::DEBUG::2012-12-02 15:16:08,564::task::852::TaskManager.Task::(_run) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::Task._run: 8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-530::DEBUG::2012-12-02 15:16:08,564::task::1177::TaskManager.Task::(stop) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::stopping in state preparing (force False) >Thread-530::DEBUG::2012-12-02 15:16:08,564::task::957::TaskManager.Task::(_decref) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::ref 1 aborting True >Thread-530::INFO::2012-12-02 15:16:08,565::task::1134::TaskManager.Task::(prepare) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-530::DEBUG::2012-12-02 15:16:08,565::task::1139::TaskManager.Task::(prepare) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::Prepare: aborted: Cannot find master domain >Thread-530::DEBUG::2012-12-02 15:16:08,565::task::957::TaskManager.Task::(_decref) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::ref 0 aborting True >Thread-530::DEBUG::2012-12-02 15:16:08,565::task::892::TaskManager.Task::(_doAbort) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::Task._doAbort: force False >Thread-530::DEBUG::2012-12-02 15:16:08,565::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-530::DEBUG::2012-12-02 15:16:08,565::task::568::TaskManager.Task::(_updateState) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::moving from state preparing -> state aborting >Thread-530::DEBUG::2012-12-02 15:16:08,565::task::523::TaskManager.Task::(__state_aborting) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::_aborting: recover policy none >Thread-530::DEBUG::2012-12-02 15:16:08,566::task::568::TaskManager.Task::(_updateState) Task=`8cb7a98c-dc02-4b33-a012-8d5eb6ff22a3`::moving from state aborting -> state failed >Thread-530::DEBUG::2012-12-02 15:16:08,566::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-530::DEBUG::2012-12-02 15:16:08,566::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-530::ERROR::2012-12-02 15:16:08,566::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-593::DEBUG::2012-12-02 15:16:12,319::task::568::TaskManager.Task::(_updateState) Task=`fe935652-7e35-451a-b054-3a2bf10d573b`::moving from state init -> state preparing >Thread-593::INFO::2012-12-02 15:16:12,319::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-593::INFO::2012-12-02 15:16:12,319::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-593::DEBUG::2012-12-02 15:16:12,320::task::1151::TaskManager.Task::(prepare) Task=`fe935652-7e35-451a-b054-3a2bf10d573b`::finished: {} >Thread-593::DEBUG::2012-12-02 15:16:12,320::task::568::TaskManager.Task::(_updateState) Task=`fe935652-7e35-451a-b054-3a2bf10d573b`::moving from state preparing -> state finished >Thread-593::DEBUG::2012-12-02 15:16:12,320::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-593::DEBUG::2012-12-02 15:16:12,320::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-593::DEBUG::2012-12-02 15:16:12,320::task::957::TaskManager.Task::(_decref) Task=`fe935652-7e35-451a-b054-3a2bf10d573b`::ref 0 aborting False >Thread-598::DEBUG::2012-12-02 15:16:18,810::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-598::DEBUG::2012-12-02 15:16:18,810::task::568::TaskManager.Task::(_updateState) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::moving from state init -> state preparing >Thread-598::INFO::2012-12-02 15:16:18,810::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-598::ERROR::2012-12-02 15:16:18,811::task::833::TaskManager.Task::(_setError) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-598::DEBUG::2012-12-02 15:16:18,811::task::852::TaskManager.Task::(_run) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::Task._run: 2684e171-d304-44ce-a2ac-ad0496c06944 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-598::DEBUG::2012-12-02 15:16:18,811::task::1177::TaskManager.Task::(stop) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::stopping in state preparing (force False) >Thread-598::DEBUG::2012-12-02 15:16:18,811::task::957::TaskManager.Task::(_decref) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::ref 1 aborting True >Thread-598::INFO::2012-12-02 15:16:18,811::task::1134::TaskManager.Task::(prepare) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-598::DEBUG::2012-12-02 15:16:18,811::task::1139::TaskManager.Task::(prepare) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::Prepare: aborted: Unknown pool id, pool not connected >Thread-598::DEBUG::2012-12-02 15:16:18,812::task::957::TaskManager.Task::(_decref) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::ref 0 aborting True >Thread-598::DEBUG::2012-12-02 15:16:18,812::task::892::TaskManager.Task::(_doAbort) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::Task._doAbort: force False >Thread-598::DEBUG::2012-12-02 15:16:18,812::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-598::DEBUG::2012-12-02 15:16:18,812::task::568::TaskManager.Task::(_updateState) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::moving from state preparing -> state aborting >Thread-598::DEBUG::2012-12-02 15:16:18,812::task::523::TaskManager.Task::(__state_aborting) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::_aborting: recover policy none >Thread-598::DEBUG::2012-12-02 15:16:18,812::task::568::TaskManager.Task::(_updateState) Task=`2684e171-d304-44ce-a2ac-ad0496c06944`::moving from state aborting -> state failed >Thread-598::DEBUG::2012-12-02 15:16:18,812::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-598::DEBUG::2012-12-02 15:16:18,812::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-598::ERROR::2012-12-02 15:16:18,813::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-599::DEBUG::2012-12-02 15:16:18,821::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-599::DEBUG::2012-12-02 15:16:18,821::task::568::TaskManager.Task::(_updateState) Task=`edd6ca45-32a6-412d-92e9-402ec39bd3e6`::moving from state init -> state preparing >Thread-599::INFO::2012-12-02 15:16:18,821::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-599::DEBUG::2012-12-02 15:16:18,821::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-599::DEBUG::2012-12-02 15:16:18,821::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-599::INFO::2012-12-02 15:16:18,822::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-599::DEBUG::2012-12-02 15:16:18,822::task::1151::TaskManager.Task::(prepare) Task=`edd6ca45-32a6-412d-92e9-402ec39bd3e6`::finished: {'allTasksStatus': {}} >Thread-599::DEBUG::2012-12-02 15:16:18,822::task::568::TaskManager.Task::(_updateState) Task=`edd6ca45-32a6-412d-92e9-402ec39bd3e6`::moving from state preparing -> state finished >Thread-599::DEBUG::2012-12-02 15:16:18,822::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-599::DEBUG::2012-12-02 15:16:18,822::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-599::DEBUG::2012-12-02 15:16:18,822::task::957::TaskManager.Task::(_decref) Task=`edd6ca45-32a6-412d-92e9-402ec39bd3e6`::ref 0 aborting False >Thread-600::DEBUG::2012-12-02 15:16:18,832::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-600::DEBUG::2012-12-02 15:16:18,833::task::568::TaskManager.Task::(_updateState) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::moving from state init -> state preparing >Thread-600::INFO::2012-12-02 15:16:18,833::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-600::ERROR::2012-12-02 15:16:18,833::task::833::TaskManager.Task::(_setError) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-600::DEBUG::2012-12-02 15:16:18,833::task::852::TaskManager.Task::(_run) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::Task._run: 065446b2-36ff-4ea7-bc65-d050f9617dcf ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-600::DEBUG::2012-12-02 15:16:18,833::task::1177::TaskManager.Task::(stop) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::stopping in state preparing (force False) >Thread-600::DEBUG::2012-12-02 15:16:18,833::task::957::TaskManager.Task::(_decref) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::ref 1 aborting True >Thread-600::INFO::2012-12-02 15:16:18,834::task::1134::TaskManager.Task::(prepare) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-600::DEBUG::2012-12-02 15:16:18,834::task::1139::TaskManager.Task::(prepare) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::Prepare: aborted: Unknown pool id, pool not connected >Thread-600::DEBUG::2012-12-02 15:16:18,834::task::957::TaskManager.Task::(_decref) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::ref 0 aborting True >Thread-600::DEBUG::2012-12-02 15:16:18,834::task::892::TaskManager.Task::(_doAbort) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::Task._doAbort: force False >Thread-600::DEBUG::2012-12-02 15:16:18,834::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-600::DEBUG::2012-12-02 15:16:18,834::task::568::TaskManager.Task::(_updateState) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::moving from state preparing -> state aborting >Thread-600::DEBUG::2012-12-02 15:16:18,834::task::523::TaskManager.Task::(__state_aborting) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::_aborting: recover policy none >Thread-600::DEBUG::2012-12-02 15:16:18,834::task::568::TaskManager.Task::(_updateState) Task=`065446b2-36ff-4ea7-bc65-d050f9617dcf`::moving from state aborting -> state failed >Thread-600::DEBUG::2012-12-02 15:16:18,835::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-600::DEBUG::2012-12-02 15:16:18,835::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-600::ERROR::2012-12-02 15:16:18,835::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-601::DEBUG::2012-12-02 15:16:18,903::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-601::DEBUG::2012-12-02 15:16:18,903::task::568::TaskManager.Task::(_updateState) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::moving from state init -> state preparing >Thread-601::INFO::2012-12-02 15:16:18,903::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-601::DEBUG::2012-12-02 15:16:18,904::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`7f6eb50c-175c-440d-aa07-831fd0b5a475`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-601::DEBUG::2012-12-02 15:16:18,904::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-601::DEBUG::2012-12-02 15:16:18,904::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-601::DEBUG::2012-12-02 15:16:18,904::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`7f6eb50c-175c-440d-aa07-831fd0b5a475`::Granted request >Thread-601::INFO::2012-12-02 15:16:18,904::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-601::DEBUG::2012-12-02 15:16:18,905::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:18,905::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:18,905::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:18,905::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:18,905::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:18,905::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:18,905::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-601::DEBUG::2012-12-02 15:16:18,906::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-601::DEBUG::2012-12-02 15:16:18,906::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-601::DEBUG::2012-12-02 15:16:18,906::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-601::DEBUG::2012-12-02 15:16:18,906::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-601::DEBUG::2012-12-02 15:16:18,932::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-601::DEBUG::2012-12-02 15:16:18,932::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-601::DEBUG::2012-12-02 15:16:19,566::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-601::DEBUG::2012-12-02 15:16:19,628::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-601::DEBUG::2012-12-02 15:16:19,629::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,629::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,629::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,629::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,630::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,630::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,630::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-601::DEBUG::2012-12-02 15:16:19,630::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,632::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-601::DEBUG::2012-12-02 15:16:19,838::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-601::WARNING::2012-12-02 15:16:19,839::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-601::DEBUG::2012-12-02 15:16:19,840::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-601::DEBUG::2012-12-02 15:16:19,869::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-601::DEBUG::2012-12-02 15:16:19,869::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-601::DEBUG::2012-12-02 15:16:19,869::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-601::DEBUG::2012-12-02 15:16:19,869::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-601::ERROR::2012-12-02 15:16:19,869::task::833::TaskManager.Task::(_setError) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-601::DEBUG::2012-12-02 15:16:19,870::task::852::TaskManager.Task::(_run) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::Task._run: 6fa7c2a3-dbe4-4767-91cd-8eee22cc275c ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-601::DEBUG::2012-12-02 15:16:19,870::task::1177::TaskManager.Task::(stop) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::stopping in state preparing (force False) >Thread-601::DEBUG::2012-12-02 15:16:19,870::task::957::TaskManager.Task::(_decref) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::ref 1 aborting True >Thread-601::INFO::2012-12-02 15:16:19,870::task::1134::TaskManager.Task::(prepare) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-601::DEBUG::2012-12-02 15:16:19,870::task::1139::TaskManager.Task::(prepare) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::Prepare: aborted: Cannot find master domain >Thread-601::DEBUG::2012-12-02 15:16:19,871::task::957::TaskManager.Task::(_decref) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::ref 0 aborting True >Thread-601::DEBUG::2012-12-02 15:16:19,871::task::892::TaskManager.Task::(_doAbort) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::Task._doAbort: force False >Thread-601::DEBUG::2012-12-02 15:16:19,871::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-601::DEBUG::2012-12-02 15:16:19,871::task::568::TaskManager.Task::(_updateState) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::moving from state preparing -> state aborting >Thread-601::DEBUG::2012-12-02 15:16:19,871::task::523::TaskManager.Task::(__state_aborting) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::_aborting: recover policy none >Thread-601::DEBUG::2012-12-02 15:16:19,871::task::568::TaskManager.Task::(_updateState) Task=`6fa7c2a3-dbe4-4767-91cd-8eee22cc275c`::moving from state aborting -> state failed >Thread-601::DEBUG::2012-12-02 15:16:19,871::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-601::DEBUG::2012-12-02 15:16:19,871::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-601::ERROR::2012-12-02 15:16:19,872::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-663::DEBUG::2012-12-02 15:16:22,426::task::568::TaskManager.Task::(_updateState) Task=`c77fedde-710e-4fd5-8bdd-88c3fcdec08b`::moving from state init -> state preparing >Thread-663::INFO::2012-12-02 15:16:22,427::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-663::INFO::2012-12-02 15:16:22,427::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-663::DEBUG::2012-12-02 15:16:22,427::task::1151::TaskManager.Task::(prepare) Task=`c77fedde-710e-4fd5-8bdd-88c3fcdec08b`::finished: {} >Thread-663::DEBUG::2012-12-02 15:16:22,427::task::568::TaskManager.Task::(_updateState) Task=`c77fedde-710e-4fd5-8bdd-88c3fcdec08b`::moving from state preparing -> state finished >Thread-663::DEBUG::2012-12-02 15:16:22,427::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-663::DEBUG::2012-12-02 15:16:22,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-663::DEBUG::2012-12-02 15:16:22,427::task::957::TaskManager.Task::(_decref) Task=`c77fedde-710e-4fd5-8bdd-88c3fcdec08b`::ref 0 aborting False >Thread-668::DEBUG::2012-12-02 15:16:30,119::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-668::DEBUG::2012-12-02 15:16:30,119::task::568::TaskManager.Task::(_updateState) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::moving from state init -> state preparing >Thread-668::INFO::2012-12-02 15:16:30,119::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-668::ERROR::2012-12-02 15:16:30,119::task::833::TaskManager.Task::(_setError) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-668::DEBUG::2012-12-02 15:16:30,119::task::852::TaskManager.Task::(_run) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::Task._run: 264352d8-9b5c-4928-b245-1b0ec6d547a5 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-668::DEBUG::2012-12-02 15:16:30,120::task::1177::TaskManager.Task::(stop) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::stopping in state preparing (force False) >Thread-668::DEBUG::2012-12-02 15:16:30,120::task::957::TaskManager.Task::(_decref) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::ref 1 aborting True >Thread-668::INFO::2012-12-02 15:16:30,120::task::1134::TaskManager.Task::(prepare) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-668::DEBUG::2012-12-02 15:16:30,120::task::1139::TaskManager.Task::(prepare) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::Prepare: aborted: Unknown pool id, pool not connected >Thread-668::DEBUG::2012-12-02 15:16:30,120::task::957::TaskManager.Task::(_decref) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::ref 0 aborting True >Thread-668::DEBUG::2012-12-02 15:16:30,120::task::892::TaskManager.Task::(_doAbort) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::Task._doAbort: force False >Thread-668::DEBUG::2012-12-02 15:16:30,120::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-668::DEBUG::2012-12-02 15:16:30,121::task::568::TaskManager.Task::(_updateState) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::moving from state preparing -> state aborting >Thread-668::DEBUG::2012-12-02 15:16:30,121::task::523::TaskManager.Task::(__state_aborting) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::_aborting: recover policy none >Thread-668::DEBUG::2012-12-02 15:16:30,121::task::568::TaskManager.Task::(_updateState) Task=`264352d8-9b5c-4928-b245-1b0ec6d547a5`::moving from state aborting -> state failed >Thread-668::DEBUG::2012-12-02 15:16:30,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-668::DEBUG::2012-12-02 15:16:30,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-668::ERROR::2012-12-02 15:16:30,121::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-669::DEBUG::2012-12-02 15:16:30,126::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-669::DEBUG::2012-12-02 15:16:30,126::task::568::TaskManager.Task::(_updateState) Task=`7b38ab8c-ae16-4738-9a18-2737add2d1b0`::moving from state init -> state preparing >Thread-669::INFO::2012-12-02 15:16:30,126::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-669::DEBUG::2012-12-02 15:16:30,126::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-669::DEBUG::2012-12-02 15:16:30,126::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-669::INFO::2012-12-02 15:16:30,127::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-669::DEBUG::2012-12-02 15:16:30,127::task::1151::TaskManager.Task::(prepare) Task=`7b38ab8c-ae16-4738-9a18-2737add2d1b0`::finished: {'allTasksStatus': {}} >Thread-669::DEBUG::2012-12-02 15:16:30,127::task::568::TaskManager.Task::(_updateState) Task=`7b38ab8c-ae16-4738-9a18-2737add2d1b0`::moving from state preparing -> state finished >Thread-669::DEBUG::2012-12-02 15:16:30,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-669::DEBUG::2012-12-02 15:16:30,127::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-669::DEBUG::2012-12-02 15:16:30,127::task::957::TaskManager.Task::(_decref) Task=`7b38ab8c-ae16-4738-9a18-2737add2d1b0`::ref 0 aborting False >Thread-670::DEBUG::2012-12-02 15:16:30,134::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-670::DEBUG::2012-12-02 15:16:30,134::task::568::TaskManager.Task::(_updateState) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::moving from state init -> state preparing >Thread-670::INFO::2012-12-02 15:16:30,134::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-670::ERROR::2012-12-02 15:16:30,134::task::833::TaskManager.Task::(_setError) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-670::DEBUG::2012-12-02 15:16:30,135::task::852::TaskManager.Task::(_run) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::Task._run: 6c5cf76a-e637-4ffd-8e4c-25891b1e1a11 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-670::DEBUG::2012-12-02 15:16:30,135::task::1177::TaskManager.Task::(stop) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::stopping in state preparing (force False) >Thread-670::DEBUG::2012-12-02 15:16:30,135::task::957::TaskManager.Task::(_decref) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::ref 1 aborting True >Thread-670::INFO::2012-12-02 15:16:30,135::task::1134::TaskManager.Task::(prepare) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-670::DEBUG::2012-12-02 15:16:30,135::task::1139::TaskManager.Task::(prepare) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::Prepare: aborted: Unknown pool id, pool not connected >Thread-670::DEBUG::2012-12-02 15:16:30,135::task::957::TaskManager.Task::(_decref) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::ref 0 aborting True >Thread-670::DEBUG::2012-12-02 15:16:30,135::task::892::TaskManager.Task::(_doAbort) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::Task._doAbort: force False >Thread-670::DEBUG::2012-12-02 15:16:30,136::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-670::DEBUG::2012-12-02 15:16:30,136::task::568::TaskManager.Task::(_updateState) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::moving from state preparing -> state aborting >Thread-670::DEBUG::2012-12-02 15:16:30,136::task::523::TaskManager.Task::(__state_aborting) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::_aborting: recover policy none >Thread-670::DEBUG::2012-12-02 15:16:30,136::task::568::TaskManager.Task::(_updateState) Task=`6c5cf76a-e637-4ffd-8e4c-25891b1e1a11`::moving from state aborting -> state failed >Thread-670::DEBUG::2012-12-02 15:16:30,136::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-670::DEBUG::2012-12-02 15:16:30,136::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-670::ERROR::2012-12-02 15:16:30,136::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-671::DEBUG::2012-12-02 15:16:30,178::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-671::DEBUG::2012-12-02 15:16:30,179::task::568::TaskManager.Task::(_updateState) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::moving from state init -> state preparing >Thread-671::INFO::2012-12-02 15:16:30,179::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-671::DEBUG::2012-12-02 15:16:30,179::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`157e8361-6cb4-42e1-8133-e4e416e8adcb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-671::DEBUG::2012-12-02 15:16:30,179::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-671::DEBUG::2012-12-02 15:16:30,179::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-671::DEBUG::2012-12-02 15:16:30,180::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`157e8361-6cb4-42e1-8133-e4e416e8adcb`::Granted request >Thread-671::INFO::2012-12-02 15:16:30,180::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-671::DEBUG::2012-12-02 15:16:30,180::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,180::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,181::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,181::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,181::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,181::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,181::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-671::DEBUG::2012-12-02 15:16:30,181::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-671::DEBUG::2012-12-02 15:16:30,181::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-671::DEBUG::2012-12-02 15:16:30,182::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-671::DEBUG::2012-12-02 15:16:30,182::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-671::DEBUG::2012-12-02 15:16:30,205::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-671::DEBUG::2012-12-02 15:16:30,205::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-671::DEBUG::2012-12-02 15:16:30,837::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-671::DEBUG::2012-12-02 15:16:30,946::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-671::DEBUG::2012-12-02 15:16:30,946::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,946::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,946::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,947::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,947::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,947::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,947::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-671::DEBUG::2012-12-02 15:16:30,947::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:30,949::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-671::DEBUG::2012-12-02 15:16:31,155::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-671::WARNING::2012-12-02 15:16:31,157::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-671::DEBUG::2012-12-02 15:16:31,157::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-671::DEBUG::2012-12-02 15:16:31,186::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-671::DEBUG::2012-12-02 15:16:31,186::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-671::DEBUG::2012-12-02 15:16:31,186::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-671::DEBUG::2012-12-02 15:16:31,186::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-671::ERROR::2012-12-02 15:16:31,186::task::833::TaskManager.Task::(_setError) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-671::DEBUG::2012-12-02 15:16:31,187::task::852::TaskManager.Task::(_run) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::Task._run: 73678fc7-f5a0-4439-b79f-5183fa09e81a ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-671::DEBUG::2012-12-02 15:16:31,187::task::1177::TaskManager.Task::(stop) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::stopping in state preparing (force False) >Thread-671::DEBUG::2012-12-02 15:16:31,187::task::957::TaskManager.Task::(_decref) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::ref 1 aborting True >Thread-671::INFO::2012-12-02 15:16:31,187::task::1134::TaskManager.Task::(prepare) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-671::DEBUG::2012-12-02 15:16:31,187::task::1139::TaskManager.Task::(prepare) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::Prepare: aborted: Cannot find master domain >Thread-671::DEBUG::2012-12-02 15:16:31,188::task::957::TaskManager.Task::(_decref) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::ref 0 aborting True >Thread-671::DEBUG::2012-12-02 15:16:31,188::task::892::TaskManager.Task::(_doAbort) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::Task._doAbort: force False >Thread-671::DEBUG::2012-12-02 15:16:31,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-671::DEBUG::2012-12-02 15:16:31,188::task::568::TaskManager.Task::(_updateState) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::moving from state preparing -> state aborting >Thread-671::DEBUG::2012-12-02 15:16:31,188::task::523::TaskManager.Task::(__state_aborting) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::_aborting: recover policy none >Thread-671::DEBUG::2012-12-02 15:16:31,188::task::568::TaskManager.Task::(_updateState) Task=`73678fc7-f5a0-4439-b79f-5183fa09e81a`::moving from state aborting -> state failed >Thread-671::DEBUG::2012-12-02 15:16:31,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-671::DEBUG::2012-12-02 15:16:31,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-671::ERROR::2012-12-02 15:16:31,189::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-733::DEBUG::2012-12-02 15:16:32,542::task::568::TaskManager.Task::(_updateState) Task=`031b2eb7-ab6c-41aa-9bd8-998c42da7b57`::moving from state init -> state preparing >Thread-733::INFO::2012-12-02 15:16:32,542::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-733::INFO::2012-12-02 15:16:32,542::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-733::DEBUG::2012-12-02 15:16:32,542::task::1151::TaskManager.Task::(prepare) Task=`031b2eb7-ab6c-41aa-9bd8-998c42da7b57`::finished: {} >Thread-733::DEBUG::2012-12-02 15:16:32,542::task::568::TaskManager.Task::(_updateState) Task=`031b2eb7-ab6c-41aa-9bd8-998c42da7b57`::moving from state preparing -> state finished >Thread-733::DEBUG::2012-12-02 15:16:32,543::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-733::DEBUG::2012-12-02 15:16:32,543::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-733::DEBUG::2012-12-02 15:16:32,543::task::957::TaskManager.Task::(_decref) Task=`031b2eb7-ab6c-41aa-9bd8-998c42da7b57`::ref 0 aborting False >Thread-739::DEBUG::2012-12-02 15:16:41,428::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-739::DEBUG::2012-12-02 15:16:41,428::task::568::TaskManager.Task::(_updateState) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::moving from state init -> state preparing >Thread-739::INFO::2012-12-02 15:16:41,428::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-739::ERROR::2012-12-02 15:16:41,428::task::833::TaskManager.Task::(_setError) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-739::DEBUG::2012-12-02 15:16:41,429::task::852::TaskManager.Task::(_run) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::Task._run: e54476ac-23b6-4219-9e81-973452c14ccf ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-739::DEBUG::2012-12-02 15:16:41,429::task::1177::TaskManager.Task::(stop) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::stopping in state preparing (force False) >Thread-739::DEBUG::2012-12-02 15:16:41,429::task::957::TaskManager.Task::(_decref) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::ref 1 aborting True >Thread-739::INFO::2012-12-02 15:16:41,429::task::1134::TaskManager.Task::(prepare) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-739::DEBUG::2012-12-02 15:16:41,429::task::1139::TaskManager.Task::(prepare) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::Prepare: aborted: Unknown pool id, pool not connected >Thread-739::DEBUG::2012-12-02 15:16:41,429::task::957::TaskManager.Task::(_decref) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::ref 0 aborting True >Thread-739::DEBUG::2012-12-02 15:16:41,430::task::892::TaskManager.Task::(_doAbort) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::Task._doAbort: force False >Thread-739::DEBUG::2012-12-02 15:16:41,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-739::DEBUG::2012-12-02 15:16:41,430::task::568::TaskManager.Task::(_updateState) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::moving from state preparing -> state aborting >Thread-739::DEBUG::2012-12-02 15:16:41,430::task::523::TaskManager.Task::(__state_aborting) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::_aborting: recover policy none >Thread-739::DEBUG::2012-12-02 15:16:41,430::task::568::TaskManager.Task::(_updateState) Task=`e54476ac-23b6-4219-9e81-973452c14ccf`::moving from state aborting -> state failed >Thread-739::DEBUG::2012-12-02 15:16:41,430::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-739::DEBUG::2012-12-02 15:16:41,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-739::ERROR::2012-12-02 15:16:41,430::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-740::DEBUG::2012-12-02 15:16:41,439::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-740::DEBUG::2012-12-02 15:16:41,439::task::568::TaskManager.Task::(_updateState) Task=`33367101-a429-4c7d-9a0a-378beb399fbe`::moving from state init -> state preparing >Thread-740::INFO::2012-12-02 15:16:41,439::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-740::DEBUG::2012-12-02 15:16:41,440::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-740::DEBUG::2012-12-02 15:16:41,440::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-740::INFO::2012-12-02 15:16:41,440::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-740::DEBUG::2012-12-02 15:16:41,440::task::1151::TaskManager.Task::(prepare) Task=`33367101-a429-4c7d-9a0a-378beb399fbe`::finished: {'allTasksStatus': {}} >Thread-740::DEBUG::2012-12-02 15:16:41,440::task::568::TaskManager.Task::(_updateState) Task=`33367101-a429-4c7d-9a0a-378beb399fbe`::moving from state preparing -> state finished >Thread-740::DEBUG::2012-12-02 15:16:41,440::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-740::DEBUG::2012-12-02 15:16:41,440::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-740::DEBUG::2012-12-02 15:16:41,440::task::957::TaskManager.Task::(_decref) Task=`33367101-a429-4c7d-9a0a-378beb399fbe`::ref 0 aborting False >Thread-741::DEBUG::2012-12-02 15:16:41,450::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-741::DEBUG::2012-12-02 15:16:41,450::task::568::TaskManager.Task::(_updateState) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::moving from state init -> state preparing >Thread-741::INFO::2012-12-02 15:16:41,451::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-741::ERROR::2012-12-02 15:16:41,451::task::833::TaskManager.Task::(_setError) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-741::DEBUG::2012-12-02 15:16:41,451::task::852::TaskManager.Task::(_run) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::Task._run: 2bbfe882-f62c-4827-b871-be3bc3118c7b ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-741::DEBUG::2012-12-02 15:16:41,451::task::1177::TaskManager.Task::(stop) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::stopping in state preparing (force False) >Thread-741::DEBUG::2012-12-02 15:16:41,451::task::957::TaskManager.Task::(_decref) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::ref 1 aborting True >Thread-741::INFO::2012-12-02 15:16:41,451::task::1134::TaskManager.Task::(prepare) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-741::DEBUG::2012-12-02 15:16:41,452::task::1139::TaskManager.Task::(prepare) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::Prepare: aborted: Unknown pool id, pool not connected >Thread-741::DEBUG::2012-12-02 15:16:41,452::task::957::TaskManager.Task::(_decref) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::ref 0 aborting True >Thread-741::DEBUG::2012-12-02 15:16:41,452::task::892::TaskManager.Task::(_doAbort) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::Task._doAbort: force False >Thread-741::DEBUG::2012-12-02 15:16:41,452::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-741::DEBUG::2012-12-02 15:16:41,452::task::568::TaskManager.Task::(_updateState) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::moving from state preparing -> state aborting >Thread-741::DEBUG::2012-12-02 15:16:41,452::task::523::TaskManager.Task::(__state_aborting) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::_aborting: recover policy none >Thread-741::DEBUG::2012-12-02 15:16:41,452::task::568::TaskManager.Task::(_updateState) Task=`2bbfe882-f62c-4827-b871-be3bc3118c7b`::moving from state aborting -> state failed >Thread-741::DEBUG::2012-12-02 15:16:41,452::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-741::DEBUG::2012-12-02 15:16:41,452::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-741::ERROR::2012-12-02 15:16:41,453::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-742::DEBUG::2012-12-02 15:16:41,506::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-742::DEBUG::2012-12-02 15:16:41,507::task::568::TaskManager.Task::(_updateState) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::moving from state init -> state preparing >Thread-742::INFO::2012-12-02 15:16:41,507::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-742::DEBUG::2012-12-02 15:16:41,507::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`239ca7cc-9212-4a5e-98b1-7e721e311c78`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-742::DEBUG::2012-12-02 15:16:41,507::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-742::DEBUG::2012-12-02 15:16:41,507::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-742::DEBUG::2012-12-02 15:16:41,507::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`239ca7cc-9212-4a5e-98b1-7e721e311c78`::Granted request >Thread-742::INFO::2012-12-02 15:16:41,508::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-742::DEBUG::2012-12-02 15:16:41,508::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:41,508::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:41,508::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:41,509::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:41,509::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:41,509::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:41,509::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-742::DEBUG::2012-12-02 15:16:41,509::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-742::DEBUG::2012-12-02 15:16:41,509::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-742::DEBUG::2012-12-02 15:16:41,509::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-742::DEBUG::2012-12-02 15:16:41,510::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-742::DEBUG::2012-12-02 15:16:41,534::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-742::DEBUG::2012-12-02 15:16:41,534::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-742::DEBUG::2012-12-02 15:16:42,168::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-742::DEBUG::2012-12-02 15:16:42,237::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-742::DEBUG::2012-12-02 15:16:42,237::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,238::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,238::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,238::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,238::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,238::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,238::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-742::DEBUG::2012-12-02 15:16:42,238::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,239::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-742::DEBUG::2012-12-02 15:16:42,444::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-742::WARNING::2012-12-02 15:16:42,446::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-742::DEBUG::2012-12-02 15:16:42,446::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-742::DEBUG::2012-12-02 15:16:42,477::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-742::DEBUG::2012-12-02 15:16:42,477::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-742::DEBUG::2012-12-02 15:16:42,477::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-742::DEBUG::2012-12-02 15:16:42,477::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-742::ERROR::2012-12-02 15:16:42,478::task::833::TaskManager.Task::(_setError) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-742::DEBUG::2012-12-02 15:16:42,478::task::852::TaskManager.Task::(_run) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::Task._run: ae0db305-cad3-4dc5-9aa6-ef59620f74dd ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-742::DEBUG::2012-12-02 15:16:42,478::task::1177::TaskManager.Task::(stop) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::stopping in state preparing (force False) >Thread-742::DEBUG::2012-12-02 15:16:42,478::task::957::TaskManager.Task::(_decref) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::ref 1 aborting True >Thread-742::INFO::2012-12-02 15:16:42,478::task::1134::TaskManager.Task::(prepare) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-742::DEBUG::2012-12-02 15:16:42,479::task::1139::TaskManager.Task::(prepare) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::Prepare: aborted: Cannot find master domain >Thread-742::DEBUG::2012-12-02 15:16:42,479::task::957::TaskManager.Task::(_decref) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::ref 0 aborting True >Thread-742::DEBUG::2012-12-02 15:16:42,479::task::892::TaskManager.Task::(_doAbort) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::Task._doAbort: force False >Thread-742::DEBUG::2012-12-02 15:16:42,479::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-742::DEBUG::2012-12-02 15:16:42,479::task::568::TaskManager.Task::(_updateState) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::moving from state preparing -> state aborting >Thread-742::DEBUG::2012-12-02 15:16:42,479::task::523::TaskManager.Task::(__state_aborting) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::_aborting: recover policy none >Thread-742::DEBUG::2012-12-02 15:16:42,479::task::568::TaskManager.Task::(_updateState) Task=`ae0db305-cad3-4dc5-9aa6-ef59620f74dd`::moving from state aborting -> state failed >Thread-742::DEBUG::2012-12-02 15:16:42,479::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-742::DEBUG::2012-12-02 15:16:42,480::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-742::ERROR::2012-12-02 15:16:42,480::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-803::DEBUG::2012-12-02 15:16:42,691::task::568::TaskManager.Task::(_updateState) Task=`94b575f5-700d-4a54-8309-b4d6c11fcfb9`::moving from state init -> state preparing >Thread-803::INFO::2012-12-02 15:16:42,691::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-803::INFO::2012-12-02 15:16:42,691::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-803::DEBUG::2012-12-02 15:16:42,691::task::1151::TaskManager.Task::(prepare) Task=`94b575f5-700d-4a54-8309-b4d6c11fcfb9`::finished: {} >Thread-803::DEBUG::2012-12-02 15:16:42,691::task::568::TaskManager.Task::(_updateState) Task=`94b575f5-700d-4a54-8309-b4d6c11fcfb9`::moving from state preparing -> state finished >Thread-803::DEBUG::2012-12-02 15:16:42,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-803::DEBUG::2012-12-02 15:16:42,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-803::DEBUG::2012-12-02 15:16:42,692::task::957::TaskManager.Task::(_decref) Task=`94b575f5-700d-4a54-8309-b4d6c11fcfb9`::ref 0 aborting False >Thread-809::DEBUG::2012-12-02 15:16:52,711::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-809::DEBUG::2012-12-02 15:16:52,712::task::568::TaskManager.Task::(_updateState) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::moving from state init -> state preparing >Thread-809::INFO::2012-12-02 15:16:52,712::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-809::ERROR::2012-12-02 15:16:52,712::task::833::TaskManager.Task::(_setError) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-809::DEBUG::2012-12-02 15:16:52,712::task::852::TaskManager.Task::(_run) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::Task._run: f42fdc2e-e6c3-4c32-ad70-124d6e103f20 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-809::DEBUG::2012-12-02 15:16:52,713::task::1177::TaskManager.Task::(stop) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::stopping in state preparing (force False) >Thread-809::DEBUG::2012-12-02 15:16:52,713::task::957::TaskManager.Task::(_decref) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::ref 1 aborting True >Thread-809::INFO::2012-12-02 15:16:52,713::task::1134::TaskManager.Task::(prepare) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-809::DEBUG::2012-12-02 15:16:52,713::task::1139::TaskManager.Task::(prepare) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::Prepare: aborted: Unknown pool id, pool not connected >Thread-809::DEBUG::2012-12-02 15:16:52,713::task::957::TaskManager.Task::(_decref) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::ref 0 aborting True >Thread-809::DEBUG::2012-12-02 15:16:52,713::task::892::TaskManager.Task::(_doAbort) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::Task._doAbort: force False >Thread-809::DEBUG::2012-12-02 15:16:52,713::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-809::DEBUG::2012-12-02 15:16:52,713::task::568::TaskManager.Task::(_updateState) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::moving from state preparing -> state aborting >Thread-809::DEBUG::2012-12-02 15:16:52,714::task::523::TaskManager.Task::(__state_aborting) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::_aborting: recover policy none >Thread-809::DEBUG::2012-12-02 15:16:52,714::task::568::TaskManager.Task::(_updateState) Task=`f42fdc2e-e6c3-4c32-ad70-124d6e103f20`::moving from state aborting -> state failed >Thread-809::DEBUG::2012-12-02 15:16:52,714::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-809::DEBUG::2012-12-02 15:16:52,714::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-809::ERROR::2012-12-02 15:16:52,714::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-810::DEBUG::2012-12-02 15:16:52,723::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-810::DEBUG::2012-12-02 15:16:52,723::task::568::TaskManager.Task::(_updateState) Task=`624fef1c-e02b-44b0-b033-c5bf20b0055e`::moving from state init -> state preparing >Thread-810::INFO::2012-12-02 15:16:52,723::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-810::DEBUG::2012-12-02 15:16:52,723::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-810::DEBUG::2012-12-02 15:16:52,723::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-810::INFO::2012-12-02 15:16:52,723::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-810::DEBUG::2012-12-02 15:16:52,724::task::1151::TaskManager.Task::(prepare) Task=`624fef1c-e02b-44b0-b033-c5bf20b0055e`::finished: {'allTasksStatus': {}} >Thread-810::DEBUG::2012-12-02 15:16:52,724::task::568::TaskManager.Task::(_updateState) Task=`624fef1c-e02b-44b0-b033-c5bf20b0055e`::moving from state preparing -> state finished >Thread-810::DEBUG::2012-12-02 15:16:52,724::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-810::DEBUG::2012-12-02 15:16:52,724::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-810::DEBUG::2012-12-02 15:16:52,724::task::957::TaskManager.Task::(_decref) Task=`624fef1c-e02b-44b0-b033-c5bf20b0055e`::ref 0 aborting False >Thread-811::DEBUG::2012-12-02 15:16:52,734::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-811::DEBUG::2012-12-02 15:16:52,735::task::568::TaskManager.Task::(_updateState) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::moving from state init -> state preparing >Thread-811::INFO::2012-12-02 15:16:52,735::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-811::ERROR::2012-12-02 15:16:52,735::task::833::TaskManager.Task::(_setError) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-811::DEBUG::2012-12-02 15:16:52,735::task::852::TaskManager.Task::(_run) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::Task._run: afd9f2bf-c86d-4394-83af-4f0be8c63849 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-811::DEBUG::2012-12-02 15:16:52,735::task::1177::TaskManager.Task::(stop) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::stopping in state preparing (force False) >Thread-811::DEBUG::2012-12-02 15:16:52,736::task::957::TaskManager.Task::(_decref) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::ref 1 aborting True >Thread-811::INFO::2012-12-02 15:16:52,736::task::1134::TaskManager.Task::(prepare) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-811::DEBUG::2012-12-02 15:16:52,736::task::1139::TaskManager.Task::(prepare) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::Prepare: aborted: Unknown pool id, pool not connected >Thread-811::DEBUG::2012-12-02 15:16:52,736::task::957::TaskManager.Task::(_decref) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::ref 0 aborting True >Thread-811::DEBUG::2012-12-02 15:16:52,736::task::892::TaskManager.Task::(_doAbort) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::Task._doAbort: force False >Thread-811::DEBUG::2012-12-02 15:16:52,736::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-811::DEBUG::2012-12-02 15:16:52,736::task::568::TaskManager.Task::(_updateState) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::moving from state preparing -> state aborting >Thread-811::DEBUG::2012-12-02 15:16:52,736::task::523::TaskManager.Task::(__state_aborting) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::_aborting: recover policy none >Thread-811::DEBUG::2012-12-02 15:16:52,737::task::568::TaskManager.Task::(_updateState) Task=`afd9f2bf-c86d-4394-83af-4f0be8c63849`::moving from state aborting -> state failed >Thread-811::DEBUG::2012-12-02 15:16:52,737::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-811::DEBUG::2012-12-02 15:16:52,737::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-811::ERROR::2012-12-02 15:16:52,737::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-812::DEBUG::2012-12-02 15:16:52,785::task::568::TaskManager.Task::(_updateState) Task=`937f8983-322f-4aa1-bd14-89275a518d25`::moving from state init -> state preparing >Thread-812::INFO::2012-12-02 15:16:52,785::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-812::INFO::2012-12-02 15:16:52,785::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-812::DEBUG::2012-12-02 15:16:52,786::task::1151::TaskManager.Task::(prepare) Task=`937f8983-322f-4aa1-bd14-89275a518d25`::finished: {} >Thread-812::DEBUG::2012-12-02 15:16:52,786::task::568::TaskManager.Task::(_updateState) Task=`937f8983-322f-4aa1-bd14-89275a518d25`::moving from state preparing -> state finished >Thread-812::DEBUG::2012-12-02 15:16:52,786::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-812::DEBUG::2012-12-02 15:16:52,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-812::DEBUG::2012-12-02 15:16:52,786::task::957::TaskManager.Task::(_decref) Task=`937f8983-322f-4aa1-bd14-89275a518d25`::ref 0 aborting False >Thread-813::DEBUG::2012-12-02 15:16:52,789::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-813::DEBUG::2012-12-02 15:16:52,790::task::568::TaskManager.Task::(_updateState) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::moving from state init -> state preparing >Thread-813::INFO::2012-12-02 15:16:52,790::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-813::DEBUG::2012-12-02 15:16:52,790::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`2ca11d32-89d7-4aa9-ac60-c0e64e5e2c0e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-813::DEBUG::2012-12-02 15:16:52,790::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-813::DEBUG::2012-12-02 15:16:52,790::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-813::DEBUG::2012-12-02 15:16:52,790::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`2ca11d32-89d7-4aa9-ac60-c0e64e5e2c0e`::Granted request >Thread-813::INFO::2012-12-02 15:16:52,791::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-813::DEBUG::2012-12-02 15:16:52,791::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:52,791::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:52,791::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:52,791::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:52,791::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:52,791::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:52,791::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-813::DEBUG::2012-12-02 15:16:52,791::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-813::DEBUG::2012-12-02 15:16:52,792::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-813::DEBUG::2012-12-02 15:16:52,792::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-813::DEBUG::2012-12-02 15:16:52,792::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-813::DEBUG::2012-12-02 15:16:52,818::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-813::DEBUG::2012-12-02 15:16:52,818::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-813::DEBUG::2012-12-02 15:16:53,452::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-813::DEBUG::2012-12-02 15:16:53,546::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-813::DEBUG::2012-12-02 15:16:53,547::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,547::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,547::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,547::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,547::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,547::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,547::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-813::DEBUG::2012-12-02 15:16:53,548::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,549::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-813::DEBUG::2012-12-02 15:16:53,753::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-813::WARNING::2012-12-02 15:16:53,754::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-813::DEBUG::2012-12-02 15:16:53,755::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-813::DEBUG::2012-12-02 15:16:53,782::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-813::DEBUG::2012-12-02 15:16:53,783::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-813::DEBUG::2012-12-02 15:16:53,783::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-813::DEBUG::2012-12-02 15:16:53,783::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-813::ERROR::2012-12-02 15:16:53,783::task::833::TaskManager.Task::(_setError) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-813::DEBUG::2012-12-02 15:16:53,783::task::852::TaskManager.Task::(_run) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::Task._run: 13a23e9d-cacf-47e9-b2df-cf5e31cc1760 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-813::DEBUG::2012-12-02 15:16:53,784::task::1177::TaskManager.Task::(stop) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::stopping in state preparing (force False) >Thread-813::DEBUG::2012-12-02 15:16:53,784::task::957::TaskManager.Task::(_decref) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::ref 1 aborting True >Thread-813::INFO::2012-12-02 15:16:53,784::task::1134::TaskManager.Task::(prepare) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-813::DEBUG::2012-12-02 15:16:53,784::task::1139::TaskManager.Task::(prepare) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::Prepare: aborted: Cannot find master domain >Thread-813::DEBUG::2012-12-02 15:16:53,784::task::957::TaskManager.Task::(_decref) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::ref 0 aborting True >Thread-813::DEBUG::2012-12-02 15:16:53,784::task::892::TaskManager.Task::(_doAbort) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::Task._doAbort: force False >Thread-813::DEBUG::2012-12-02 15:16:53,784::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-813::DEBUG::2012-12-02 15:16:53,785::task::568::TaskManager.Task::(_updateState) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::moving from state preparing -> state aborting >Thread-813::DEBUG::2012-12-02 15:16:53,785::task::523::TaskManager.Task::(__state_aborting) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::_aborting: recover policy none >Thread-813::DEBUG::2012-12-02 15:16:53,785::task::568::TaskManager.Task::(_updateState) Task=`13a23e9d-cacf-47e9-b2df-cf5e31cc1760`::moving from state aborting -> state failed >Thread-813::DEBUG::2012-12-02 15:16:53,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-813::DEBUG::2012-12-02 15:16:53,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-813::ERROR::2012-12-02 15:16:53,785::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-879::DEBUG::2012-12-02 15:17:02,897::task::568::TaskManager.Task::(_updateState) Task=`3290ca42-1caa-4547-90b2-9d16ef093141`::moving from state init -> state preparing >Thread-879::INFO::2012-12-02 15:17:02,897::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-879::INFO::2012-12-02 15:17:02,897::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-879::DEBUG::2012-12-02 15:17:02,897::task::1151::TaskManager.Task::(prepare) Task=`3290ca42-1caa-4547-90b2-9d16ef093141`::finished: {} >Thread-879::DEBUG::2012-12-02 15:17:02,898::task::568::TaskManager.Task::(_updateState) Task=`3290ca42-1caa-4547-90b2-9d16ef093141`::moving from state preparing -> state finished >Thread-879::DEBUG::2012-12-02 15:17:02,898::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-879::DEBUG::2012-12-02 15:17:02,898::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-879::DEBUG::2012-12-02 15:17:02,898::task::957::TaskManager.Task::(_decref) Task=`3290ca42-1caa-4547-90b2-9d16ef093141`::ref 0 aborting False >Thread-881::DEBUG::2012-12-02 15:17:03,988::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-881::DEBUG::2012-12-02 15:17:03,988::task::568::TaskManager.Task::(_updateState) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::moving from state init -> state preparing >Thread-881::INFO::2012-12-02 15:17:03,988::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-881::ERROR::2012-12-02 15:17:03,988::task::833::TaskManager.Task::(_setError) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-881::DEBUG::2012-12-02 15:17:03,989::task::852::TaskManager.Task::(_run) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::Task._run: da577893-718c-4b09-b2c9-ff2035cd0ded ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-881::DEBUG::2012-12-02 15:17:03,989::task::1177::TaskManager.Task::(stop) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::stopping in state preparing (force False) >Thread-881::DEBUG::2012-12-02 15:17:03,989::task::957::TaskManager.Task::(_decref) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::ref 1 aborting True >Thread-881::INFO::2012-12-02 15:17:03,989::task::1134::TaskManager.Task::(prepare) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-881::DEBUG::2012-12-02 15:17:03,989::task::1139::TaskManager.Task::(prepare) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::Prepare: aborted: Unknown pool id, pool not connected >Thread-881::DEBUG::2012-12-02 15:17:03,989::task::957::TaskManager.Task::(_decref) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::ref 0 aborting True >Thread-881::DEBUG::2012-12-02 15:17:03,989::task::892::TaskManager.Task::(_doAbort) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::Task._doAbort: force False >Thread-881::DEBUG::2012-12-02 15:17:03,989::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-881::DEBUG::2012-12-02 15:17:03,990::task::568::TaskManager.Task::(_updateState) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::moving from state preparing -> state aborting >Thread-881::DEBUG::2012-12-02 15:17:03,990::task::523::TaskManager.Task::(__state_aborting) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::_aborting: recover policy none >Thread-881::DEBUG::2012-12-02 15:17:03,990::task::568::TaskManager.Task::(_updateState) Task=`da577893-718c-4b09-b2c9-ff2035cd0ded`::moving from state aborting -> state failed >Thread-881::DEBUG::2012-12-02 15:17:03,990::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-881::DEBUG::2012-12-02 15:17:03,990::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-881::ERROR::2012-12-02 15:17:03,990::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-882::DEBUG::2012-12-02 15:17:03,999::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-882::DEBUG::2012-12-02 15:17:03,999::task::568::TaskManager.Task::(_updateState) Task=`1407abba-6ba1-48a4-8a02-421f6f445d86`::moving from state init -> state preparing >Thread-882::INFO::2012-12-02 15:17:03,999::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-882::DEBUG::2012-12-02 15:17:03,999::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-882::DEBUG::2012-12-02 15:17:04,000::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-882::INFO::2012-12-02 15:17:04,000::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-882::DEBUG::2012-12-02 15:17:04,000::task::1151::TaskManager.Task::(prepare) Task=`1407abba-6ba1-48a4-8a02-421f6f445d86`::finished: {'allTasksStatus': {}} >Thread-882::DEBUG::2012-12-02 15:17:04,000::task::568::TaskManager.Task::(_updateState) Task=`1407abba-6ba1-48a4-8a02-421f6f445d86`::moving from state preparing -> state finished >Thread-882::DEBUG::2012-12-02 15:17:04,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-882::DEBUG::2012-12-02 15:17:04,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-882::DEBUG::2012-12-02 15:17:04,000::task::957::TaskManager.Task::(_decref) Task=`1407abba-6ba1-48a4-8a02-421f6f445d86`::ref 0 aborting False >Thread-883::DEBUG::2012-12-02 15:17:04,010::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-883::DEBUG::2012-12-02 15:17:04,011::task::568::TaskManager.Task::(_updateState) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::moving from state init -> state preparing >Thread-883::INFO::2012-12-02 15:17:04,011::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-883::ERROR::2012-12-02 15:17:04,011::task::833::TaskManager.Task::(_setError) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-883::DEBUG::2012-12-02 15:17:04,011::task::852::TaskManager.Task::(_run) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::Task._run: 1e90a611-c92a-4aed-b60a-4e7e5a00982b ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-883::DEBUG::2012-12-02 15:17:04,011::task::1177::TaskManager.Task::(stop) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::stopping in state preparing (force False) >Thread-883::DEBUG::2012-12-02 15:17:04,011::task::957::TaskManager.Task::(_decref) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::ref 1 aborting True >Thread-883::INFO::2012-12-02 15:17:04,012::task::1134::TaskManager.Task::(prepare) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-883::DEBUG::2012-12-02 15:17:04,012::task::1139::TaskManager.Task::(prepare) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::Prepare: aborted: Unknown pool id, pool not connected >Thread-883::DEBUG::2012-12-02 15:17:04,012::task::957::TaskManager.Task::(_decref) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::ref 0 aborting True >Thread-883::DEBUG::2012-12-02 15:17:04,012::task::892::TaskManager.Task::(_doAbort) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::Task._doAbort: force False >Thread-883::DEBUG::2012-12-02 15:17:04,012::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-883::DEBUG::2012-12-02 15:17:04,012::task::568::TaskManager.Task::(_updateState) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::moving from state preparing -> state aborting >Thread-883::DEBUG::2012-12-02 15:17:04,012::task::523::TaskManager.Task::(__state_aborting) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::_aborting: recover policy none >Thread-883::DEBUG::2012-12-02 15:17:04,012::task::568::TaskManager.Task::(_updateState) Task=`1e90a611-c92a-4aed-b60a-4e7e5a00982b`::moving from state aborting -> state failed >Thread-883::DEBUG::2012-12-02 15:17:04,013::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-883::DEBUG::2012-12-02 15:17:04,013::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-883::ERROR::2012-12-02 15:17:04,013::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-884::DEBUG::2012-12-02 15:17:04,063::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-884::DEBUG::2012-12-02 15:17:04,063::task::568::TaskManager.Task::(_updateState) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::moving from state init -> state preparing >Thread-884::INFO::2012-12-02 15:17:04,063::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-884::DEBUG::2012-12-02 15:17:04,064::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`8f08a9fb-a532-48a0-afbc-dfe294cf11fe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-884::DEBUG::2012-12-02 15:17:04,064::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-884::DEBUG::2012-12-02 15:17:04,064::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-884::DEBUG::2012-12-02 15:17:04,064::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`8f08a9fb-a532-48a0-afbc-dfe294cf11fe`::Granted request >Thread-884::INFO::2012-12-02 15:17:04,064::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-884::DEBUG::2012-12-02 15:17:04,065::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,065::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,065::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,065::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,065::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,065::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,065::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-884::DEBUG::2012-12-02 15:17:04,066::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-884::DEBUG::2012-12-02 15:17:04,066::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-884::DEBUG::2012-12-02 15:17:04,066::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-884::DEBUG::2012-12-02 15:17:04,066::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-884::DEBUG::2012-12-02 15:17:04,088::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-884::DEBUG::2012-12-02 15:17:04,089::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-884::DEBUG::2012-12-02 15:17:04,722::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-884::DEBUG::2012-12-02 15:17:04,778::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-884::DEBUG::2012-12-02 15:17:04,779::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,779::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,779::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,779::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,779::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,780::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,780::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-884::DEBUG::2012-12-02 15:17:04,780::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-884::DEBUG::2012-12-02 15:17:04,781::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-884::DEBUG::2012-12-02 15:17:04,990::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-884::WARNING::2012-12-02 15:17:04,992::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-884::DEBUG::2012-12-02 15:17:04,992::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-928::WARNING::2012-12-02 15:17:05,011::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-931::WARNING::2012-12-02 15:17:05,013::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-933::WARNING::2012-12-02 15:17:05,014::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-935::WARNING::2012-12-02 15:17:05,015::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-937::WARNING::2012-12-02 15:17:05,016::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-938::WARNING::2012-12-02 15:17:05,016::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-940::WARNING::2012-12-02 15:17:05,017::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-941::WARNING::2012-12-02 15:17:05,018::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-942::WARNING::2012-12-02 15:17:05,018::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-943::WARNING::2012-12-02 15:17:05,019::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-944::WARNING::2012-12-02 15:17:05,019::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-945::WARNING::2012-12-02 15:17:05,020::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-884::DEBUG::2012-12-02 15:17:05,046::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-884::DEBUG::2012-12-02 15:17:05,046::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-884::DEBUG::2012-12-02 15:17:05,046::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-884::DEBUG::2012-12-02 15:17:05,046::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-884::ERROR::2012-12-02 15:17:05,047::task::833::TaskManager.Task::(_setError) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-884::DEBUG::2012-12-02 15:17:05,047::task::852::TaskManager.Task::(_run) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::Task._run: b53a5f1b-4b37-4391-8479-1cc1f70291ad ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-884::DEBUG::2012-12-02 15:17:05,047::task::1177::TaskManager.Task::(stop) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::stopping in state preparing (force False) >Thread-884::DEBUG::2012-12-02 15:17:05,047::task::957::TaskManager.Task::(_decref) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::ref 1 aborting True >Thread-884::INFO::2012-12-02 15:17:05,047::task::1134::TaskManager.Task::(prepare) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-884::DEBUG::2012-12-02 15:17:05,047::task::1139::TaskManager.Task::(prepare) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::Prepare: aborted: Cannot find master domain >Thread-884::DEBUG::2012-12-02 15:17:05,048::task::957::TaskManager.Task::(_decref) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::ref 0 aborting True >Thread-884::DEBUG::2012-12-02 15:17:05,048::task::892::TaskManager.Task::(_doAbort) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::Task._doAbort: force False >Thread-884::DEBUG::2012-12-02 15:17:05,048::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-884::DEBUG::2012-12-02 15:17:05,048::task::568::TaskManager.Task::(_updateState) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::moving from state preparing -> state aborting >Thread-884::DEBUG::2012-12-02 15:17:05,048::task::523::TaskManager.Task::(__state_aborting) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::_aborting: recover policy none >Thread-884::DEBUG::2012-12-02 15:17:05,048::task::568::TaskManager.Task::(_updateState) Task=`b53a5f1b-4b37-4391-8479-1cc1f70291ad`::moving from state aborting -> state failed >Thread-884::DEBUG::2012-12-02 15:17:05,048::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-884::DEBUG::2012-12-02 15:17:05,048::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-884::ERROR::2012-12-02 15:17:05,049::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-949::DEBUG::2012-12-02 15:17:13,000::task::568::TaskManager.Task::(_updateState) Task=`cf44912b-73b7-4d12-9af4-3fdc36fb47f6`::moving from state init -> state preparing >Thread-949::INFO::2012-12-02 15:17:13,000::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-949::INFO::2012-12-02 15:17:13,000::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-949::DEBUG::2012-12-02 15:17:13,000::task::1151::TaskManager.Task::(prepare) Task=`cf44912b-73b7-4d12-9af4-3fdc36fb47f6`::finished: {} >Thread-949::DEBUG::2012-12-02 15:17:13,000::task::568::TaskManager.Task::(_updateState) Task=`cf44912b-73b7-4d12-9af4-3fdc36fb47f6`::moving from state preparing -> state finished >Thread-949::DEBUG::2012-12-02 15:17:13,001::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-949::DEBUG::2012-12-02 15:17:13,001::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-949::DEBUG::2012-12-02 15:17:13,001::task::957::TaskManager.Task::(_decref) Task=`cf44912b-73b7-4d12-9af4-3fdc36fb47f6`::ref 0 aborting False >Thread-952::DEBUG::2012-12-02 15:17:15,289::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-952::DEBUG::2012-12-02 15:17:15,289::task::568::TaskManager.Task::(_updateState) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::moving from state init -> state preparing >Thread-952::INFO::2012-12-02 15:17:15,289::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-952::ERROR::2012-12-02 15:17:15,289::task::833::TaskManager.Task::(_setError) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-952::DEBUG::2012-12-02 15:17:15,290::task::852::TaskManager.Task::(_run) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::Task._run: 2e23ce3d-6b84-422b-998d-42efe6150001 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-952::DEBUG::2012-12-02 15:17:15,290::task::1177::TaskManager.Task::(stop) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::stopping in state preparing (force False) >Thread-952::DEBUG::2012-12-02 15:17:15,290::task::957::TaskManager.Task::(_decref) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::ref 1 aborting True >Thread-952::INFO::2012-12-02 15:17:15,290::task::1134::TaskManager.Task::(prepare) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-952::DEBUG::2012-12-02 15:17:15,290::task::1139::TaskManager.Task::(prepare) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::Prepare: aborted: Unknown pool id, pool not connected >Thread-952::DEBUG::2012-12-02 15:17:15,290::task::957::TaskManager.Task::(_decref) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::ref 0 aborting True >Thread-952::DEBUG::2012-12-02 15:17:15,290::task::892::TaskManager.Task::(_doAbort) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::Task._doAbort: force False >Thread-952::DEBUG::2012-12-02 15:17:15,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-952::DEBUG::2012-12-02 15:17:15,291::task::568::TaskManager.Task::(_updateState) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::moving from state preparing -> state aborting >Thread-952::DEBUG::2012-12-02 15:17:15,291::task::523::TaskManager.Task::(__state_aborting) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::_aborting: recover policy none >Thread-952::DEBUG::2012-12-02 15:17:15,291::task::568::TaskManager.Task::(_updateState) Task=`2e23ce3d-6b84-422b-998d-42efe6150001`::moving from state aborting -> state failed >Thread-952::DEBUG::2012-12-02 15:17:15,291::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-952::DEBUG::2012-12-02 15:17:15,291::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-952::ERROR::2012-12-02 15:17:15,291::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-953::DEBUG::2012-12-02 15:17:15,300::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-953::DEBUG::2012-12-02 15:17:15,300::task::568::TaskManager.Task::(_updateState) Task=`2e7f481f-4f79-4c31-a3f0-f0dc936948f5`::moving from state init -> state preparing >Thread-953::INFO::2012-12-02 15:17:15,301::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-953::DEBUG::2012-12-02 15:17:15,301::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-953::DEBUG::2012-12-02 15:17:15,301::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-953::INFO::2012-12-02 15:17:15,301::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-953::DEBUG::2012-12-02 15:17:15,301::task::1151::TaskManager.Task::(prepare) Task=`2e7f481f-4f79-4c31-a3f0-f0dc936948f5`::finished: {'allTasksStatus': {}} >Thread-953::DEBUG::2012-12-02 15:17:15,301::task::568::TaskManager.Task::(_updateState) Task=`2e7f481f-4f79-4c31-a3f0-f0dc936948f5`::moving from state preparing -> state finished >Thread-953::DEBUG::2012-12-02 15:17:15,301::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-953::DEBUG::2012-12-02 15:17:15,301::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-953::DEBUG::2012-12-02 15:17:15,301::task::957::TaskManager.Task::(_decref) Task=`2e7f481f-4f79-4c31-a3f0-f0dc936948f5`::ref 0 aborting False >Thread-954::DEBUG::2012-12-02 15:17:15,311::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-954::DEBUG::2012-12-02 15:17:15,312::task::568::TaskManager.Task::(_updateState) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::moving from state init -> state preparing >Thread-954::INFO::2012-12-02 15:17:15,312::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-954::ERROR::2012-12-02 15:17:15,312::task::833::TaskManager.Task::(_setError) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-954::DEBUG::2012-12-02 15:17:15,312::task::852::TaskManager.Task::(_run) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::Task._run: 4590f506-5e13-4fba-a3f0-283a956a9cc8 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-954::DEBUG::2012-12-02 15:17:15,312::task::1177::TaskManager.Task::(stop) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::stopping in state preparing (force False) >Thread-954::DEBUG::2012-12-02 15:17:15,313::task::957::TaskManager.Task::(_decref) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::ref 1 aborting True >Thread-954::INFO::2012-12-02 15:17:15,313::task::1134::TaskManager.Task::(prepare) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-954::DEBUG::2012-12-02 15:17:15,313::task::1139::TaskManager.Task::(prepare) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::Prepare: aborted: Unknown pool id, pool not connected >Thread-954::DEBUG::2012-12-02 15:17:15,313::task::957::TaskManager.Task::(_decref) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::ref 0 aborting True >Thread-954::DEBUG::2012-12-02 15:17:15,313::task::892::TaskManager.Task::(_doAbort) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::Task._doAbort: force False >Thread-954::DEBUG::2012-12-02 15:17:15,313::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-954::DEBUG::2012-12-02 15:17:15,313::task::568::TaskManager.Task::(_updateState) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::moving from state preparing -> state aborting >Thread-954::DEBUG::2012-12-02 15:17:15,313::task::523::TaskManager.Task::(__state_aborting) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::_aborting: recover policy none >Thread-954::DEBUG::2012-12-02 15:17:15,314::task::568::TaskManager.Task::(_updateState) Task=`4590f506-5e13-4fba-a3f0-283a956a9cc8`::moving from state aborting -> state failed >Thread-954::DEBUG::2012-12-02 15:17:15,314::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-954::DEBUG::2012-12-02 15:17:15,314::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-954::ERROR::2012-12-02 15:17:15,314::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-955::DEBUG::2012-12-02 15:17:15,381::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-955::DEBUG::2012-12-02 15:17:15,382::task::568::TaskManager.Task::(_updateState) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::moving from state init -> state preparing >Thread-955::INFO::2012-12-02 15:17:15,382::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', msdUUID='4e891d8c-9947-468e-88ec-29e49d68e083', masterVersion=2, options=None) >Thread-955::DEBUG::2012-12-02 15:17:15,382::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`75e79454-244e-4ee5-a2f1-31b543ec6acd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-955::DEBUG::2012-12-02 15:17:15,382::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' for lock type 'exclusive' >Thread-955::DEBUG::2012-12-02 15:17:15,383::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free. Now locking as 'exclusive' (1 active user) >Thread-955::DEBUG::2012-12-02 15:17:15,383::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6`ReqID=`75e79454-244e-4ee5-a2f1-31b543ec6acd`::Granted request >Thread-955::INFO::2012-12-02 15:17:15,383::sp::622::Storage.StoragePool::(connect) Connect host #3 to the storage pool da40b79b-9f51-4240-9cb2-2ab10e793fb6 with master domain: 4e891d8c-9947-468e-88ec-29e49d68e083 (ver = 2) >Thread-955::DEBUG::2012-12-02 15:17:15,383::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:15,383::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:15,384::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:15,384::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:15,384::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:15,384::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:15,384::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-955::DEBUG::2012-12-02 15:17:15,384::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-955::DEBUG::2012-12-02 15:17:15,385::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-955::DEBUG::2012-12-02 15:17:15,385::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-955::DEBUG::2012-12-02 15:17:15,385::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-955::DEBUG::2012-12-02 15:17:15,409::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-955::DEBUG::2012-12-02 15:17:15,409::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-955::DEBUG::2012-12-02 15:17:16,053::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-955::DEBUG::2012-12-02 15:17:16,120::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-955::DEBUG::2012-12-02 15:17:16,121::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,121::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,121::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,121::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,122::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,122::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,122::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-955::DEBUG::2012-12-02 15:17:16,122::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,124::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4e891d8c-9947-468e-88ec-29e49d68e083' (cwd None) >Thread-955::DEBUG::2012-12-02 15:17:16,329::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found\n'; <rc> = 5 >Thread-955::WARNING::2012-12-02 15:17:16,330::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "4e891d8c-9947-468e-88ec-29e49d68e083" not found'] >Thread-955::DEBUG::2012-12-02 15:17:16,330::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-955::DEBUG::2012-12-02 15:17:16,363::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' >Thread-955::DEBUG::2012-12-02 15:17:16,363::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' (0 active users) >Thread-955::DEBUG::2012-12-02 15:17:16,363::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6' is free, finding out if anyone is waiting for it. >Thread-955::DEBUG::2012-12-02 15:17:16,363::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.da40b79b-9f51-4240-9cb2-2ab10e793fb6', Clearing records. >Thread-955::ERROR::2012-12-02 15:17:16,363::task::833::TaskManager.Task::(_setError) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083' >Thread-955::DEBUG::2012-12-02 15:17:16,364::task::852::TaskManager.Task::(_run) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::Task._run: 4c60ef5b-e646-449f-b3ae-dd38de57f50d ('da40b79b-9f51-4240-9cb2-2ab10e793fb6', 3, 'da40b79b-9f51-4240-9cb2-2ab10e793fb6', '4e891d8c-9947-468e-88ec-29e49d68e083', 2) {} failed - stopping task >Thread-955::DEBUG::2012-12-02 15:17:16,364::task::1177::TaskManager.Task::(stop) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::stopping in state preparing (force False) >Thread-955::DEBUG::2012-12-02 15:17:16,364::task::957::TaskManager.Task::(_decref) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::ref 1 aborting True >Thread-955::INFO::2012-12-02 15:17:16,364::task::1134::TaskManager.Task::(prepare) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-955::DEBUG::2012-12-02 15:17:16,364::task::1139::TaskManager.Task::(prepare) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::Prepare: aborted: Cannot find master domain >Thread-955::DEBUG::2012-12-02 15:17:16,365::task::957::TaskManager.Task::(_decref) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::ref 0 aborting True >Thread-955::DEBUG::2012-12-02 15:17:16,365::task::892::TaskManager.Task::(_doAbort) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::Task._doAbort: force False >Thread-955::DEBUG::2012-12-02 15:17:16,365::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-955::DEBUG::2012-12-02 15:17:16,365::task::568::TaskManager.Task::(_updateState) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::moving from state preparing -> state aborting >Thread-955::DEBUG::2012-12-02 15:17:16,365::task::523::TaskManager.Task::(__state_aborting) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::_aborting: recover policy none >Thread-955::DEBUG::2012-12-02 15:17:16,365::task::568::TaskManager.Task::(_updateState) Task=`4c60ef5b-e646-449f-b3ae-dd38de57f50d`::moving from state aborting -> state failed >Thread-955::DEBUG::2012-12-02 15:17:16,365::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-955::DEBUG::2012-12-02 15:17:16,365::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-955::ERROR::2012-12-02 15:17:16,366::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=da40b79b-9f51-4240-9cb2-2ab10e793fb6, msdUUID=4e891d8c-9947-468e-88ec-29e49d68e083'", 'code': 304}} >Thread-1019::DEBUG::2012-12-02 15:17:23,088::task::568::TaskManager.Task::(_updateState) Task=`f9cd8ec8-7eda-4c8b-9726-086c4dfb8df6`::moving from state init -> state preparing >Thread-1019::INFO::2012-12-02 15:17:23,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1019::INFO::2012-12-02 15:17:23,088::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1019::DEBUG::2012-12-02 15:17:23,089::task::1151::TaskManager.Task::(prepare) Task=`f9cd8ec8-7eda-4c8b-9726-086c4dfb8df6`::finished: {} >Thread-1019::DEBUG::2012-12-02 15:17:23,089::task::568::TaskManager.Task::(_updateState) Task=`f9cd8ec8-7eda-4c8b-9726-086c4dfb8df6`::moving from state preparing -> state finished >Thread-1019::DEBUG::2012-12-02 15:17:23,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1019::DEBUG::2012-12-02 15:17:23,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1019::DEBUG::2012-12-02 15:17:23,089::task::957::TaskManager.Task::(_decref) Task=`f9cd8ec8-7eda-4c8b-9726-086c4dfb8df6`::ref 0 aborting False >Thread-1022::DEBUG::2012-12-02 15:17:26,607::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-1022::DEBUG::2012-12-02 15:17:26,608::task::568::TaskManager.Task::(_updateState) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::moving from state init -> state preparing >Thread-1022::INFO::2012-12-02 15:17:26,608::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', options=None) >Thread-1022::ERROR::2012-12-02 15:17:26,608::task::833::TaskManager.Task::(_setError) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) >Thread-1022::DEBUG::2012-12-02 15:17:26,608::task::852::TaskManager.Task::(_run) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::Task._run: ba70cdea-10c6-4118-a838-d495b83a0f70 ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',) {} failed - stopping task >Thread-1022::DEBUG::2012-12-02 15:17:26,608::task::1177::TaskManager.Task::(stop) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::stopping in state preparing (force False) >Thread-1022::DEBUG::2012-12-02 15:17:26,609::task::957::TaskManager.Task::(_decref) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::ref 1 aborting True >Thread-1022::INFO::2012-12-02 15:17:26,609::task::1134::TaskManager.Task::(prepare) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1022::DEBUG::2012-12-02 15:17:26,609::task::1139::TaskManager.Task::(prepare) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1022::DEBUG::2012-12-02 15:17:26,609::task::957::TaskManager.Task::(_decref) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::ref 0 aborting True >Thread-1022::DEBUG::2012-12-02 15:17:26,609::task::892::TaskManager.Task::(_doAbort) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::Task._doAbort: force False >Thread-1022::DEBUG::2012-12-02 15:17:26,609::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1022::DEBUG::2012-12-02 15:17:26,609::task::568::TaskManager.Task::(_updateState) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::moving from state preparing -> state aborting >Thread-1022::DEBUG::2012-12-02 15:17:26,609::task::523::TaskManager.Task::(__state_aborting) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::_aborting: recover policy none >Thread-1022::DEBUG::2012-12-02 15:17:26,610::task::568::TaskManager.Task::(_updateState) Task=`ba70cdea-10c6-4118-a838-d495b83a0f70`::moving from state aborting -> state failed >Thread-1022::DEBUG::2012-12-02 15:17:26,610::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1022::DEBUG::2012-12-02 15:17:26,610::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1022::ERROR::2012-12-02 15:17:26,610::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('da40b79b-9f51-4240-9cb2-2ab10e793fb6',)", 'code': 309}} >Thread-1023::DEBUG::2012-12-02 15:17:26,619::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-1023::DEBUG::2012-12-02 15:17:26,619::task::568::TaskManager.Task::(_updateState) Task=`e21158fd-1d36-498e-b4b7-c135e1872b86`::moving from state init -> state preparing >Thread-1023::INFO::2012-12-02 15:17:26,619::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1023::DEBUG::2012-12-02 15:17:26,619::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1023::DEBUG::2012-12-02 15:17:26,619::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1023::INFO::2012-12-02 15:17:26,620::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1023::DEBUG::2012-12-02 15:17:26,620::task::1151::TaskManager.Task::(prepare) Task=`e21158fd-1d36-498e-b4b7-c135e1872b86`::finished: {'allTasksStatus': {}} >Thread-1023::DEBUG::2012-12-02 15:17:26,620::task::568::TaskManager.Task::(_updateState) Task=`e21158fd-1d36-498e-b4b7-c135e1872b86`::moving from state preparing -> state finished >Thread-1023::DEBUG::2012-12-02 15:17:26,620::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1023::DEBUG::2012-12-02 15:17:26,620::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1023::DEBUG::2012-12-02 15:17:26,620::task::957::TaskManager.Task::(_decref) Task=`e21158fd-1d36-498e-b4b7-c135e1872b86`::ref 0 aborting False >Thread-1024::DEBUG::2012-12-02 15:17:27,176::task::568::TaskManager.Task::(_updateState) Task=`208bb454-12a8-4dab-8ac6-7ec04b80dd74`::moving from state init -> state preparing >Thread-1024::INFO::2012-12-02 15:17:27,176::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1024::INFO::2012-12-02 15:17:27,176::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1024::DEBUG::2012-12-02 15:17:27,177::task::1151::TaskManager.Task::(prepare) Task=`208bb454-12a8-4dab-8ac6-7ec04b80dd74`::finished: {} >Thread-1024::DEBUG::2012-12-02 15:17:27,177::task::568::TaskManager.Task::(_updateState) Task=`208bb454-12a8-4dab-8ac6-7ec04b80dd74`::moving from state preparing -> state finished >Thread-1024::DEBUG::2012-12-02 15:17:27,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1024::DEBUG::2012-12-02 15:17:27,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1024::DEBUG::2012-12-02 15:17:27,177::task::957::TaskManager.Task::(_decref) Task=`208bb454-12a8-4dab-8ac6-7ec04b80dd74`::ref 0 aborting False >Thread-1026::DEBUG::2012-12-02 15:17:27,251::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-1026::DEBUG::2012-12-02 15:17:27,251::task::568::TaskManager.Task::(_updateState) Task=`91e62960-a2af-4572-804b-935b8115e3f1`::moving from state init -> state preparing >Thread-1026::INFO::2012-12-02 15:17:27,251::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', hostID=3, scsiKey='da40b79b-9f51-4240-9cb2-2ab10e793fb6', remove=False, options=None) >Thread-1026::WARNING::2012-12-02 15:17:27,251::hsm::887::Storage.HSM::(disconnectStoragePool) disconnect sp: da40b79b-9f51-4240-9cb2-2ab10e793fb6 failed. Known pools {} >Thread-1026::INFO::2012-12-02 15:17:27,251::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None >Thread-1026::DEBUG::2012-12-02 15:17:27,252::task::1151::TaskManager.Task::(prepare) Task=`91e62960-a2af-4572-804b-935b8115e3f1`::finished: None >Thread-1026::DEBUG::2012-12-02 15:17:27,252::task::568::TaskManager.Task::(_updateState) Task=`91e62960-a2af-4572-804b-935b8115e3f1`::moving from state preparing -> state finished >Thread-1026::DEBUG::2012-12-02 15:17:27,252::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1026::DEBUG::2012-12-02 15:17:27,252::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1026::DEBUG::2012-12-02 15:17:27,252::task::957::TaskManager.Task::(_decref) Task=`91e62960-a2af-4572-804b-935b8115e3f1`::ref 0 aborting False >Thread-1027::DEBUG::2012-12-02 15:17:27,282::BindingXMLRPC::160::vds::(wrapper) [10.35.1.163] >Thread-1027::DEBUG::2012-12-02 15:17:27,282::task::568::TaskManager.Task::(_updateState) Task=`7eba23fd-9a9a-49ba-a212-df431131282d`::moving from state init -> state preparing >Thread-1027::INFO::2012-12-02 15:17:27,282::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='da40b79b-9f51-4240-9cb2-2ab10e793fb6', conList=[], options=None) >Thread-1027::DEBUG::2012-12-02 15:17:27,283::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1027::DEBUG::2012-12-02 15:17:27,283::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1027::DEBUG::2012-12-02 15:17:27,283::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1027::DEBUG::2012-12-02 15:17:27,283::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1027::DEBUG::2012-12-02 15:17:27,283::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1027::DEBUG::2012-12-02 15:17:27,308::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1027::DEBUG::2012-12-02 15:17:27,308::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1027::DEBUG::2012-12-02 15:17:27,953::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1027::DEBUG::2012-12-02 15:17:28,012::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1027::DEBUG::2012-12-02 15:17:28,013::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1027::DEBUG::2012-12-02 15:17:28,013::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1027::DEBUG::2012-12-02 15:17:28,013::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1027::DEBUG::2012-12-02 15:17:28,013::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1027::DEBUG::2012-12-02 15:17:28,013::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1027::DEBUG::2012-12-02 15:17:28,014::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1027::DEBUG::2012-12-02 15:17:28,014::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1027::INFO::2012-12-02 15:17:28,014::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': []} >Thread-1027::DEBUG::2012-12-02 15:17:28,014::task::1151::TaskManager.Task::(prepare) Task=`7eba23fd-9a9a-49ba-a212-df431131282d`::finished: {'statuslist': []} >Thread-1027::DEBUG::2012-12-02 15:17:28,014::task::568::TaskManager.Task::(_updateState) Task=`7eba23fd-9a9a-49ba-a212-df431131282d`::moving from state preparing -> state finished >Thread-1027::DEBUG::2012-12-02 15:17:28,014::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1027::DEBUG::2012-12-02 15:17:28,014::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1027::DEBUG::2012-12-02 15:17:28,015::task::957::TaskManager.Task::(_decref) Task=`7eba23fd-9a9a-49ba-a212-df431131282d`::ref 0 aborting False >Thread-1028::DEBUG::2012-12-02 22:47:52,544::BindingXMLRPC::160::vds::(wrapper) [10.35.1.62] >Thread-1028::DEBUG::2012-12-02 22:47:52,548::task::568::TaskManager.Task::(_updateState) Task=`faa323c6-ebf4-4650-a1ff-c04b3ecda87b`::moving from state init -> state preparing >Thread-1028::INFO::2012-12-02 22:47:52,548::logUtils::37::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-1028::INFO::2012-12-02 22:47:52,548::logUtils::39::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-1028::DEBUG::2012-12-02 22:47:52,548::task::1151::TaskManager.Task::(prepare) Task=`faa323c6-ebf4-4650-a1ff-c04b3ecda87b`::finished: {'poollist': []} >Thread-1028::DEBUG::2012-12-02 22:47:52,548::task::568::TaskManager.Task::(_updateState) Task=`faa323c6-ebf4-4650-a1ff-c04b3ecda87b`::moving from state preparing -> state finished >Thread-1028::DEBUG::2012-12-02 22:47:52,548::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1028::DEBUG::2012-12-02 22:47:52,548::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1028::DEBUG::2012-12-02 22:47:52,548::task::957::TaskManager.Task::(_decref) Task=`faa323c6-ebf4-4650-a1ff-c04b3ecda87b`::ref 0 aborting False >Thread-1029::DEBUG::2012-12-02 22:47:53,784::BindingXMLRPC::160::vds::(wrapper) [10.35.1.62] >Thread-1029::DEBUG::2012-12-02 22:47:53,785::task::568::TaskManager.Task::(_updateState) Task=`2878faec-64f3-4da0-b975-d099f0cc6661`::moving from state init -> state preparing >Thread-1029::INFO::2012-12-02 22:47:53,785::logUtils::37::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList(options=None) >Thread-1029::INFO::2012-12-02 22:47:53,785::logUtils::39::dispatcher::(wrapper) Run and protect: getConnectedStoragePoolsList, Return response: {'poollist': []} >Thread-1029::DEBUG::2012-12-02 22:47:53,785::task::1151::TaskManager.Task::(prepare) Task=`2878faec-64f3-4da0-b975-d099f0cc6661`::finished: {'poollist': []} >Thread-1029::DEBUG::2012-12-02 22:47:53,785::task::568::TaskManager.Task::(_updateState) Task=`2878faec-64f3-4da0-b975-d099f0cc6661`::moving from state preparing -> state finished >Thread-1029::DEBUG::2012-12-02 22:47:53,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1029::DEBUG::2012-12-02 22:47:53,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1029::DEBUG::2012-12-02 22:47:53,786::task::957::TaskManager.Task::(_decref) Task=`2878faec-64f3-4da0-b975-d099f0cc6661`::ref 0 aborting False >Thread-1030::DEBUG::2012-12-02 22:48:16,280::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-1030::DEBUG::2012-12-02 22:48:16,297::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-1030::DEBUG::2012-12-02 22:48:16,310::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-1030::DEBUG::2012-12-02 22:48:16,310::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-1030::DEBUG::2012-12-02 22:48:16,322::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-1030::DEBUG::2012-12-02 22:48:16,323::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-1030::DEBUG::2012-12-02 22:48:16,333::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1030::DEBUG::2012-12-02 22:48:16,333::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-1030::DEBUG::2012-12-02 22:48:16,344::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1030::DEBUG::2012-12-02 22:48:16,345::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-1030::DEBUG::2012-12-02 22:48:16,357::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1030::DEBUG::2012-12-02 22:48:16,358::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '3401.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.1.62'}} >Thread-1031::DEBUG::2012-12-02 22:48:16,934::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-1031::DEBUG::2012-12-02 22:48:16,945::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-1031::DEBUG::2012-12-02 22:48:16,957::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-1031::DEBUG::2012-12-02 22:48:16,957::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-1031::DEBUG::2012-12-02 22:48:16,967::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-1031::DEBUG::2012-12-02 22:48:16,967::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-1031::DEBUG::2012-12-02 22:48:16,979::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1031::DEBUG::2012-12-02 22:48:16,979::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-1031::DEBUG::2012-12-02 22:48:16,992::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1031::DEBUG::2012-12-02 22:48:16,992::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-1031::DEBUG::2012-12-02 22:48:17,005::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1031::DEBUG::2012-12-02 22:48:17,006::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.202.48'}} >Thread-1037::DEBUG::2012-12-02 22:48:28,076::task::568::TaskManager.Task::(_updateState) Task=`d67544f5-ad5e-4ad0-a766-1af17d4a3248`::moving from state init -> state preparing >Thread-1037::INFO::2012-12-02 22:48:28,076::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1037::INFO::2012-12-02 22:48:28,076::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1037::DEBUG::2012-12-02 22:48:28,076::task::1151::TaskManager.Task::(prepare) Task=`d67544f5-ad5e-4ad0-a766-1af17d4a3248`::finished: {} >Thread-1037::DEBUG::2012-12-02 22:48:28,076::task::568::TaskManager.Task::(_updateState) Task=`d67544f5-ad5e-4ad0-a766-1af17d4a3248`::moving from state preparing -> state finished >Thread-1037::DEBUG::2012-12-02 22:48:28,076::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1037::DEBUG::2012-12-02 22:48:28,077::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1037::DEBUG::2012-12-02 22:48:28,077::task::957::TaskManager.Task::(_decref) Task=`d67544f5-ad5e-4ad0-a766-1af17d4a3248`::ref 0 aborting False >Thread-1043::DEBUG::2012-12-02 22:48:39,028::task::568::TaskManager.Task::(_updateState) Task=`9681c2a5-12ba-46d2-af47-b27c7d303793`::moving from state init -> state preparing >Thread-1043::INFO::2012-12-02 22:48:39,028::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1043::INFO::2012-12-02 22:48:39,028::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1043::DEBUG::2012-12-02 22:48:39,028::task::1151::TaskManager.Task::(prepare) Task=`9681c2a5-12ba-46d2-af47-b27c7d303793`::finished: {} >Thread-1043::DEBUG::2012-12-02 22:48:39,028::task::568::TaskManager.Task::(_updateState) Task=`9681c2a5-12ba-46d2-af47-b27c7d303793`::moving from state preparing -> state finished >Thread-1043::DEBUG::2012-12-02 22:48:39,028::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1043::DEBUG::2012-12-02 22:48:39,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1043::DEBUG::2012-12-02 22:48:39,029::task::957::TaskManager.Task::(_decref) Task=`9681c2a5-12ba-46d2-af47-b27c7d303793`::ref 0 aborting False >Thread-1049::DEBUG::2012-12-02 22:48:49,999::task::568::TaskManager.Task::(_updateState) Task=`64155bea-04e7-4171-a9f3-2e0f9bb24a82`::moving from state init -> state preparing >Thread-1049::INFO::2012-12-02 22:48:49,999::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1049::INFO::2012-12-02 22:48:49,999::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1049::DEBUG::2012-12-02 22:48:50,000::task::1151::TaskManager.Task::(prepare) Task=`64155bea-04e7-4171-a9f3-2e0f9bb24a82`::finished: {} >Thread-1049::DEBUG::2012-12-02 22:48:50,000::task::568::TaskManager.Task::(_updateState) Task=`64155bea-04e7-4171-a9f3-2e0f9bb24a82`::moving from state preparing -> state finished >Thread-1049::DEBUG::2012-12-02 22:48:50,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1049::DEBUG::2012-12-02 22:48:50,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1049::DEBUG::2012-12-02 22:48:50,000::task::957::TaskManager.Task::(_decref) Task=`64155bea-04e7-4171-a9f3-2e0f9bb24a82`::ref 0 aborting False >Thread-1055::DEBUG::2012-12-02 22:49:00,974::task::568::TaskManager.Task::(_updateState) Task=`447ed1c8-61df-442d-998f-a29a40db8ae7`::moving from state init -> state preparing >Thread-1055::INFO::2012-12-02 22:49:00,975::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1055::INFO::2012-12-02 22:49:00,975::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1055::DEBUG::2012-12-02 22:49:00,975::task::1151::TaskManager.Task::(prepare) Task=`447ed1c8-61df-442d-998f-a29a40db8ae7`::finished: {} >Thread-1055::DEBUG::2012-12-02 22:49:00,975::task::568::TaskManager.Task::(_updateState) Task=`447ed1c8-61df-442d-998f-a29a40db8ae7`::moving from state preparing -> state finished >Thread-1055::DEBUG::2012-12-02 22:49:00,975::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1055::DEBUG::2012-12-02 22:49:00,975::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1055::DEBUG::2012-12-02 22:49:00,975::task::957::TaskManager.Task::(_decref) Task=`447ed1c8-61df-442d-998f-a29a40db8ae7`::ref 0 aborting False >Thread-1061::DEBUG::2012-12-02 22:49:11,937::task::568::TaskManager.Task::(_updateState) Task=`ec16038a-0bf0-4792-b434-b152d07e91b8`::moving from state init -> state preparing >Thread-1061::INFO::2012-12-02 22:49:11,938::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1061::INFO::2012-12-02 22:49:11,938::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1061::DEBUG::2012-12-02 22:49:11,938::task::1151::TaskManager.Task::(prepare) Task=`ec16038a-0bf0-4792-b434-b152d07e91b8`::finished: {} >Thread-1061::DEBUG::2012-12-02 22:49:11,938::task::568::TaskManager.Task::(_updateState) Task=`ec16038a-0bf0-4792-b434-b152d07e91b8`::moving from state preparing -> state finished >Thread-1061::DEBUG::2012-12-02 22:49:11,938::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1061::DEBUG::2012-12-02 22:49:11,938::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1061::DEBUG::2012-12-02 22:49:11,938::task::957::TaskManager.Task::(_decref) Task=`ec16038a-0bf0-4792-b434-b152d07e91b8`::ref 0 aborting False >Thread-1067::DEBUG::2012-12-02 22:49:22,850::task::568::TaskManager.Task::(_updateState) Task=`2f037017-fcd7-4378-b355-db5533a171c9`::moving from state init -> state preparing >Thread-1067::INFO::2012-12-02 22:49:22,850::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1067::INFO::2012-12-02 22:49:22,850::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1067::DEBUG::2012-12-02 22:49:22,850::task::1151::TaskManager.Task::(prepare) Task=`2f037017-fcd7-4378-b355-db5533a171c9`::finished: {} >Thread-1067::DEBUG::2012-12-02 22:49:22,851::task::568::TaskManager.Task::(_updateState) Task=`2f037017-fcd7-4378-b355-db5533a171c9`::moving from state preparing -> state finished >Thread-1067::DEBUG::2012-12-02 22:49:22,851::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1067::DEBUG::2012-12-02 22:49:22,851::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1067::DEBUG::2012-12-02 22:49:22,851::task::957::TaskManager.Task::(_decref) Task=`2f037017-fcd7-4378-b355-db5533a171c9`::ref 0 aborting False >Thread-1073::DEBUG::2012-12-02 22:49:33,835::task::568::TaskManager.Task::(_updateState) Task=`7e6200d0-dcc5-4f43-8e34-144164b6e2ca`::moving from state init -> state preparing >Thread-1073::INFO::2012-12-02 22:49:33,835::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1073::INFO::2012-12-02 22:49:33,836::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1073::DEBUG::2012-12-02 22:49:33,836::task::1151::TaskManager.Task::(prepare) Task=`7e6200d0-dcc5-4f43-8e34-144164b6e2ca`::finished: {} >Thread-1073::DEBUG::2012-12-02 22:49:33,836::task::568::TaskManager.Task::(_updateState) Task=`7e6200d0-dcc5-4f43-8e34-144164b6e2ca`::moving from state preparing -> state finished >Thread-1073::DEBUG::2012-12-02 22:49:33,836::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1073::DEBUG::2012-12-02 22:49:33,836::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1073::DEBUG::2012-12-02 22:49:33,836::task::957::TaskManager.Task::(_decref) Task=`7e6200d0-dcc5-4f43-8e34-144164b6e2ca`::ref 0 aborting False >Thread-1079::DEBUG::2012-12-02 22:49:44,774::task::568::TaskManager.Task::(_updateState) Task=`377d4144-e131-4e84-9ed7-8887e8bd9960`::moving from state init -> state preparing >Thread-1079::INFO::2012-12-02 22:49:44,774::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1079::INFO::2012-12-02 22:49:44,774::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1079::DEBUG::2012-12-02 22:49:44,774::task::1151::TaskManager.Task::(prepare) Task=`377d4144-e131-4e84-9ed7-8887e8bd9960`::finished: {} >Thread-1079::DEBUG::2012-12-02 22:49:44,775::task::568::TaskManager.Task::(_updateState) Task=`377d4144-e131-4e84-9ed7-8887e8bd9960`::moving from state preparing -> state finished >Thread-1079::DEBUG::2012-12-02 22:49:44,775::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1079::DEBUG::2012-12-02 22:49:44,775::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1079::DEBUG::2012-12-02 22:49:44,775::task::957::TaskManager.Task::(_decref) Task=`377d4144-e131-4e84-9ed7-8887e8bd9960`::ref 0 aborting False >Thread-1085::DEBUG::2012-12-02 22:49:55,676::task::568::TaskManager.Task::(_updateState) Task=`b83463c8-284c-4da3-97f0-316a901f3727`::moving from state init -> state preparing >Thread-1085::INFO::2012-12-02 22:49:55,676::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1085::INFO::2012-12-02 22:49:55,676::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1085::DEBUG::2012-12-02 22:49:55,676::task::1151::TaskManager.Task::(prepare) Task=`b83463c8-284c-4da3-97f0-316a901f3727`::finished: {} >Thread-1085::DEBUG::2012-12-02 22:49:55,676::task::568::TaskManager.Task::(_updateState) Task=`b83463c8-284c-4da3-97f0-316a901f3727`::moving from state preparing -> state finished >Thread-1085::DEBUG::2012-12-02 22:49:55,676::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1085::DEBUG::2012-12-02 22:49:55,677::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1085::DEBUG::2012-12-02 22:49:55,677::task::957::TaskManager.Task::(_decref) Task=`b83463c8-284c-4da3-97f0-316a901f3727`::ref 0 aborting False >Thread-1091::DEBUG::2012-12-02 22:50:06,629::task::568::TaskManager.Task::(_updateState) Task=`0142b293-664f-4ccf-8623-71bca2739a53`::moving from state init -> state preparing >Thread-1091::INFO::2012-12-02 22:50:06,630::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1091::INFO::2012-12-02 22:50:06,630::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1091::DEBUG::2012-12-02 22:50:06,630::task::1151::TaskManager.Task::(prepare) Task=`0142b293-664f-4ccf-8623-71bca2739a53`::finished: {} >Thread-1091::DEBUG::2012-12-02 22:50:06,630::task::568::TaskManager.Task::(_updateState) Task=`0142b293-664f-4ccf-8623-71bca2739a53`::moving from state preparing -> state finished >Thread-1091::DEBUG::2012-12-02 22:50:06,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1091::DEBUG::2012-12-02 22:50:06,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1091::DEBUG::2012-12-02 22:50:06,631::task::957::TaskManager.Task::(_decref) Task=`0142b293-664f-4ccf-8623-71bca2739a53`::ref 0 aborting False >Thread-1097::DEBUG::2012-12-02 22:50:17,652::task::568::TaskManager.Task::(_updateState) Task=`3c2787c9-1b36-4733-ba23-5cad4c4fd836`::moving from state init -> state preparing >Thread-1097::INFO::2012-12-02 22:50:17,652::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1097::INFO::2012-12-02 22:50:17,652::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1097::DEBUG::2012-12-02 22:50:17,652::task::1151::TaskManager.Task::(prepare) Task=`3c2787c9-1b36-4733-ba23-5cad4c4fd836`::finished: {} >Thread-1097::DEBUG::2012-12-02 22:50:17,653::task::568::TaskManager.Task::(_updateState) Task=`3c2787c9-1b36-4733-ba23-5cad4c4fd836`::moving from state preparing -> state finished >Thread-1097::DEBUG::2012-12-02 22:50:17,653::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1097::DEBUG::2012-12-02 22:50:17,653::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1097::DEBUG::2012-12-02 22:50:17,653::task::957::TaskManager.Task::(_decref) Task=`3c2787c9-1b36-4733-ba23-5cad4c4fd836`::ref 0 aborting False >Thread-1103::DEBUG::2012-12-02 22:50:28,570::task::568::TaskManager.Task::(_updateState) Task=`26c52151-ffab-430d-857e-7149152f8b46`::moving from state init -> state preparing >Thread-1103::INFO::2012-12-02 22:50:28,570::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1103::INFO::2012-12-02 22:50:28,570::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1103::DEBUG::2012-12-02 22:50:28,570::task::1151::TaskManager.Task::(prepare) Task=`26c52151-ffab-430d-857e-7149152f8b46`::finished: {} >Thread-1103::DEBUG::2012-12-02 22:50:28,570::task::568::TaskManager.Task::(_updateState) Task=`26c52151-ffab-430d-857e-7149152f8b46`::moving from state preparing -> state finished >Thread-1103::DEBUG::2012-12-02 22:50:28,570::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1103::DEBUG::2012-12-02 22:50:28,570::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1103::DEBUG::2012-12-02 22:50:28,571::task::957::TaskManager.Task::(_decref) Task=`26c52151-ffab-430d-857e-7149152f8b46`::ref 0 aborting False >Thread-1109::DEBUG::2012-12-02 22:50:39,490::task::568::TaskManager.Task::(_updateState) Task=`7cd7db69-dc71-4c4d-b262-288236ce1227`::moving from state init -> state preparing >Thread-1109::INFO::2012-12-02 22:50:39,490::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1109::INFO::2012-12-02 22:50:39,490::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1109::DEBUG::2012-12-02 22:50:39,490::task::1151::TaskManager.Task::(prepare) Task=`7cd7db69-dc71-4c4d-b262-288236ce1227`::finished: {} >Thread-1109::DEBUG::2012-12-02 22:50:39,491::task::568::TaskManager.Task::(_updateState) Task=`7cd7db69-dc71-4c4d-b262-288236ce1227`::moving from state preparing -> state finished >Thread-1109::DEBUG::2012-12-02 22:50:39,491::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1109::DEBUG::2012-12-02 22:50:39,491::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1109::DEBUG::2012-12-02 22:50:39,491::task::957::TaskManager.Task::(_decref) Task=`7cd7db69-dc71-4c4d-b262-288236ce1227`::ref 0 aborting False >Thread-1115::DEBUG::2012-12-02 22:50:50,457::task::568::TaskManager.Task::(_updateState) Task=`3e21177f-af0b-4d54-9cdc-79ddb675d983`::moving from state init -> state preparing >Thread-1115::INFO::2012-12-02 22:50:50,457::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1115::INFO::2012-12-02 22:50:50,458::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1115::DEBUG::2012-12-02 22:50:50,458::task::1151::TaskManager.Task::(prepare) Task=`3e21177f-af0b-4d54-9cdc-79ddb675d983`::finished: {} >Thread-1115::DEBUG::2012-12-02 22:50:50,458::task::568::TaskManager.Task::(_updateState) Task=`3e21177f-af0b-4d54-9cdc-79ddb675d983`::moving from state preparing -> state finished >Thread-1115::DEBUG::2012-12-02 22:50:50,458::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1115::DEBUG::2012-12-02 22:50:50,458::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1115::DEBUG::2012-12-02 22:50:50,458::task::957::TaskManager.Task::(_decref) Task=`3e21177f-af0b-4d54-9cdc-79ddb675d983`::ref 0 aborting False >Thread-1121::DEBUG::2012-12-02 22:51:01,463::task::568::TaskManager.Task::(_updateState) Task=`b185fafb-63ad-4bc8-9acb-668d9e49bd7a`::moving from state init -> state preparing >Thread-1121::INFO::2012-12-02 22:51:01,464::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1121::INFO::2012-12-02 22:51:01,464::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1121::DEBUG::2012-12-02 22:51:01,464::task::1151::TaskManager.Task::(prepare) Task=`b185fafb-63ad-4bc8-9acb-668d9e49bd7a`::finished: {} >Thread-1121::DEBUG::2012-12-02 22:51:01,464::task::568::TaskManager.Task::(_updateState) Task=`b185fafb-63ad-4bc8-9acb-668d9e49bd7a`::moving from state preparing -> state finished >Thread-1121::DEBUG::2012-12-02 22:51:01,464::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1121::DEBUG::2012-12-02 22:51:01,464::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1121::DEBUG::2012-12-02 22:51:01,464::task::957::TaskManager.Task::(_decref) Task=`b185fafb-63ad-4bc8-9acb-668d9e49bd7a`::ref 0 aborting False >Thread-1127::DEBUG::2012-12-02 22:51:12,387::task::568::TaskManager.Task::(_updateState) Task=`a364a0f5-1db8-4817-9eba-699e005eec7e`::moving from state init -> state preparing >Thread-1127::INFO::2012-12-02 22:51:12,387::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1127::INFO::2012-12-02 22:51:12,387::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1127::DEBUG::2012-12-02 22:51:12,387::task::1151::TaskManager.Task::(prepare) Task=`a364a0f5-1db8-4817-9eba-699e005eec7e`::finished: {} >Thread-1127::DEBUG::2012-12-02 22:51:12,388::task::568::TaskManager.Task::(_updateState) Task=`a364a0f5-1db8-4817-9eba-699e005eec7e`::moving from state preparing -> state finished >Thread-1127::DEBUG::2012-12-02 22:51:12,388::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1127::DEBUG::2012-12-02 22:51:12,388::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1127::DEBUG::2012-12-02 22:51:12,388::task::957::TaskManager.Task::(_decref) Task=`a364a0f5-1db8-4817-9eba-699e005eec7e`::ref 0 aborting False >Thread-1133::DEBUG::2012-12-02 22:51:23,313::task::568::TaskManager.Task::(_updateState) Task=`b953af2a-887a-49dd-ab08-e907d810b332`::moving from state init -> state preparing >Thread-1133::INFO::2012-12-02 22:51:23,314::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1133::INFO::2012-12-02 22:51:23,314::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1133::DEBUG::2012-12-02 22:51:23,314::task::1151::TaskManager.Task::(prepare) Task=`b953af2a-887a-49dd-ab08-e907d810b332`::finished: {} >Thread-1133::DEBUG::2012-12-02 22:51:23,314::task::568::TaskManager.Task::(_updateState) Task=`b953af2a-887a-49dd-ab08-e907d810b332`::moving from state preparing -> state finished >Thread-1133::DEBUG::2012-12-02 22:51:23,314::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1133::DEBUG::2012-12-02 22:51:23,314::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1133::DEBUG::2012-12-02 22:51:23,314::task::957::TaskManager.Task::(_decref) Task=`b953af2a-887a-49dd-ab08-e907d810b332`::ref 0 aborting False >Thread-1139::DEBUG::2012-12-02 22:51:34,277::task::568::TaskManager.Task::(_updateState) Task=`f9a3a751-47b2-443f-9d93-bee580e42189`::moving from state init -> state preparing >Thread-1139::INFO::2012-12-02 22:51:34,277::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1139::INFO::2012-12-02 22:51:34,277::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1139::DEBUG::2012-12-02 22:51:34,277::task::1151::TaskManager.Task::(prepare) Task=`f9a3a751-47b2-443f-9d93-bee580e42189`::finished: {} >Thread-1139::DEBUG::2012-12-02 22:51:34,277::task::568::TaskManager.Task::(_updateState) Task=`f9a3a751-47b2-443f-9d93-bee580e42189`::moving from state preparing -> state finished >Thread-1139::DEBUG::2012-12-02 22:51:34,278::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1139::DEBUG::2012-12-02 22:51:34,278::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1139::DEBUG::2012-12-02 22:51:34,278::task::957::TaskManager.Task::(_decref) Task=`f9a3a751-47b2-443f-9d93-bee580e42189`::ref 0 aborting False >Thread-1145::DEBUG::2012-12-02 22:51:45,271::task::568::TaskManager.Task::(_updateState) Task=`98e4685f-8123-4750-af93-8715596106cc`::moving from state init -> state preparing >Thread-1145::INFO::2012-12-02 22:51:45,271::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1145::INFO::2012-12-02 22:51:45,271::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1145::DEBUG::2012-12-02 22:51:45,271::task::1151::TaskManager.Task::(prepare) Task=`98e4685f-8123-4750-af93-8715596106cc`::finished: {} >Thread-1145::DEBUG::2012-12-02 22:51:45,271::task::568::TaskManager.Task::(_updateState) Task=`98e4685f-8123-4750-af93-8715596106cc`::moving from state preparing -> state finished >Thread-1145::DEBUG::2012-12-02 22:51:45,272::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1145::DEBUG::2012-12-02 22:51:45,272::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1145::DEBUG::2012-12-02 22:51:45,272::task::957::TaskManager.Task::(_decref) Task=`98e4685f-8123-4750-af93-8715596106cc`::ref 0 aborting False >Thread-1151::DEBUG::2012-12-02 22:51:56,211::task::568::TaskManager.Task::(_updateState) Task=`76906bf8-59db-4aba-8684-5a888583b012`::moving from state init -> state preparing >Thread-1151::INFO::2012-12-02 22:51:56,211::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1151::INFO::2012-12-02 22:51:56,211::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1151::DEBUG::2012-12-02 22:51:56,211::task::1151::TaskManager.Task::(prepare) Task=`76906bf8-59db-4aba-8684-5a888583b012`::finished: {} >Thread-1151::DEBUG::2012-12-02 22:51:56,211::task::568::TaskManager.Task::(_updateState) Task=`76906bf8-59db-4aba-8684-5a888583b012`::moving from state preparing -> state finished >Thread-1151::DEBUG::2012-12-02 22:51:56,211::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1151::DEBUG::2012-12-02 22:51:56,212::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1151::DEBUG::2012-12-02 22:51:56,212::task::957::TaskManager.Task::(_decref) Task=`76906bf8-59db-4aba-8684-5a888583b012`::ref 0 aborting False >Thread-1157::DEBUG::2012-12-02 22:52:07,204::task::568::TaskManager.Task::(_updateState) Task=`6b044f6d-4994-4182-b83b-172af6fd3ad3`::moving from state init -> state preparing >Thread-1157::INFO::2012-12-02 22:52:07,204::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1157::INFO::2012-12-02 22:52:07,204::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1157::DEBUG::2012-12-02 22:52:07,204::task::1151::TaskManager.Task::(prepare) Task=`6b044f6d-4994-4182-b83b-172af6fd3ad3`::finished: {} >Thread-1157::DEBUG::2012-12-02 22:52:07,204::task::568::TaskManager.Task::(_updateState) Task=`6b044f6d-4994-4182-b83b-172af6fd3ad3`::moving from state preparing -> state finished >Thread-1157::DEBUG::2012-12-02 22:52:07,204::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1157::DEBUG::2012-12-02 22:52:07,205::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1157::DEBUG::2012-12-02 22:52:07,205::task::957::TaskManager.Task::(_decref) Task=`6b044f6d-4994-4182-b83b-172af6fd3ad3`::ref 0 aborting False >Thread-1163::DEBUG::2012-12-02 22:52:18,180::task::568::TaskManager.Task::(_updateState) Task=`e81b42ae-56e9-4973-ad46-bb9399e23b13`::moving from state init -> state preparing >Thread-1163::INFO::2012-12-02 22:52:18,180::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1163::INFO::2012-12-02 22:52:18,180::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1163::DEBUG::2012-12-02 22:52:18,180::task::1151::TaskManager.Task::(prepare) Task=`e81b42ae-56e9-4973-ad46-bb9399e23b13`::finished: {} >Thread-1163::DEBUG::2012-12-02 22:52:18,180::task::568::TaskManager.Task::(_updateState) Task=`e81b42ae-56e9-4973-ad46-bb9399e23b13`::moving from state preparing -> state finished >Thread-1163::DEBUG::2012-12-02 22:52:18,180::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1163::DEBUG::2012-12-02 22:52:18,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1163::DEBUG::2012-12-02 22:52:18,181::task::957::TaskManager.Task::(_decref) Task=`e81b42ae-56e9-4973-ad46-bb9399e23b13`::ref 0 aborting False >Thread-1169::DEBUG::2012-12-02 22:52:29,116::task::568::TaskManager.Task::(_updateState) Task=`37ff441a-334e-4545-9870-f1d9efd81826`::moving from state init -> state preparing >Thread-1169::INFO::2012-12-02 22:52:29,116::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1169::INFO::2012-12-02 22:52:29,116::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1169::DEBUG::2012-12-02 22:52:29,116::task::1151::TaskManager.Task::(prepare) Task=`37ff441a-334e-4545-9870-f1d9efd81826`::finished: {} >Thread-1169::DEBUG::2012-12-02 22:52:29,116::task::568::TaskManager.Task::(_updateState) Task=`37ff441a-334e-4545-9870-f1d9efd81826`::moving from state preparing -> state finished >Thread-1169::DEBUG::2012-12-02 22:52:29,116::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1169::DEBUG::2012-12-02 22:52:29,117::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1169::DEBUG::2012-12-02 22:52:29,117::task::957::TaskManager.Task::(_decref) Task=`37ff441a-334e-4545-9870-f1d9efd81826`::ref 0 aborting False >Thread-1173::DEBUG::2012-12-02 22:52:34,428::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1173::DEBUG::2012-12-02 22:52:34,428::task::568::TaskManager.Task::(_updateState) Task=`6caee201-62c4-48a5-920e-2156f31becf8`::moving from state init -> state preparing >Thread-1173::INFO::2012-12-02 22:52:34,428::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e', 'port': ''}], options=None) >Thread-1173::INFO::2012-12-02 22:52:34,428::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e'}]} >Thread-1173::DEBUG::2012-12-02 22:52:34,428::task::1151::TaskManager.Task::(prepare) Task=`6caee201-62c4-48a5-920e-2156f31becf8`::finished: {'statuslist': [{'status': 0, 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e'}]} >Thread-1173::DEBUG::2012-12-02 22:52:34,429::task::568::TaskManager.Task::(_updateState) Task=`6caee201-62c4-48a5-920e-2156f31becf8`::moving from state preparing -> state finished >Thread-1173::DEBUG::2012-12-02 22:52:34,429::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1173::DEBUG::2012-12-02 22:52:34,429::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1173::DEBUG::2012-12-02 22:52:34,429::task::957::TaskManager.Task::(_decref) Task=`6caee201-62c4-48a5-920e-2156f31becf8`::ref 0 aborting False >Thread-1174::DEBUG::2012-12-02 22:52:34,582::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1174::DEBUG::2012-12-02 22:52:34,582::task::568::TaskManager.Task::(_updateState) Task=`1453a8d0-5ac1-4a79-957e-885ed55f91e9`::moving from state init -> state preparing >Thread-1174::INFO::2012-12-02 22:52:34,582::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e', 'port': ''}], options=None) >Thread-1174::DEBUG::2012-12-02 22:52:34,588::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251' (cwd None) >Thread-1174::DEBUG::2012-12-02 22:52:34,705::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1174::DEBUG::2012-12-02 22:52:34,706::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1174::DEBUG::2012-12-02 22:52:34,706::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1174::DEBUG::2012-12-02 22:52:34,706::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1174::DEBUG::2012-12-02 22:52:34,706::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1174::DEBUG::2012-12-02 22:52:34,706::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1174::INFO::2012-12-02 22:52:34,707::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e'}]} >Thread-1174::DEBUG::2012-12-02 22:52:34,707::task::1151::TaskManager.Task::(prepare) Task=`1453a8d0-5ac1-4a79-957e-885ed55f91e9`::finished: {'statuslist': [{'status': 0, 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e'}]} >Thread-1174::DEBUG::2012-12-02 22:52:34,707::task::568::TaskManager.Task::(_updateState) Task=`1453a8d0-5ac1-4a79-957e-885ed55f91e9`::moving from state preparing -> state finished >Thread-1174::DEBUG::2012-12-02 22:52:34,707::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1174::DEBUG::2012-12-02 22:52:34,707::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1174::DEBUG::2012-12-02 22:52:34,707::task::957::TaskManager.Task::(_decref) Task=`1453a8d0-5ac1-4a79-957e-885ed55f91e9`::ref 0 aborting False >Thread-1177::DEBUG::2012-12-02 22:52:40,078::task::568::TaskManager.Task::(_updateState) Task=`a81c2c6f-8067-4c32-afce-3d428b5c4b04`::moving from state init -> state preparing >Thread-1177::INFO::2012-12-02 22:52:40,078::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1177::INFO::2012-12-02 22:52:40,078::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1177::DEBUG::2012-12-02 22:52:40,078::task::1151::TaskManager.Task::(prepare) Task=`a81c2c6f-8067-4c32-afce-3d428b5c4b04`::finished: {} >Thread-1177::DEBUG::2012-12-02 22:52:40,078::task::568::TaskManager.Task::(_updateState) Task=`a81c2c6f-8067-4c32-afce-3d428b5c4b04`::moving from state preparing -> state finished >Thread-1177::DEBUG::2012-12-02 22:52:40,078::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1177::DEBUG::2012-12-02 22:52:40,079::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1177::DEBUG::2012-12-02 22:52:40,079::task::957::TaskManager.Task::(_decref) Task=`a81c2c6f-8067-4c32-afce-3d428b5c4b04`::ref 0 aborting False >Thread-1183::DEBUG::2012-12-02 22:52:51,015::task::568::TaskManager.Task::(_updateState) Task=`f8ec8c3d-c0d6-4207-b280-464b08adaae9`::moving from state init -> state preparing >Thread-1183::INFO::2012-12-02 22:52:51,015::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1183::INFO::2012-12-02 22:52:51,015::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1183::DEBUG::2012-12-02 22:52:51,016::task::1151::TaskManager.Task::(prepare) Task=`f8ec8c3d-c0d6-4207-b280-464b08adaae9`::finished: {} >Thread-1183::DEBUG::2012-12-02 22:52:51,016::task::568::TaskManager.Task::(_updateState) Task=`f8ec8c3d-c0d6-4207-b280-464b08adaae9`::moving from state preparing -> state finished >Thread-1183::DEBUG::2012-12-02 22:52:51,016::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1183::DEBUG::2012-12-02 22:52:51,016::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1183::DEBUG::2012-12-02 22:52:51,016::task::957::TaskManager.Task::(_decref) Task=`f8ec8c3d-c0d6-4207-b280-464b08adaae9`::ref 0 aborting False >Thread-1189::DEBUG::2012-12-02 22:53:00,841::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1189::DEBUG::2012-12-02 22:53:00,841::task::568::TaskManager.Task::(_updateState) Task=`313ad955-13af-4c1f-9bed-1aceac2a6e41`::moving from state init -> state preparing >Thread-1189::INFO::2012-12-02 22:53:00,841::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='2ee486e4-abfd-4f11-9b29-42bfc74d7064', masterVersion=1, options=None) >Thread-1189::DEBUG::2012-12-02 22:53:00,842::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ad029c00-e042-4d3f-be64-86a124284570`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1189::DEBUG::2012-12-02 22:53:00,842::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1189::DEBUG::2012-12-02 22:53:00,842::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1189::DEBUG::2012-12-02 22:53:00,842::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ad029c00-e042-4d3f-be64-86a124284570`::Granted request >Thread-1189::INFO::2012-12-02 22:53:00,842::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 2ee486e4-abfd-4f11-9b29-42bfc74d7064 (ver = 1) >Thread-1189::DEBUG::2012-12-02 22:53:00,843::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:00,843::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:00,843::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:00,843::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:00,844::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:00,844::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:00,844::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1189::DEBUG::2012-12-02 22:53:00,844::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1189::DEBUG::2012-12-02 22:53:00,844::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1189::DEBUG::2012-12-02 22:53:00,844::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1189::DEBUG::2012-12-02 22:53:00,844::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1189::DEBUG::2012-12-02 22:53:00,873::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1189::DEBUG::2012-12-02 22:53:00,873::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1189::DEBUG::2012-12-02 22:53:01,506::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1189::DEBUG::2012-12-02 22:53:01,643::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1189::DEBUG::2012-12-02 22:53:01,643::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,644::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,644::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,644::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,644::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,644::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,644::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1189::DEBUG::2012-12-02 22:53:01,645::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,646::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1189::DEBUG::2012-12-02 22:53:01,869::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1189::WARNING::2012-12-02 22:53:01,870::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1189::DEBUG::2012-12-02 22:53:01,873::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:01,913::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1189::DEBUG::2012-12-02 22:53:01,913::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-1189::DEBUG::2012-12-02 22:53:01,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdf', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'ROLE=Master', 'SDUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d81685bac393bea554677c49f61002a3a775706e'] >Thread-1251::DEBUG::2012-12-02 22:53:01,964::task::568::TaskManager.Task::(_updateState) Task=`846e1bcb-b232-4ff9-b410-1f4987fe452d`::moving from state init -> state preparing >Thread-1251::INFO::2012-12-02 22:53:01,965::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1251::INFO::2012-12-02 22:53:01,965::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1251::DEBUG::2012-12-02 22:53:01,965::task::1151::TaskManager.Task::(prepare) Task=`846e1bcb-b232-4ff9-b410-1f4987fe452d`::finished: {} >Thread-1251::DEBUG::2012-12-02 22:53:01,965::task::568::TaskManager.Task::(_updateState) Task=`846e1bcb-b232-4ff9-b410-1f4987fe452d`::moving from state preparing -> state finished >Thread-1251::DEBUG::2012-12-02 22:53:01,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1251::DEBUG::2012-12-02 22:53:01,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1251::DEBUG::2012-12-02 22:53:01,966::task::957::TaskManager.Task::(_decref) Task=`846e1bcb-b232-4ff9-b410-1f4987fe452d`::ref 0 aborting False >Thread-1189::DEBUG::2012-12-02 22:53:01,996::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-1189::DEBUG::2012-12-02 22:53:01,996::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '2ee486e4-abfd-4f11-9b29-42bfc74d7064_imageNS' >Thread-1189::DEBUG::2012-12-02 22:53:01,996::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '2ee486e4-abfd-4f11-9b29-42bfc74d7064_volumeNS' >Thread-1189::DEBUG::2012-12-02 22:53:01,997::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 2ee486e4-abfd-4f11-9b29-42bfc74d7064 verified, version 1 >Thread-1189::DEBUG::2012-12-02 22:53:01,997::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-1189::DEBUG::2012-12-02 22:53:01,997::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1189::DEBUG::2012-12-02 22:53:02,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdf', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'ROLE=Master', 'SDUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d81685bac393bea554677c49f61002a3a775706e'] >Thread-1189::DEBUG::2012-12-02 22:53:02,005::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `479cf195-bb99-45c5-b400-433d810eaa48` started monitoring domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Thread-1189::DEBUG::2012-12-02 22:53:02,005::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1189::DEBUG::2012-12-02 22:53:02,006::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1189::DEBUG::2012-12-02 22:53:02,007::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free' (cwd None) >Thread-1252::INFO::2012-12-02 22:53:02,032::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 2ee486e4-abfd-4f11-9b29-42bfc74d7064 (id: 2) >Thread-1252::DEBUG::2012-12-02 22:53:02,032::safelease::178::SANLock::(acquireHostId) Host id for domain 2ee486e4-abfd-4f11-9b29-42bfc74d7064 successfully acquired (id: 2) >Thread-1189::DEBUG::2012-12-02 22:53:02,398::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >Thread-1189::DEBUG::2012-12-02 22:53:02,400::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1189::INFO::2012-12-02 22:53:02,429::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1189::INFO::2012-12-02 22:53:02,429::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-1189::DEBUG::2012-12-02 22:53:02,430::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1189::DEBUG::2012-12-02 22:53:02,430::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1189::DEBUG::2012-12-02 22:53:02,430::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1189::DEBUG::2012-12-02 22:53:02,430::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1189::INFO::2012-12-02 22:53:02,430::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-1189::DEBUG::2012-12-02 22:53:02,430::task::1151::TaskManager.Task::(prepare) Task=`313ad955-13af-4c1f-9bed-1aceac2a6e41`::finished: True >Thread-1189::DEBUG::2012-12-02 22:53:02,430::task::568::TaskManager.Task::(_updateState) Task=`313ad955-13af-4c1f-9bed-1aceac2a6e41`::moving from state preparing -> state finished >Thread-1189::DEBUG::2012-12-02 22:53:02,431::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1189::DEBUG::2012-12-02 22:53:02,431::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1189::DEBUG::2012-12-02 22:53:02,431::task::957::TaskManager.Task::(_decref) Task=`313ad955-13af-4c1f-9bed-1aceac2a6e41`::ref 0 aborting False >Thread-1318::DEBUG::2012-12-02 22:53:12,888::task::568::TaskManager.Task::(_updateState) Task=`2e988b35-5872-470e-9b5d-d2661495c9f9`::moving from state init -> state preparing >Thread-1318::INFO::2012-12-02 22:53:12,888::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1318::INFO::2012-12-02 22:53:12,888::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00924801826477', 'lastCheck': 1354481592.0538149, 'code': 0, 'valid': True}} >Thread-1318::DEBUG::2012-12-02 22:53:12,889::task::1151::TaskManager.Task::(prepare) Task=`2e988b35-5872-470e-9b5d-d2661495c9f9`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00924801826477', 'lastCheck': 1354481592.0538149, 'code': 0, 'valid': True}} >Thread-1318::DEBUG::2012-12-02 22:53:12,889::task::568::TaskManager.Task::(_updateState) Task=`2e988b35-5872-470e-9b5d-d2661495c9f9`::moving from state preparing -> state finished >Thread-1318::DEBUG::2012-12-02 22:53:12,889::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1318::DEBUG::2012-12-02 22:53:12,889::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1318::DEBUG::2012-12-02 22:53:12,889::task::957::TaskManager.Task::(_decref) Task=`2e988b35-5872-470e-9b5d-d2661495c9f9`::ref 0 aborting False >Thread-1324::DEBUG::2012-12-02 22:53:23,841::task::568::TaskManager.Task::(_updateState) Task=`ebac902e-e975-4c73-a335-24c8f43788e9`::moving from state init -> state preparing >Thread-1324::INFO::2012-12-02 22:53:23,841::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1324::INFO::2012-12-02 22:53:23,841::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0076220035553', 'lastCheck': 1354481602.073467, 'code': 0, 'valid': True}} >Thread-1324::DEBUG::2012-12-02 22:53:23,841::task::1151::TaskManager.Task::(prepare) Task=`ebac902e-e975-4c73-a335-24c8f43788e9`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0076220035553', 'lastCheck': 1354481602.073467, 'code': 0, 'valid': True}} >Thread-1324::DEBUG::2012-12-02 22:53:23,842::task::568::TaskManager.Task::(_updateState) Task=`ebac902e-e975-4c73-a335-24c8f43788e9`::moving from state preparing -> state finished >Thread-1324::DEBUG::2012-12-02 22:53:23,842::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1324::DEBUG::2012-12-02 22:53:23,842::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1324::DEBUG::2012-12-02 22:53:23,842::task::957::TaskManager.Task::(_decref) Task=`ebac902e-e975-4c73-a335-24c8f43788e9`::ref 0 aborting False >Thread-1330::DEBUG::2012-12-02 22:53:34,823::task::568::TaskManager.Task::(_updateState) Task=`b26daad3-c186-4f32-865e-a84613a98e55`::moving from state init -> state preparing >Thread-1330::INFO::2012-12-02 22:53:34,824::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1330::INFO::2012-12-02 22:53:34,824::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00836896896362', 'lastCheck': 1354481612.09742, 'code': 0, 'valid': True}} >Thread-1330::DEBUG::2012-12-02 22:53:34,824::task::1151::TaskManager.Task::(prepare) Task=`b26daad3-c186-4f32-865e-a84613a98e55`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00836896896362', 'lastCheck': 1354481612.09742, 'code': 0, 'valid': True}} >Thread-1330::DEBUG::2012-12-02 22:53:34,824::task::568::TaskManager.Task::(_updateState) Task=`b26daad3-c186-4f32-865e-a84613a98e55`::moving from state preparing -> state finished >Thread-1330::DEBUG::2012-12-02 22:53:34,824::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1330::DEBUG::2012-12-02 22:53:34,824::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1330::DEBUG::2012-12-02 22:53:34,825::task::957::TaskManager.Task::(_decref) Task=`b26daad3-c186-4f32-865e-a84613a98e55`::ref 0 aborting False >Thread-1336::DEBUG::2012-12-02 22:53:45,857::task::568::TaskManager.Task::(_updateState) Task=`71089f94-2fde-40f8-859b-c8fe9fe0498b`::moving from state init -> state preparing >Thread-1336::INFO::2012-12-02 22:53:45,857::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1336::INFO::2012-12-02 22:53:45,858::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00699305534363', 'lastCheck': 1354481622.1151021, 'code': 0, 'valid': True}} >Thread-1336::DEBUG::2012-12-02 22:53:45,858::task::1151::TaskManager.Task::(prepare) Task=`71089f94-2fde-40f8-859b-c8fe9fe0498b`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00699305534363', 'lastCheck': 1354481622.1151021, 'code': 0, 'valid': True}} >Thread-1336::DEBUG::2012-12-02 22:53:45,858::task::568::TaskManager.Task::(_updateState) Task=`71089f94-2fde-40f8-859b-c8fe9fe0498b`::moving from state preparing -> state finished >Thread-1336::DEBUG::2012-12-02 22:53:45,858::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1336::DEBUG::2012-12-02 22:53:45,858::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1336::DEBUG::2012-12-02 22:53:45,858::task::957::TaskManager.Task::(_decref) Task=`71089f94-2fde-40f8-859b-c8fe9fe0498b`::ref 0 aborting False >Thread-1342::DEBUG::2012-12-02 22:53:56,801::task::568::TaskManager.Task::(_updateState) Task=`6b880c87-a689-4e56-994a-efbceaff1a7e`::moving from state init -> state preparing >Thread-1342::INFO::2012-12-02 22:53:56,801::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1342::INFO::2012-12-02 22:53:56,801::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00748419761658', 'lastCheck': 1354481632.133476, 'code': 0, 'valid': True}} >Thread-1342::DEBUG::2012-12-02 22:53:56,801::task::1151::TaskManager.Task::(prepare) Task=`6b880c87-a689-4e56-994a-efbceaff1a7e`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00748419761658', 'lastCheck': 1354481632.133476, 'code': 0, 'valid': True}} >Thread-1342::DEBUG::2012-12-02 22:53:56,801::task::568::TaskManager.Task::(_updateState) Task=`6b880c87-a689-4e56-994a-efbceaff1a7e`::moving from state preparing -> state finished >Thread-1342::DEBUG::2012-12-02 22:53:56,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1342::DEBUG::2012-12-02 22:53:56,801::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1342::DEBUG::2012-12-02 22:53:56,802::task::957::TaskManager.Task::(_decref) Task=`6b880c87-a689-4e56-994a-efbceaff1a7e`::ref 0 aborting False >Thread-1348::DEBUG::2012-12-02 22:54:07,776::task::568::TaskManager.Task::(_updateState) Task=`2ebbc10a-8f28-4b95-ae3d-354a484b8c94`::moving from state init -> state preparing >Thread-1348::INFO::2012-12-02 22:54:07,776::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1348::INFO::2012-12-02 22:54:07,776::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00928902626038', 'lastCheck': 1354481642.153295, 'code': 0, 'valid': True}} >Thread-1348::DEBUG::2012-12-02 22:54:07,776::task::1151::TaskManager.Task::(prepare) Task=`2ebbc10a-8f28-4b95-ae3d-354a484b8c94`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00928902626038', 'lastCheck': 1354481642.153295, 'code': 0, 'valid': True}} >Thread-1348::DEBUG::2012-12-02 22:54:07,777::task::568::TaskManager.Task::(_updateState) Task=`2ebbc10a-8f28-4b95-ae3d-354a484b8c94`::moving from state preparing -> state finished >Thread-1348::DEBUG::2012-12-02 22:54:07,777::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1348::DEBUG::2012-12-02 22:54:07,777::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1348::DEBUG::2012-12-02 22:54:07,777::task::957::TaskManager.Task::(_decref) Task=`2ebbc10a-8f28-4b95-ae3d-354a484b8c94`::ref 0 aborting False >Thread-1354::DEBUG::2012-12-02 22:54:18,782::task::568::TaskManager.Task::(_updateState) Task=`50483fd0-a1c4-4fa9-8002-bfd866d5c94a`::moving from state init -> state preparing >Thread-1354::INFO::2012-12-02 22:54:18,782::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1354::INFO::2012-12-02 22:54:18,783::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00605988502502', 'lastCheck': 1354481652.168997, 'code': 0, 'valid': True}} >Thread-1354::DEBUG::2012-12-02 22:54:18,783::task::1151::TaskManager.Task::(prepare) Task=`50483fd0-a1c4-4fa9-8002-bfd866d5c94a`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00605988502502', 'lastCheck': 1354481652.168997, 'code': 0, 'valid': True}} >Thread-1354::DEBUG::2012-12-02 22:54:18,783::task::568::TaskManager.Task::(_updateState) Task=`50483fd0-a1c4-4fa9-8002-bfd866d5c94a`::moving from state preparing -> state finished >Thread-1354::DEBUG::2012-12-02 22:54:18,783::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1354::DEBUG::2012-12-02 22:54:18,783::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1354::DEBUG::2012-12-02 22:54:18,783::task::957::TaskManager.Task::(_decref) Task=`50483fd0-a1c4-4fa9-8002-bfd866d5c94a`::ref 0 aborting False >Thread-1360::DEBUG::2012-12-02 22:54:29,683::task::568::TaskManager.Task::(_updateState) Task=`75d7d1c9-4191-4922-9241-a725436a0a09`::moving from state init -> state preparing >Thread-1360::INFO::2012-12-02 22:54:29,684::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1360::INFO::2012-12-02 22:54:29,684::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0090000629425', 'lastCheck': 1354481662.1884, 'code': 0, 'valid': True}} >Thread-1360::DEBUG::2012-12-02 22:54:29,684::task::1151::TaskManager.Task::(prepare) Task=`75d7d1c9-4191-4922-9241-a725436a0a09`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0090000629425', 'lastCheck': 1354481662.1884, 'code': 0, 'valid': True}} >Thread-1360::DEBUG::2012-12-02 22:54:29,684::task::568::TaskManager.Task::(_updateState) Task=`75d7d1c9-4191-4922-9241-a725436a0a09`::moving from state preparing -> state finished >Thread-1360::DEBUG::2012-12-02 22:54:29,684::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1360::DEBUG::2012-12-02 22:54:29,684::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1360::DEBUG::2012-12-02 22:54:29,684::task::957::TaskManager.Task::(_decref) Task=`75d7d1c9-4191-4922-9241-a725436a0a09`::ref 0 aborting False >Thread-1366::DEBUG::2012-12-02 22:54:40,642::task::568::TaskManager.Task::(_updateState) Task=`47786e71-6104-45d4-a1fe-c0e9098ba1de`::moving from state init -> state preparing >Thread-1366::INFO::2012-12-02 22:54:40,642::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1366::INFO::2012-12-02 22:54:40,642::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00463819503784', 'lastCheck': 1354481672.2066779, 'code': 0, 'valid': True}} >Thread-1366::DEBUG::2012-12-02 22:54:40,642::task::1151::TaskManager.Task::(prepare) Task=`47786e71-6104-45d4-a1fe-c0e9098ba1de`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00463819503784', 'lastCheck': 1354481672.2066779, 'code': 0, 'valid': True}} >Thread-1366::DEBUG::2012-12-02 22:54:40,642::task::568::TaskManager.Task::(_updateState) Task=`47786e71-6104-45d4-a1fe-c0e9098ba1de`::moving from state preparing -> state finished >Thread-1366::DEBUG::2012-12-02 22:54:40,643::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1366::DEBUG::2012-12-02 22:54:40,643::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1366::DEBUG::2012-12-02 22:54:40,643::task::957::TaskManager.Task::(_decref) Task=`47786e71-6104-45d4-a1fe-c0e9098ba1de`::ref 0 aborting False >Thread-1372::DEBUG::2012-12-02 22:54:51,592::task::568::TaskManager.Task::(_updateState) Task=`dfb44e6a-623e-4b27-b037-171569ad8956`::moving from state init -> state preparing >Thread-1372::INFO::2012-12-02 22:54:51,592::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1372::INFO::2012-12-02 22:54:51,593::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00739598274231', 'lastCheck': 1354481682.2255919, 'code': 0, 'valid': True}} >Thread-1372::DEBUG::2012-12-02 22:54:51,593::task::1151::TaskManager.Task::(prepare) Task=`dfb44e6a-623e-4b27-b037-171569ad8956`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00739598274231', 'lastCheck': 1354481682.2255919, 'code': 0, 'valid': True}} >Thread-1372::DEBUG::2012-12-02 22:54:51,593::task::568::TaskManager.Task::(_updateState) Task=`dfb44e6a-623e-4b27-b037-171569ad8956`::moving from state preparing -> state finished >Thread-1372::DEBUG::2012-12-02 22:54:51,593::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1372::DEBUG::2012-12-02 22:54:51,593::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1372::DEBUG::2012-12-02 22:54:51,593::task::957::TaskManager.Task::(_decref) Task=`dfb44e6a-623e-4b27-b037-171569ad8956`::ref 0 aborting False >Thread-1378::DEBUG::2012-12-02 22:55:02,537::task::568::TaskManager.Task::(_updateState) Task=`99841ace-ece7-4e2a-8c11-d654baa29cf6`::moving from state init -> state preparing >Thread-1378::INFO::2012-12-02 22:55:02,537::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1378::INFO::2012-12-02 22:55:02,538::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00461196899414', 'lastCheck': 1354481702.258029, 'code': 0, 'valid': True}} >Thread-1378::DEBUG::2012-12-02 22:55:02,538::task::1151::TaskManager.Task::(prepare) Task=`99841ace-ece7-4e2a-8c11-d654baa29cf6`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00461196899414', 'lastCheck': 1354481702.258029, 'code': 0, 'valid': True}} >Thread-1378::DEBUG::2012-12-02 22:55:02,538::task::568::TaskManager.Task::(_updateState) Task=`99841ace-ece7-4e2a-8c11-d654baa29cf6`::moving from state preparing -> state finished >Thread-1378::DEBUG::2012-12-02 22:55:02,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1378::DEBUG::2012-12-02 22:55:02,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1378::DEBUG::2012-12-02 22:55:02,538::task::957::TaskManager.Task::(_decref) Task=`99841ace-ece7-4e2a-8c11-d654baa29cf6`::ref 0 aborting False >Thread-1382::DEBUG::2012-12-02 22:55:07,459::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1382::DEBUG::2012-12-02 22:55:07,460::task::568::TaskManager.Task::(_updateState) Task=`7cef6e70-74dc-47bd-b3bb-47b5346db7ce`::moving from state init -> state preparing >Thread-1382::INFO::2012-12-02 22:55:07,460::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-1382::INFO::2012-12-02 22:55:07,460::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1382::DEBUG::2012-12-02 22:55:07,460::task::1151::TaskManager.Task::(prepare) Task=`7cef6e70-74dc-47bd-b3bb-47b5346db7ce`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1382::DEBUG::2012-12-02 22:55:07,460::task::568::TaskManager.Task::(_updateState) Task=`7cef6e70-74dc-47bd-b3bb-47b5346db7ce`::moving from state preparing -> state finished >Thread-1382::DEBUG::2012-12-02 22:55:07,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1382::DEBUG::2012-12-02 22:55:07,461::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1382::DEBUG::2012-12-02 22:55:07,461::task::957::TaskManager.Task::(_decref) Task=`7cef6e70-74dc-47bd-b3bb-47b5346db7ce`::ref 0 aborting False >Thread-1383::DEBUG::2012-12-02 22:55:07,614::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1383::DEBUG::2012-12-02 22:55:07,614::task::568::TaskManager.Task::(_updateState) Task=`e2f3e7a7-3897-4d9e-bbe7-a76691abb642`::moving from state init -> state preparing >Thread-1383::INFO::2012-12-02 22:55:07,614::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-1383::DEBUG::2012-12-02 22:55:07,619::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252' (cwd None) >Thread-1383::DEBUG::2012-12-02 22:55:07,756::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1383::DEBUG::2012-12-02 22:55:07,757::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1383::DEBUG::2012-12-02 22:55:07,757::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1383::DEBUG::2012-12-02 22:55:07,757::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1383::DEBUG::2012-12-02 22:55:07,757::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1383::DEBUG::2012-12-02 22:55:07,758::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1383::INFO::2012-12-02 22:55:07,758::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1383::DEBUG::2012-12-02 22:55:07,758::task::1151::TaskManager.Task::(prepare) Task=`e2f3e7a7-3897-4d9e-bbe7-a76691abb642`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1383::DEBUG::2012-12-02 22:55:07,758::task::568::TaskManager.Task::(_updateState) Task=`e2f3e7a7-3897-4d9e-bbe7-a76691abb642`::moving from state preparing -> state finished >Thread-1383::DEBUG::2012-12-02 22:55:07,758::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1383::DEBUG::2012-12-02 22:55:07,758::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1383::DEBUG::2012-12-02 22:55:07,759::task::957::TaskManager.Task::(_decref) Task=`e2f3e7a7-3897-4d9e-bbe7-a76691abb642`::ref 0 aborting False >Thread-1386::DEBUG::2012-12-02 22:55:13,457::task::568::TaskManager.Task::(_updateState) Task=`5b236604-29be-47c7-a647-fedf6ea80deb`::moving from state init -> state preparing >Thread-1386::INFO::2012-12-02 22:55:13,458::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1386::INFO::2012-12-02 22:55:13,458::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0065131187439', 'lastCheck': 1354481712.2751911, 'code': 0, 'valid': True}} >Thread-1386::DEBUG::2012-12-02 22:55:13,458::task::1151::TaskManager.Task::(prepare) Task=`5b236604-29be-47c7-a647-fedf6ea80deb`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0065131187439', 'lastCheck': 1354481712.2751911, 'code': 0, 'valid': True}} >Thread-1386::DEBUG::2012-12-02 22:55:13,458::task::568::TaskManager.Task::(_updateState) Task=`5b236604-29be-47c7-a647-fedf6ea80deb`::moving from state preparing -> state finished >Thread-1386::DEBUG::2012-12-02 22:55:13,458::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1386::DEBUG::2012-12-02 22:55:13,458::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1386::DEBUG::2012-12-02 22:55:13,458::task::957::TaskManager.Task::(_decref) Task=`5b236604-29be-47c7-a647-fedf6ea80deb`::ref 0 aborting False >Thread-1392::DEBUG::2012-12-02 22:55:24,369::task::568::TaskManager.Task::(_updateState) Task=`0db02607-8115-4a81-8d4a-9447fb9036e9`::moving from state init -> state preparing >Thread-1392::INFO::2012-12-02 22:55:24,369::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1392::INFO::2012-12-02 22:55:24,370::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0100901126862', 'lastCheck': 1354481722.3173549, 'code': 0, 'valid': True}} >Thread-1392::DEBUG::2012-12-02 22:55:24,370::task::1151::TaskManager.Task::(prepare) Task=`0db02607-8115-4a81-8d4a-9447fb9036e9`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0100901126862', 'lastCheck': 1354481722.3173549, 'code': 0, 'valid': True}} >Thread-1392::DEBUG::2012-12-02 22:55:24,370::task::568::TaskManager.Task::(_updateState) Task=`0db02607-8115-4a81-8d4a-9447fb9036e9`::moving from state preparing -> state finished >Thread-1392::DEBUG::2012-12-02 22:55:24,370::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1392::DEBUG::2012-12-02 22:55:24,370::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1392::DEBUG::2012-12-02 22:55:24,370::task::957::TaskManager.Task::(_decref) Task=`0db02607-8115-4a81-8d4a-9447fb9036e9`::ref 0 aborting False >Thread-1398::DEBUG::2012-12-02 22:55:35,306::task::568::TaskManager.Task::(_updateState) Task=`2dc500c3-362d-4e0b-baeb-1691d169c440`::moving from state init -> state preparing >Thread-1398::INFO::2012-12-02 22:55:35,306::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1398::INFO::2012-12-02 22:55:35,306::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00952696800232', 'lastCheck': 1354481732.341749, 'code': 0, 'valid': True}} >Thread-1398::DEBUG::2012-12-02 22:55:35,307::task::1151::TaskManager.Task::(prepare) Task=`2dc500c3-362d-4e0b-baeb-1691d169c440`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00952696800232', 'lastCheck': 1354481732.341749, 'code': 0, 'valid': True}} >Thread-1398::DEBUG::2012-12-02 22:55:35,307::task::568::TaskManager.Task::(_updateState) Task=`2dc500c3-362d-4e0b-baeb-1691d169c440`::moving from state preparing -> state finished >Thread-1398::DEBUG::2012-12-02 22:55:35,307::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1398::DEBUG::2012-12-02 22:55:35,307::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1398::DEBUG::2012-12-02 22:55:35,307::task::957::TaskManager.Task::(_decref) Task=`2dc500c3-362d-4e0b-baeb-1691d169c440`::ref 0 aborting False >Thread-1404::DEBUG::2012-12-02 22:55:46,237::task::568::TaskManager.Task::(_updateState) Task=`63ac6354-9d56-41bb-b71b-bdfda8dde400`::moving from state init -> state preparing >Thread-1404::INFO::2012-12-02 22:55:46,237::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1404::INFO::2012-12-02 22:55:46,237::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00474691390991', 'lastCheck': 1354481742.356097, 'code': 0, 'valid': True}} >Thread-1404::DEBUG::2012-12-02 22:55:46,237::task::1151::TaskManager.Task::(prepare) Task=`63ac6354-9d56-41bb-b71b-bdfda8dde400`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00474691390991', 'lastCheck': 1354481742.356097, 'code': 0, 'valid': True}} >Thread-1404::DEBUG::2012-12-02 22:55:46,238::task::568::TaskManager.Task::(_updateState) Task=`63ac6354-9d56-41bb-b71b-bdfda8dde400`::moving from state preparing -> state finished >Thread-1404::DEBUG::2012-12-02 22:55:46,238::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1404::DEBUG::2012-12-02 22:55:46,238::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1404::DEBUG::2012-12-02 22:55:46,238::task::957::TaskManager.Task::(_decref) Task=`63ac6354-9d56-41bb-b71b-bdfda8dde400`::ref 0 aborting False >Thread-1410::DEBUG::2012-12-02 22:55:57,234::task::568::TaskManager.Task::(_updateState) Task=`b9d593dd-0c10-4c61-a0ac-94b91cbe117e`::moving from state init -> state preparing >Thread-1410::INFO::2012-12-02 22:55:57,235::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1410::INFO::2012-12-02 22:55:57,235::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00186586380005', 'lastCheck': 1354481752.369288, 'code': 0, 'valid': True}} >Thread-1410::DEBUG::2012-12-02 22:55:57,235::task::1151::TaskManager.Task::(prepare) Task=`b9d593dd-0c10-4c61-a0ac-94b91cbe117e`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00186586380005', 'lastCheck': 1354481752.369288, 'code': 0, 'valid': True}} >Thread-1410::DEBUG::2012-12-02 22:55:57,235::task::568::TaskManager.Task::(_updateState) Task=`b9d593dd-0c10-4c61-a0ac-94b91cbe117e`::moving from state preparing -> state finished >Thread-1410::DEBUG::2012-12-02 22:55:57,235::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1410::DEBUG::2012-12-02 22:55:57,235::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1410::DEBUG::2012-12-02 22:55:57,235::task::957::TaskManager.Task::(_decref) Task=`b9d593dd-0c10-4c61-a0ac-94b91cbe117e`::ref 0 aborting False >Thread-1416::DEBUG::2012-12-02 22:56:08,165::task::568::TaskManager.Task::(_updateState) Task=`9a45e26b-00c3-464d-8667-e63360451148`::moving from state init -> state preparing >Thread-1416::INFO::2012-12-02 22:56:08,165::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1416::INFO::2012-12-02 22:56:08,165::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00340914726257', 'lastCheck': 1354481762.38887, 'code': 0, 'valid': True}} >Thread-1416::DEBUG::2012-12-02 22:56:08,165::task::1151::TaskManager.Task::(prepare) Task=`9a45e26b-00c3-464d-8667-e63360451148`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00340914726257', 'lastCheck': 1354481762.38887, 'code': 0, 'valid': True}} >Thread-1416::DEBUG::2012-12-02 22:56:08,166::task::568::TaskManager.Task::(_updateState) Task=`9a45e26b-00c3-464d-8667-e63360451148`::moving from state preparing -> state finished >Thread-1416::DEBUG::2012-12-02 22:56:08,166::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1416::DEBUG::2012-12-02 22:56:08,166::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1416::DEBUG::2012-12-02 22:56:08,166::task::957::TaskManager.Task::(_decref) Task=`9a45e26b-00c3-464d-8667-e63360451148`::ref 0 aborting False >Thread-1422::DEBUG::2012-12-02 22:56:19,168::task::568::TaskManager.Task::(_updateState) Task=`d0047e8f-1db0-44b0-82cb-e12f1bd9d21c`::moving from state init -> state preparing >Thread-1422::INFO::2012-12-02 22:56:19,169::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1422::INFO::2012-12-02 22:56:19,169::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00466203689575', 'lastCheck': 1354481772.4039321, 'code': 0, 'valid': True}} >Thread-1422::DEBUG::2012-12-02 22:56:19,169::task::1151::TaskManager.Task::(prepare) Task=`d0047e8f-1db0-44b0-82cb-e12f1bd9d21c`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00466203689575', 'lastCheck': 1354481772.4039321, 'code': 0, 'valid': True}} >Thread-1422::DEBUG::2012-12-02 22:56:19,169::task::568::TaskManager.Task::(_updateState) Task=`d0047e8f-1db0-44b0-82cb-e12f1bd9d21c`::moving from state preparing -> state finished >Thread-1422::DEBUG::2012-12-02 22:56:19,169::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1422::DEBUG::2012-12-02 22:56:19,169::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1422::DEBUG::2012-12-02 22:56:19,169::task::957::TaskManager.Task::(_decref) Task=`d0047e8f-1db0-44b0-82cb-e12f1bd9d21c`::ref 0 aborting False >Thread-1428::DEBUG::2012-12-02 22:56:30,094::task::568::TaskManager.Task::(_updateState) Task=`2e4372d8-5cf9-4bca-af12-9a5af9aaec66`::moving from state init -> state preparing >Thread-1428::INFO::2012-12-02 22:56:30,094::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1428::INFO::2012-12-02 22:56:30,094::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00270891189575', 'lastCheck': 1354481782.415416, 'code': 0, 'valid': True}} >Thread-1428::DEBUG::2012-12-02 22:56:30,094::task::1151::TaskManager.Task::(prepare) Task=`2e4372d8-5cf9-4bca-af12-9a5af9aaec66`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00270891189575', 'lastCheck': 1354481782.415416, 'code': 0, 'valid': True}} >Thread-1428::DEBUG::2012-12-02 22:56:30,094::task::568::TaskManager.Task::(_updateState) Task=`2e4372d8-5cf9-4bca-af12-9a5af9aaec66`::moving from state preparing -> state finished >Thread-1428::DEBUG::2012-12-02 22:56:30,095::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1428::DEBUG::2012-12-02 22:56:30,095::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1428::DEBUG::2012-12-02 22:56:30,095::task::957::TaskManager.Task::(_decref) Task=`2e4372d8-5cf9-4bca-af12-9a5af9aaec66`::ref 0 aborting False >Thread-1434::DEBUG::2012-12-02 22:56:39,004::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1434::DEBUG::2012-12-02 22:56:39,004::task::568::TaskManager.Task::(_updateState) Task=`4b0e7be9-8336-4601-a659-73cd1f4df526`::moving from state init -> state preparing >Thread-1434::INFO::2012-12-02 22:56:39,004::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-1434::INFO::2012-12-02 22:56:39,005::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1434::DEBUG::2012-12-02 22:56:39,005::task::1151::TaskManager.Task::(prepare) Task=`4b0e7be9-8336-4601-a659-73cd1f4df526`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1434::DEBUG::2012-12-02 22:56:39,005::task::568::TaskManager.Task::(_updateState) Task=`4b0e7be9-8336-4601-a659-73cd1f4df526`::moving from state preparing -> state finished >Thread-1434::DEBUG::2012-12-02 22:56:39,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1434::DEBUG::2012-12-02 22:56:39,005::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1434::DEBUG::2012-12-02 22:56:39,005::task::957::TaskManager.Task::(_decref) Task=`4b0e7be9-8336-4601-a659-73cd1f4df526`::ref 0 aborting False >Thread-1435::DEBUG::2012-12-02 22:56:39,150::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1435::DEBUG::2012-12-02 22:56:39,150::task::568::TaskManager.Task::(_updateState) Task=`4ed0fd6e-0621-4e7a-9560-1238dd1c02aa`::moving from state init -> state preparing >Thread-1435::INFO::2012-12-02 22:56:39,150::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-1435::DEBUG::2012-12-02 22:56:39,156::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1435::DEBUG::2012-12-02 22:56:39,156::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1435::DEBUG::2012-12-02 22:56:39,156::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1435::DEBUG::2012-12-02 22:56:39,156::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1435::DEBUG::2012-12-02 22:56:39,157::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1435::DEBUG::2012-12-02 22:56:39,157::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1435::INFO::2012-12-02 22:56:39,157::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1435::DEBUG::2012-12-02 22:56:39,157::task::1151::TaskManager.Task::(prepare) Task=`4ed0fd6e-0621-4e7a-9560-1238dd1c02aa`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1435::DEBUG::2012-12-02 22:56:39,157::task::568::TaskManager.Task::(_updateState) Task=`4ed0fd6e-0621-4e7a-9560-1238dd1c02aa`::moving from state preparing -> state finished >Thread-1435::DEBUG::2012-12-02 22:56:39,157::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1435::DEBUG::2012-12-02 22:56:39,157::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1435::DEBUG::2012-12-02 22:56:39,157::task::957::TaskManager.Task::(_decref) Task=`4ed0fd6e-0621-4e7a-9560-1238dd1c02aa`::ref 0 aborting False >Thread-1436::DEBUG::2012-12-02 22:56:39,707::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1436::DEBUG::2012-12-02 22:56:39,708::task::568::TaskManager.Task::(_updateState) Task=`a3f72cad-116b-4ee0-8e41-a4992182dcf3`::moving from state init -> state preparing >Thread-1436::INFO::2012-12-02 22:56:39,708::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='2ee486e4-abfd-4f11-9b29-42bfc74d7064', masterVersion=1, options=None) >Thread-1436::DEBUG::2012-12-02 22:56:39,708::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d566eacd-46f4-43e2-8fbc-587ff6fed061`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1436::DEBUG::2012-12-02 22:56:39,708::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-1436::DEBUG::2012-12-02 22:56:39,709::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-1436::DEBUG::2012-12-02 22:56:39,709::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d566eacd-46f4-43e2-8fbc-587ff6fed061`::Granted request >Thread-1436::DEBUG::2012-12-02 22:56:39,709::task::794::TaskManager.Task::(resourceAcquired) Task=`a3f72cad-116b-4ee0-8e41-a4992182dcf3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-1436::DEBUG::2012-12-02 22:56:39,709::task::957::TaskManager.Task::(_decref) Task=`a3f72cad-116b-4ee0-8e41-a4992182dcf3`::ref 1 aborting False >Thread-1436::DEBUG::2012-12-02 22:56:39,709::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1436::DEBUG::2012-12-02 22:56:39,709::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1436::DEBUG::2012-12-02 22:56:39,709::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1436::DEBUG::2012-12-02 22:56:39,710::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1436::DEBUG::2012-12-02 22:56:39,710::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1436::DEBUG::2012-12-02 22:56:39,735::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1436::DEBUG::2012-12-02 22:56:39,735::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1436::DEBUG::2012-12-02 22:56:40,370::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1436::DEBUG::2012-12-02 22:56:40,468::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1436::DEBUG::2012-12-02 22:56:40,469::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,469::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,469::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,469::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,469::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,470::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,470::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1436::INFO::2012-12-02 22:56:40,470::fileSD::260::Storage.StorageDomain::(validate) sdUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1436::DEBUG::2012-12-02 22:56:40,482::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdf', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'ROLE=Master', 'SDUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d81685bac393bea554677c49f61002a3a775706e'] >Thread-1436::DEBUG::2012-12-02 22:56:40,483::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,483::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,483::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,483::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,483::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,484::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:40,484::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1436::DEBUG::2012-12-02 22:56:40,484::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1436::DEBUG::2012-12-02 22:56:40,484::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1436::DEBUG::2012-12-02 22:56:40,484::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1436::DEBUG::2012-12-02 22:56:40,484::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1436::DEBUG::2012-12-02 22:56:40,509::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1436::DEBUG::2012-12-02 22:56:40,510::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1437::DEBUG::2012-12-02 22:56:41,036::task::568::TaskManager.Task::(_updateState) Task=`a195e63c-e3f7-4c49-b859-0714ce208f6a`::moving from state init -> state preparing >Thread-1437::INFO::2012-12-02 22:56:41,036::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1437::INFO::2012-12-02 22:56:41,036::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00244903564453', 'lastCheck': 1354481792.4330311, 'code': 0, 'valid': True}} >Thread-1437::DEBUG::2012-12-02 22:56:41,037::task::1151::TaskManager.Task::(prepare) Task=`a195e63c-e3f7-4c49-b859-0714ce208f6a`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00244903564453', 'lastCheck': 1354481792.4330311, 'code': 0, 'valid': True}} >Thread-1437::DEBUG::2012-12-02 22:56:41,037::task::568::TaskManager.Task::(_updateState) Task=`a195e63c-e3f7-4c49-b859-0714ce208f6a`::moving from state preparing -> state finished >Thread-1437::DEBUG::2012-12-02 22:56:41,037::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1437::DEBUG::2012-12-02 22:56:41,037::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1437::DEBUG::2012-12-02 22:56:41,037::task::957::TaskManager.Task::(_decref) Task=`a195e63c-e3f7-4c49-b859-0714ce208f6a`::ref 0 aborting False >Thread-1436::DEBUG::2012-12-02 22:56:41,142::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1436::DEBUG::2012-12-02 22:56:41,222::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1436::DEBUG::2012-12-02 22:56:41,223::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,223::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,223::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,223::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,223::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,223::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,223::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1436::DEBUG::2012-12-02 22:56:41,224::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,225::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1436::DEBUG::2012-12-02 22:56:41,444::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1436::WARNING::2012-12-02 22:56:41,446::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1436::DEBUG::2012-12-02 22:56:41,447::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,483::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1436::DEBUG::2012-12-02 22:56:41,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-1436::DEBUG::2012-12-02 22:56:41,494::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdf', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'ROLE=Master', 'SDUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d81685bac393bea554677c49f61002a3a775706e'] >Thread-1436::DEBUG::2012-12-02 22:56:41,496::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-1436::WARNING::2012-12-02 22:56:41,497::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 2ee486e4-abfd-4f11-9b29-42bfc74d7064_imageNS already registered >Thread-1436::WARNING::2012-12-02 22:56:41,497::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 2ee486e4-abfd-4f11-9b29-42bfc74d7064_volumeNS already registered >Thread-1436::DEBUG::2012-12-02 22:56:41,497::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 2ee486e4-abfd-4f11-9b29-42bfc74d7064 verified, version 1 >Thread-1436::DEBUG::2012-12-02 22:56:41,497::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-1436::DEBUG::2012-12-02 22:56:41,497::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1436::DEBUG::2012-12-02 22:56:41,505::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdf', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'ROLE=Master', 'SDUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d81685bac393bea554677c49f61002a3a775706e'] >Thread-1436::DEBUG::2012-12-02 22:56:41,505::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1436::DEBUG::2012-12-02 22:56:41,505::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1436::DEBUG::2012-12-02 22:56:41,506::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free' (cwd None) >Thread-1436::DEBUG::2012-12-02 22:56:41,897::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >Thread-1436::DEBUG::2012-12-02 22:56:41,900::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1436::INFO::2012-12-02 22:56:41,931::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1436::INFO::2012-12-02 22:56:41,931::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-1436::INFO::2012-12-02 22:56:41,931::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-1436::DEBUG::2012-12-02 22:56:41,932::task::1151::TaskManager.Task::(prepare) Task=`a3f72cad-116b-4ee0-8e41-a4992182dcf3`::finished: None >Thread-1436::DEBUG::2012-12-02 22:56:41,932::task::568::TaskManager.Task::(_updateState) Task=`a3f72cad-116b-4ee0-8e41-a4992182dcf3`::moving from state preparing -> state finished >Thread-1436::DEBUG::2012-12-02 22:56:41,932::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-1436::DEBUG::2012-12-02 22:56:41,932::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1436::DEBUG::2012-12-02 22:56:41,932::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1436::DEBUG::2012-12-02 22:56:41,932::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1436::DEBUG::2012-12-02 22:56:41,933::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1436::DEBUG::2012-12-02 22:56:41,933::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1436::DEBUG::2012-12-02 22:56:41,933::task::957::TaskManager.Task::(_decref) Task=`a3f72cad-116b-4ee0-8e41-a4992182dcf3`::ref 0 aborting False >Thread-1562::DEBUG::2012-12-02 22:56:42,082::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1562::DEBUG::2012-12-02 22:56:42,083::task::568::TaskManager.Task::(_updateState) Task=`b0a28f90-7880-4080-9e84-b89cb3c88bc9`::moving from state init -> state preparing >Thread-1562::INFO::2012-12-02 22:56:42,083::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-1562::DEBUG::2012-12-02 22:56:42,083::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252' (cwd None) >Thread-1562::DEBUG::2012-12-02 22:56:42,111::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1562::DEBUG::2012-12-02 22:56:42,112::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1562::DEBUG::2012-12-02 22:56:42,112::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1562::DEBUG::2012-12-02 22:56:42,112::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1562::DEBUG::2012-12-02 22:56:42,112::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1562::DEBUG::2012-12-02 22:56:42,136::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1562::DEBUG::2012-12-02 22:56:42,136::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1562::DEBUG::2012-12-02 22:56:42,771::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1562::DEBUG::2012-12-02 22:56:42,852::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1562::DEBUG::2012-12-02 22:56:42,852::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1562::DEBUG::2012-12-02 22:56:42,852::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1562::DEBUG::2012-12-02 22:56:42,852::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1562::DEBUG::2012-12-02 22:56:42,853::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1562::DEBUG::2012-12-02 22:56:42,853::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1562::DEBUG::2012-12-02 22:56:42,853::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1562::DEBUG::2012-12-02 22:56:42,853::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1562::INFO::2012-12-02 22:56:42,853::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1562::DEBUG::2012-12-02 22:56:42,853::task::1151::TaskManager.Task::(prepare) Task=`b0a28f90-7880-4080-9e84-b89cb3c88bc9`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-1562::DEBUG::2012-12-02 22:56:42,854::task::568::TaskManager.Task::(_updateState) Task=`b0a28f90-7880-4080-9e84-b89cb3c88bc9`::moving from state preparing -> state finished >Thread-1562::DEBUG::2012-12-02 22:56:42,854::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1562::DEBUG::2012-12-02 22:56:42,854::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1562::DEBUG::2012-12-02 22:56:42,854::task::957::TaskManager.Task::(_decref) Task=`b0a28f90-7880-4080-9e84-b89cb3c88bc9`::ref 0 aborting False >Thread-1567::DEBUG::2012-12-02 22:56:51,979::task::568::TaskManager.Task::(_updateState) Task=`231689ad-0f13-4869-ae9e-fb2dcb4732a0`::moving from state init -> state preparing >Thread-1567::INFO::2012-12-02 22:56:51,979::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1567::INFO::2012-12-02 22:56:51,979::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00680804252625', 'lastCheck': 1354481802.449652, 'code': 0, 'valid': True}} >Thread-1567::DEBUG::2012-12-02 22:56:51,979::task::1151::TaskManager.Task::(prepare) Task=`231689ad-0f13-4869-ae9e-fb2dcb4732a0`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00680804252625', 'lastCheck': 1354481802.449652, 'code': 0, 'valid': True}} >Thread-1567::DEBUG::2012-12-02 22:56:51,979::task::568::TaskManager.Task::(_updateState) Task=`231689ad-0f13-4869-ae9e-fb2dcb4732a0`::moving from state preparing -> state finished >Thread-1567::DEBUG::2012-12-02 22:56:51,979::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1567::DEBUG::2012-12-02 22:56:51,980::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1567::DEBUG::2012-12-02 22:56:51,980::task::957::TaskManager.Task::(_decref) Task=`231689ad-0f13-4869-ae9e-fb2dcb4732a0`::ref 0 aborting False >Thread-1573::DEBUG::2012-12-02 22:57:02,959::task::568::TaskManager.Task::(_updateState) Task=`5e99f56d-c56e-46d1-a74e-60c9c0aaa51d`::moving from state init -> state preparing >Thread-1573::INFO::2012-12-02 22:57:02,959::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1573::INFO::2012-12-02 22:57:02,959::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00986814498901', 'lastCheck': 1354481822.4894681, 'code': 0, 'valid': True}} >Thread-1573::DEBUG::2012-12-02 22:57:02,959::task::1151::TaskManager.Task::(prepare) Task=`5e99f56d-c56e-46d1-a74e-60c9c0aaa51d`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00986814498901', 'lastCheck': 1354481822.4894681, 'code': 0, 'valid': True}} >Thread-1573::DEBUG::2012-12-02 22:57:02,960::task::568::TaskManager.Task::(_updateState) Task=`5e99f56d-c56e-46d1-a74e-60c9c0aaa51d`::moving from state preparing -> state finished >Thread-1573::DEBUG::2012-12-02 22:57:02,960::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1573::DEBUG::2012-12-02 22:57:02,960::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1573::DEBUG::2012-12-02 22:57:02,960::task::957::TaskManager.Task::(_decref) Task=`5e99f56d-c56e-46d1-a74e-60c9c0aaa51d`::ref 0 aborting False >Thread-1579::DEBUG::2012-12-02 22:57:13,912::task::568::TaskManager.Task::(_updateState) Task=`0a655462-986b-4382-a442-615f2b709bc4`::moving from state init -> state preparing >Thread-1579::INFO::2012-12-02 22:57:13,912::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1579::INFO::2012-12-02 22:57:13,912::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00935792922974', 'lastCheck': 1354481832.5091059, 'code': 0, 'valid': True}} >Thread-1579::DEBUG::2012-12-02 22:57:13,912::task::1151::TaskManager.Task::(prepare) Task=`0a655462-986b-4382-a442-615f2b709bc4`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00935792922974', 'lastCheck': 1354481832.5091059, 'code': 0, 'valid': True}} >Thread-1579::DEBUG::2012-12-02 22:57:13,912::task::568::TaskManager.Task::(_updateState) Task=`0a655462-986b-4382-a442-615f2b709bc4`::moving from state preparing -> state finished >Thread-1579::DEBUG::2012-12-02 22:57:13,912::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1579::DEBUG::2012-12-02 22:57:13,912::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1579::DEBUG::2012-12-02 22:57:13,913::task::957::TaskManager.Task::(_decref) Task=`0a655462-986b-4382-a442-615f2b709bc4`::ref 0 aborting False >Thread-1585::DEBUG::2012-12-02 22:57:24,887::task::568::TaskManager.Task::(_updateState) Task=`9e5ebfb6-f5e1-4591-b70d-603fbe3e5c44`::moving from state init -> state preparing >Thread-1585::INFO::2012-12-02 22:57:24,887::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1585::INFO::2012-12-02 22:57:24,887::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0099880695343', 'lastCheck': 1354481842.5288811, 'code': 0, 'valid': True}} >Thread-1585::DEBUG::2012-12-02 22:57:24,887::task::1151::TaskManager.Task::(prepare) Task=`9e5ebfb6-f5e1-4591-b70d-603fbe3e5c44`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0099880695343', 'lastCheck': 1354481842.5288811, 'code': 0, 'valid': True}} >Thread-1585::DEBUG::2012-12-02 22:57:24,887::task::568::TaskManager.Task::(_updateState) Task=`9e5ebfb6-f5e1-4591-b70d-603fbe3e5c44`::moving from state preparing -> state finished >Thread-1585::DEBUG::2012-12-02 22:57:24,887::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1585::DEBUG::2012-12-02 22:57:24,887::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1585::DEBUG::2012-12-02 22:57:24,888::task::957::TaskManager.Task::(_decref) Task=`9e5ebfb6-f5e1-4591-b70d-603fbe3e5c44`::ref 0 aborting False >Thread-1591::DEBUG::2012-12-02 22:57:35,845::task::568::TaskManager.Task::(_updateState) Task=`40ab085c-7a2c-4033-bc69-5aba8cdde182`::moving from state init -> state preparing >Thread-1591::INFO::2012-12-02 22:57:35,845::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1591::INFO::2012-12-02 22:57:35,845::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00831413269043', 'lastCheck': 1354481852.5508051, 'code': 0, 'valid': True}} >Thread-1591::DEBUG::2012-12-02 22:57:35,845::task::1151::TaskManager.Task::(prepare) Task=`40ab085c-7a2c-4033-bc69-5aba8cdde182`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00831413269043', 'lastCheck': 1354481852.5508051, 'code': 0, 'valid': True}} >Thread-1591::DEBUG::2012-12-02 22:57:35,845::task::568::TaskManager.Task::(_updateState) Task=`40ab085c-7a2c-4033-bc69-5aba8cdde182`::moving from state preparing -> state finished >Thread-1591::DEBUG::2012-12-02 22:57:35,845::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1591::DEBUG::2012-12-02 22:57:35,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1591::DEBUG::2012-12-02 22:57:35,845::task::957::TaskManager.Task::(_decref) Task=`40ab085c-7a2c-4033-bc69-5aba8cdde182`::ref 0 aborting False >Thread-1597::DEBUG::2012-12-02 22:57:46,825::task::568::TaskManager.Task::(_updateState) Task=`4ec17259-0762-4983-b3b4-23441907df16`::moving from state init -> state preparing >Thread-1597::INFO::2012-12-02 22:57:46,826::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1597::INFO::2012-12-02 22:57:46,826::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00810289382935', 'lastCheck': 1354481862.569376, 'code': 0, 'valid': True}} >Thread-1597::DEBUG::2012-12-02 22:57:46,826::task::1151::TaskManager.Task::(prepare) Task=`4ec17259-0762-4983-b3b4-23441907df16`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00810289382935', 'lastCheck': 1354481862.569376, 'code': 0, 'valid': True}} >Thread-1597::DEBUG::2012-12-02 22:57:46,826::task::568::TaskManager.Task::(_updateState) Task=`4ec17259-0762-4983-b3b4-23441907df16`::moving from state preparing -> state finished >Thread-1597::DEBUG::2012-12-02 22:57:46,826::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1597::DEBUG::2012-12-02 22:57:46,826::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1597::DEBUG::2012-12-02 22:57:46,826::task::957::TaskManager.Task::(_decref) Task=`4ec17259-0762-4983-b3b4-23441907df16`::ref 0 aborting False >Thread-1603::DEBUG::2012-12-02 22:57:57,763::task::568::TaskManager.Task::(_updateState) Task=`7ea8beca-078f-4be8-b040-ccd95fb2d662`::moving from state init -> state preparing >Thread-1603::INFO::2012-12-02 22:57:57,763::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1603::INFO::2012-12-02 22:57:57,764::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00803709030151', 'lastCheck': 1354481872.5870199, 'code': 0, 'valid': True}} >Thread-1603::DEBUG::2012-12-02 22:57:57,764::task::1151::TaskManager.Task::(prepare) Task=`7ea8beca-078f-4be8-b040-ccd95fb2d662`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00803709030151', 'lastCheck': 1354481872.5870199, 'code': 0, 'valid': True}} >Thread-1603::DEBUG::2012-12-02 22:57:57,764::task::568::TaskManager.Task::(_updateState) Task=`7ea8beca-078f-4be8-b040-ccd95fb2d662`::moving from state preparing -> state finished >Thread-1603::DEBUG::2012-12-02 22:57:57,764::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1603::DEBUG::2012-12-02 22:57:57,764::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1603::DEBUG::2012-12-02 22:57:57,764::task::957::TaskManager.Task::(_decref) Task=`7ea8beca-078f-4be8-b040-ccd95fb2d662`::ref 0 aborting False >Thread-1252::DEBUG::2012-12-02 22:58:02,587::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1252::DEBUG::2012-12-02 22:58:02,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1252::DEBUG::2012-12-02 22:58:02,589::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1252::DEBUG::2012-12-02 22:58:02,795::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1252::WARNING::2012-12-02 22:58:02,796::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1252::DEBUG::2012-12-02 22:58:02,796::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1252::DEBUG::2012-12-02 22:58:02,832::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1252::DEBUG::2012-12-02 22:58:02,832::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-1252::DEBUG::2012-12-02 22:58:02,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdf', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'ROLE=Master', 'SDUUID=2ee486e4-abfd-4f11-9b29-42bfc74d7064', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d81685bac393bea554677c49f61002a3a775706e'] >Thread-1252::DEBUG::2012-12-02 22:58:02,844::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-1252::WARNING::2012-12-02 22:58:02,844::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 2ee486e4-abfd-4f11-9b29-42bfc74d7064_imageNS already registered >Thread-1252::WARNING::2012-12-02 22:58:02,844::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 2ee486e4-abfd-4f11-9b29-42bfc74d7064_volumeNS already registered >Thread-1670::DEBUG::2012-12-02 22:58:08,711::task::568::TaskManager.Task::(_updateState) Task=`3280825a-1f3d-4c49-a2ff-67d3e9e2c522`::moving from state init -> state preparing >Thread-1670::INFO::2012-12-02 22:58:08,711::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1670::INFO::2012-12-02 22:58:08,712::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00504279136658', 'lastCheck': 1354481882.8590901, 'code': 0, 'valid': True}} >Thread-1670::DEBUG::2012-12-02 22:58:08,712::task::1151::TaskManager.Task::(prepare) Task=`3280825a-1f3d-4c49-a2ff-67d3e9e2c522`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00504279136658', 'lastCheck': 1354481882.8590901, 'code': 0, 'valid': True}} >Thread-1670::DEBUG::2012-12-02 22:58:08,712::task::568::TaskManager.Task::(_updateState) Task=`3280825a-1f3d-4c49-a2ff-67d3e9e2c522`::moving from state preparing -> state finished >Thread-1670::DEBUG::2012-12-02 22:58:08,712::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1670::DEBUG::2012-12-02 22:58:08,712::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1670::DEBUG::2012-12-02 22:58:08,712::task::957::TaskManager.Task::(_decref) Task=`3280825a-1f3d-4c49-a2ff-67d3e9e2c522`::ref 0 aborting False >Thread-1676::DEBUG::2012-12-02 22:58:19,701::task::568::TaskManager.Task::(_updateState) Task=`d6156eb0-152b-4765-a132-479b1051ee26`::moving from state init -> state preparing >Thread-1676::INFO::2012-12-02 22:58:19,702::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1676::INFO::2012-12-02 22:58:19,702::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00959014892578', 'lastCheck': 1354481892.8789511, 'code': 0, 'valid': True}} >Thread-1676::DEBUG::2012-12-02 22:58:19,702::task::1151::TaskManager.Task::(prepare) Task=`d6156eb0-152b-4765-a132-479b1051ee26`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.00959014892578', 'lastCheck': 1354481892.8789511, 'code': 0, 'valid': True}} >Thread-1676::DEBUG::2012-12-02 22:58:19,702::task::568::TaskManager.Task::(_updateState) Task=`d6156eb0-152b-4765-a132-479b1051ee26`::moving from state preparing -> state finished >Thread-1676::DEBUG::2012-12-02 22:58:19,702::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1676::DEBUG::2012-12-02 22:58:19,702::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1676::DEBUG::2012-12-02 22:58:19,702::task::957::TaskManager.Task::(_decref) Task=`d6156eb0-152b-4765-a132-479b1051ee26`::ref 0 aborting False >Thread-1682::DEBUG::2012-12-02 22:58:30,628::task::568::TaskManager.Task::(_updateState) Task=`659c53d1-6e26-4088-a027-22e8e4d9beef`::moving from state init -> state preparing >Thread-1682::INFO::2012-12-02 22:58:30,628::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1682::INFO::2012-12-02 22:58:30,628::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0219540596008', 'lastCheck': 1354481902.9151151, 'code': 0, 'valid': True}} >Thread-1682::DEBUG::2012-12-02 22:58:30,629::task::1151::TaskManager.Task::(prepare) Task=`659c53d1-6e26-4088-a027-22e8e4d9beef`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0.0219540596008', 'lastCheck': 1354481902.9151151, 'code': 0, 'valid': True}} >Thread-1682::DEBUG::2012-12-02 22:58:30,629::task::568::TaskManager.Task::(_updateState) Task=`659c53d1-6e26-4088-a027-22e8e4d9beef`::moving from state preparing -> state finished >Thread-1682::DEBUG::2012-12-02 22:58:30,629::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1682::DEBUG::2012-12-02 22:58:30,629::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1682::DEBUG::2012-12-02 22:58:30,629::task::957::TaskManager.Task::(_decref) Task=`659c53d1-6e26-4088-a027-22e8e4d9beef`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:58:32,925::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1252::DEBUG::2012-12-02 22:58:32,926::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` changed its status to Invalid >Thread-1688::DEBUG::2012-12-02 22:58:41,561::task::568::TaskManager.Task::(_updateState) Task=`35e25028-292f-4fee-9de7-51a30f5716e8`::moving from state init -> state preparing >Thread-1688::INFO::2012-12-02 22:58:41,561::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1688::INFO::2012-12-02 22:58:41,562::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481912.9261949, 'code': 200, 'valid': False}} >Thread-1688::DEBUG::2012-12-02 22:58:41,562::task::1151::TaskManager.Task::(prepare) Task=`35e25028-292f-4fee-9de7-51a30f5716e8`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481912.9261949, 'code': 200, 'valid': False}} >Thread-1688::DEBUG::2012-12-02 22:58:41,562::task::568::TaskManager.Task::(_updateState) Task=`35e25028-292f-4fee-9de7-51a30f5716e8`::moving from state preparing -> state finished >Thread-1688::DEBUG::2012-12-02 22:58:41,562::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1688::DEBUG::2012-12-02 22:58:41,562::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1688::DEBUG::2012-12-02 22:58:41,562::task::957::TaskManager.Task::(_decref) Task=`35e25028-292f-4fee-9de7-51a30f5716e8`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:58:42,934::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1694::DEBUG::2012-12-02 22:58:52,501::task::568::TaskManager.Task::(_updateState) Task=`f1665dd7-71c9-4f55-8bf6-4d8c7de482c0`::moving from state init -> state preparing >Thread-1694::INFO::2012-12-02 22:58:52,501::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1694::INFO::2012-12-02 22:58:52,501::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481922.9348819, 'code': 200, 'valid': False}} >Thread-1694::DEBUG::2012-12-02 22:58:52,501::task::1151::TaskManager.Task::(prepare) Task=`f1665dd7-71c9-4f55-8bf6-4d8c7de482c0`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481922.9348819, 'code': 200, 'valid': False}} >Thread-1694::DEBUG::2012-12-02 22:58:52,501::task::568::TaskManager.Task::(_updateState) Task=`f1665dd7-71c9-4f55-8bf6-4d8c7de482c0`::moving from state preparing -> state finished >Thread-1694::DEBUG::2012-12-02 22:58:52,501::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1694::DEBUG::2012-12-02 22:58:52,501::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1694::DEBUG::2012-12-02 22:58:52,502::task::957::TaskManager.Task::(_decref) Task=`f1665dd7-71c9-4f55-8bf6-4d8c7de482c0`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:58:52,941::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1252::ERROR::2012-12-02 22:59:02,948::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1700::DEBUG::2012-12-02 22:59:03,459::task::568::TaskManager.Task::(_updateState) Task=`5a4718e8-d357-4825-acd0-54ef6091017f`::moving from state init -> state preparing >Thread-1700::INFO::2012-12-02 22:59:03,460::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1700::INFO::2012-12-02 22:59:03,460::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481942.9483509, 'code': 200, 'valid': False}} >Thread-1700::DEBUG::2012-12-02 22:59:03,460::task::1151::TaskManager.Task::(prepare) Task=`5a4718e8-d357-4825-acd0-54ef6091017f`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481942.9483509, 'code': 200, 'valid': False}} >Thread-1700::DEBUG::2012-12-02 22:59:03,460::task::568::TaskManager.Task::(_updateState) Task=`5a4718e8-d357-4825-acd0-54ef6091017f`::moving from state preparing -> state finished >Thread-1700::DEBUG::2012-12-02 22:59:03,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1700::DEBUG::2012-12-02 22:59:03,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1700::DEBUG::2012-12-02 22:59:03,461::task::957::TaskManager.Task::(_decref) Task=`5a4718e8-d357-4825-acd0-54ef6091017f`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:59:12,958::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1706::DEBUG::2012-12-02 22:59:14,385::task::568::TaskManager.Task::(_updateState) Task=`e41dafa3-37ec-42cc-93a9-9eb36a71c851`::moving from state init -> state preparing >Thread-1706::INFO::2012-12-02 22:59:14,385::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1706::INFO::2012-12-02 22:59:14,385::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481952.9589541, 'code': 200, 'valid': False}} >Thread-1706::DEBUG::2012-12-02 22:59:14,385::task::1151::TaskManager.Task::(prepare) Task=`e41dafa3-37ec-42cc-93a9-9eb36a71c851`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481952.9589541, 'code': 200, 'valid': False}} >Thread-1706::DEBUG::2012-12-02 22:59:14,386::task::568::TaskManager.Task::(_updateState) Task=`e41dafa3-37ec-42cc-93a9-9eb36a71c851`::moving from state preparing -> state finished >Thread-1706::DEBUG::2012-12-02 22:59:14,386::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1706::DEBUG::2012-12-02 22:59:14,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1706::DEBUG::2012-12-02 22:59:14,386::task::957::TaskManager.Task::(_decref) Task=`e41dafa3-37ec-42cc-93a9-9eb36a71c851`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:59:22,965::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1712::DEBUG::2012-12-02 22:59:25,339::task::568::TaskManager.Task::(_updateState) Task=`e0b78f4c-8c20-4a70-860e-3c73201fe39d`::moving from state init -> state preparing >Thread-1712::INFO::2012-12-02 22:59:25,339::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1712::INFO::2012-12-02 22:59:25,339::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481962.965713, 'code': 200, 'valid': False}} >Thread-1712::DEBUG::2012-12-02 22:59:25,339::task::1151::TaskManager.Task::(prepare) Task=`e0b78f4c-8c20-4a70-860e-3c73201fe39d`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481962.965713, 'code': 200, 'valid': False}} >Thread-1712::DEBUG::2012-12-02 22:59:25,340::task::568::TaskManager.Task::(_updateState) Task=`e0b78f4c-8c20-4a70-860e-3c73201fe39d`::moving from state preparing -> state finished >Thread-1712::DEBUG::2012-12-02 22:59:25,340::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1712::DEBUG::2012-12-02 22:59:25,340::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1712::DEBUG::2012-12-02 22:59:25,340::task::957::TaskManager.Task::(_decref) Task=`e0b78f4c-8c20-4a70-860e-3c73201fe39d`::ref 0 aborting False >Thread-1716::DEBUG::2012-12-02 22:59:30,903::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1716::DEBUG::2012-12-02 22:59:30,903::task::568::TaskManager.Task::(_updateState) Task=`33420376-4121-4cf4-868f-7c91caad307a`::moving from state init -> state preparing >Thread-1716::INFO::2012-12-02 22:59:30,903::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1716::INFO::2012-12-02 22:59:30,903::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1716::DEBUG::2012-12-02 22:59:30,904::task::1151::TaskManager.Task::(prepare) Task=`33420376-4121-4cf4-868f-7c91caad307a`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1716::DEBUG::2012-12-02 22:59:30,904::task::568::TaskManager.Task::(_updateState) Task=`33420376-4121-4cf4-868f-7c91caad307a`::moving from state preparing -> state finished >Thread-1716::DEBUG::2012-12-02 22:59:30,904::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1716::DEBUG::2012-12-02 22:59:30,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1716::DEBUG::2012-12-02 22:59:30,904::task::957::TaskManager.Task::(_decref) Task=`33420376-4121-4cf4-868f-7c91caad307a`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:59:32,972::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1719::DEBUG::2012-12-02 22:59:36,328::task::568::TaskManager.Task::(_updateState) Task=`acc2fceb-4fb5-482a-8c36-d076bee1a252`::moving from state init -> state preparing >Thread-1719::INFO::2012-12-02 22:59:36,328::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1719::INFO::2012-12-02 22:59:36,328::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481972.9725111, 'code': 200, 'valid': False}} >Thread-1719::DEBUG::2012-12-02 22:59:36,328::task::1151::TaskManager.Task::(prepare) Task=`acc2fceb-4fb5-482a-8c36-d076bee1a252`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481972.9725111, 'code': 200, 'valid': False}} >Thread-1719::DEBUG::2012-12-02 22:59:36,328::task::568::TaskManager.Task::(_updateState) Task=`acc2fceb-4fb5-482a-8c36-d076bee1a252`::moving from state preparing -> state finished >Thread-1719::DEBUG::2012-12-02 22:59:36,328::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1719::DEBUG::2012-12-02 22:59:36,329::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1719::DEBUG::2012-12-02 22:59:36,329::task::957::TaskManager.Task::(_decref) Task=`acc2fceb-4fb5-482a-8c36-d076bee1a252`::ref 0 aborting False >Thread-1723::DEBUG::2012-12-02 22:59:41,428::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1723::DEBUG::2012-12-02 22:59:41,428::task::568::TaskManager.Task::(_updateState) Task=`c7adf245-d5e2-4625-9ae3-02a7289fe6c4`::moving from state init -> state preparing >Thread-1723::INFO::2012-12-02 22:59:41,428::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1723::INFO::2012-12-02 22:59:41,428::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1723::DEBUG::2012-12-02 22:59:41,428::task::1151::TaskManager.Task::(prepare) Task=`c7adf245-d5e2-4625-9ae3-02a7289fe6c4`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1723::DEBUG::2012-12-02 22:59:41,429::task::568::TaskManager.Task::(_updateState) Task=`c7adf245-d5e2-4625-9ae3-02a7289fe6c4`::moving from state preparing -> state finished >Thread-1723::DEBUG::2012-12-02 22:59:41,429::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1723::DEBUG::2012-12-02 22:59:41,429::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1723::DEBUG::2012-12-02 22:59:41,429::task::957::TaskManager.Task::(_decref) Task=`c7adf245-d5e2-4625-9ae3-02a7289fe6c4`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:59:42,978::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1726::DEBUG::2012-12-02 22:59:47,248::task::568::TaskManager.Task::(_updateState) Task=`3a639c82-f16c-4d1c-84b4-52e3aa0954c8`::moving from state init -> state preparing >Thread-1726::INFO::2012-12-02 22:59:47,248::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1726::INFO::2012-12-02 22:59:47,248::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481982.9784751, 'code': 200, 'valid': False}} >Thread-1726::DEBUG::2012-12-02 22:59:47,248::task::1151::TaskManager.Task::(prepare) Task=`3a639c82-f16c-4d1c-84b4-52e3aa0954c8`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481982.9784751, 'code': 200, 'valid': False}} >Thread-1726::DEBUG::2012-12-02 22:59:47,249::task::568::TaskManager.Task::(_updateState) Task=`3a639c82-f16c-4d1c-84b4-52e3aa0954c8`::moving from state preparing -> state finished >Thread-1726::DEBUG::2012-12-02 22:59:47,249::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1726::DEBUG::2012-12-02 22:59:47,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1726::DEBUG::2012-12-02 22:59:47,249::task::957::TaskManager.Task::(_decref) Task=`3a639c82-f16c-4d1c-84b4-52e3aa0954c8`::ref 0 aborting False >Thread-1730::DEBUG::2012-12-02 22:59:51,888::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1730::DEBUG::2012-12-02 22:59:51,888::task::568::TaskManager.Task::(_updateState) Task=`cbddf83e-7a23-4524-9945-db325df180db`::moving from state init -> state preparing >Thread-1730::INFO::2012-12-02 22:59:51,888::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1730::INFO::2012-12-02 22:59:51,888::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1730::DEBUG::2012-12-02 22:59:51,888::task::1151::TaskManager.Task::(prepare) Task=`cbddf83e-7a23-4524-9945-db325df180db`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1730::DEBUG::2012-12-02 22:59:51,889::task::568::TaskManager.Task::(_updateState) Task=`cbddf83e-7a23-4524-9945-db325df180db`::moving from state preparing -> state finished >Thread-1730::DEBUG::2012-12-02 22:59:51,889::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1730::DEBUG::2012-12-02 22:59:51,889::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1730::DEBUG::2012-12-02 22:59:51,889::task::957::TaskManager.Task::(_decref) Task=`cbddf83e-7a23-4524-9945-db325df180db`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 22:59:52,984::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1733::DEBUG::2012-12-02 22:59:58,170::task::568::TaskManager.Task::(_updateState) Task=`a0dd7e2d-64ec-42e5-9028-82b8c31c384d`::moving from state init -> state preparing >Thread-1733::INFO::2012-12-02 22:59:58,170::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1733::INFO::2012-12-02 22:59:58,170::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481992.985218, 'code': 200, 'valid': False}} >Thread-1733::DEBUG::2012-12-02 22:59:58,170::task::1151::TaskManager.Task::(prepare) Task=`a0dd7e2d-64ec-42e5-9028-82b8c31c384d`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354481992.985218, 'code': 200, 'valid': False}} >Thread-1733::DEBUG::2012-12-02 22:59:58,170::task::568::TaskManager.Task::(_updateState) Task=`a0dd7e2d-64ec-42e5-9028-82b8c31c384d`::moving from state preparing -> state finished >Thread-1733::DEBUG::2012-12-02 22:59:58,170::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1733::DEBUG::2012-12-02 22:59:58,170::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1733::DEBUG::2012-12-02 22:59:58,171::task::957::TaskManager.Task::(_decref) Task=`a0dd7e2d-64ec-42e5-9028-82b8c31c384d`::ref 0 aborting False >Thread-1736::DEBUG::2012-12-02 23:00:02,394::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1736::DEBUG::2012-12-02 23:00:02,395::task::568::TaskManager.Task::(_updateState) Task=`61688f11-a76d-44cb-b107-c95f05cae5f5`::moving from state init -> state preparing >Thread-1736::INFO::2012-12-02 23:00:02,395::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1736::INFO::2012-12-02 23:00:02,395::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1736::DEBUG::2012-12-02 23:00:02,395::task::1151::TaskManager.Task::(prepare) Task=`61688f11-a76d-44cb-b107-c95f05cae5f5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1736::DEBUG::2012-12-02 23:00:02,395::task::568::TaskManager.Task::(_updateState) Task=`61688f11-a76d-44cb-b107-c95f05cae5f5`::moving from state preparing -> state finished >Thread-1736::DEBUG::2012-12-02 23:00:02,395::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1736::DEBUG::2012-12-02 23:00:02,395::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1736::DEBUG::2012-12-02 23:00:02,395::task::957::TaskManager.Task::(_decref) Task=`61688f11-a76d-44cb-b107-c95f05cae5f5`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:00:02,991::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1740::DEBUG::2012-12-02 23:00:09,106::task::568::TaskManager.Task::(_updateState) Task=`7030328d-8733-438e-b2bd-f2fc15013297`::moving from state init -> state preparing >Thread-1740::INFO::2012-12-02 23:00:09,106::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1740::INFO::2012-12-02 23:00:09,107::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482002.9922571, 'code': 200, 'valid': False}} >Thread-1740::DEBUG::2012-12-02 23:00:09,107::task::1151::TaskManager.Task::(prepare) Task=`7030328d-8733-438e-b2bd-f2fc15013297`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482002.9922571, 'code': 200, 'valid': False}} >Thread-1740::DEBUG::2012-12-02 23:00:09,107::task::568::TaskManager.Task::(_updateState) Task=`7030328d-8733-438e-b2bd-f2fc15013297`::moving from state preparing -> state finished >Thread-1740::DEBUG::2012-12-02 23:00:09,107::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1740::DEBUG::2012-12-02 23:00:09,107::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1740::DEBUG::2012-12-02 23:00:09,107::task::957::TaskManager.Task::(_decref) Task=`7030328d-8733-438e-b2bd-f2fc15013297`::ref 0 aborting False >Thread-1743::DEBUG::2012-12-02 23:00:12,901::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1743::DEBUG::2012-12-02 23:00:12,902::task::568::TaskManager.Task::(_updateState) Task=`a6b366dd-a2ac-42df-8bd1-574ad565236e`::moving from state init -> state preparing >Thread-1743::INFO::2012-12-02 23:00:12,902::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1743::INFO::2012-12-02 23:00:12,902::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1743::DEBUG::2012-12-02 23:00:12,902::task::1151::TaskManager.Task::(prepare) Task=`a6b366dd-a2ac-42df-8bd1-574ad565236e`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1743::DEBUG::2012-12-02 23:00:12,902::task::568::TaskManager.Task::(_updateState) Task=`a6b366dd-a2ac-42df-8bd1-574ad565236e`::moving from state preparing -> state finished >Thread-1743::DEBUG::2012-12-02 23:00:12,903::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1743::DEBUG::2012-12-02 23:00:12,903::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1743::DEBUG::2012-12-02 23:00:12,903::task::957::TaskManager.Task::(_decref) Task=`a6b366dd-a2ac-42df-8bd1-574ad565236e`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:00:12,999::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1747::DEBUG::2012-12-02 23:00:20,067::task::568::TaskManager.Task::(_updateState) Task=`7ca9a521-e7a1-4404-a7f7-22e590bedc07`::moving from state init -> state preparing >Thread-1747::INFO::2012-12-02 23:00:20,067::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1747::INFO::2012-12-02 23:00:20,067::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482012.999759, 'code': 200, 'valid': False}} >Thread-1747::DEBUG::2012-12-02 23:00:20,067::task::1151::TaskManager.Task::(prepare) Task=`7ca9a521-e7a1-4404-a7f7-22e590bedc07`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482012.999759, 'code': 200, 'valid': False}} >Thread-1747::DEBUG::2012-12-02 23:00:20,067::task::568::TaskManager.Task::(_updateState) Task=`7ca9a521-e7a1-4404-a7f7-22e590bedc07`::moving from state preparing -> state finished >Thread-1747::DEBUG::2012-12-02 23:00:20,067::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1747::DEBUG::2012-12-02 23:00:20,067::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1747::DEBUG::2012-12-02 23:00:20,068::task::957::TaskManager.Task::(_decref) Task=`7ca9a521-e7a1-4404-a7f7-22e590bedc07`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:00:23,006::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1750::DEBUG::2012-12-02 23:00:23,346::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1750::DEBUG::2012-12-02 23:00:23,346::task::568::TaskManager.Task::(_updateState) Task=`756a50dd-e4d5-4b55-989f-a8810cc53f5c`::moving from state init -> state preparing >Thread-1750::INFO::2012-12-02 23:00:23,347::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1750::INFO::2012-12-02 23:00:23,347::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1750::DEBUG::2012-12-02 23:00:23,347::task::1151::TaskManager.Task::(prepare) Task=`756a50dd-e4d5-4b55-989f-a8810cc53f5c`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1750::DEBUG::2012-12-02 23:00:23,347::task::568::TaskManager.Task::(_updateState) Task=`756a50dd-e4d5-4b55-989f-a8810cc53f5c`::moving from state preparing -> state finished >Thread-1750::DEBUG::2012-12-02 23:00:23,347::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1750::DEBUG::2012-12-02 23:00:23,347::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1750::DEBUG::2012-12-02 23:00:23,347::task::957::TaskManager.Task::(_decref) Task=`756a50dd-e4d5-4b55-989f-a8810cc53f5c`::ref 0 aborting False >Thread-1754::DEBUG::2012-12-02 23:00:31,040::task::568::TaskManager.Task::(_updateState) Task=`82827efd-a3f9-420e-9cf7-73334b5d1907`::moving from state init -> state preparing >Thread-1754::INFO::2012-12-02 23:00:31,040::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1754::INFO::2012-12-02 23:00:31,040::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482023.0064991, 'code': 200, 'valid': False}} >Thread-1754::DEBUG::2012-12-02 23:00:31,041::task::1151::TaskManager.Task::(prepare) Task=`82827efd-a3f9-420e-9cf7-73334b5d1907`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482023.0064991, 'code': 200, 'valid': False}} >Thread-1754::DEBUG::2012-12-02 23:00:31,041::task::568::TaskManager.Task::(_updateState) Task=`82827efd-a3f9-420e-9cf7-73334b5d1907`::moving from state preparing -> state finished >Thread-1754::DEBUG::2012-12-02 23:00:31,041::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1754::DEBUG::2012-12-02 23:00:31,041::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1754::DEBUG::2012-12-02 23:00:31,041::task::957::TaskManager.Task::(_decref) Task=`82827efd-a3f9-420e-9cf7-73334b5d1907`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:00:33,012::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1757::DEBUG::2012-12-02 23:00:33,816::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1757::DEBUG::2012-12-02 23:00:33,816::task::568::TaskManager.Task::(_updateState) Task=`693b45df-b461-466c-b3c6-73823796a873`::moving from state init -> state preparing >Thread-1757::INFO::2012-12-02 23:00:33,816::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1757::INFO::2012-12-02 23:00:33,817::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1757::DEBUG::2012-12-02 23:00:33,817::task::1151::TaskManager.Task::(prepare) Task=`693b45df-b461-466c-b3c6-73823796a873`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1757::DEBUG::2012-12-02 23:00:33,817::task::568::TaskManager.Task::(_updateState) Task=`693b45df-b461-466c-b3c6-73823796a873`::moving from state preparing -> state finished >Thread-1757::DEBUG::2012-12-02 23:00:33,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1757::DEBUG::2012-12-02 23:00:33,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1757::DEBUG::2012-12-02 23:00:33,817::task::957::TaskManager.Task::(_decref) Task=`693b45df-b461-466c-b3c6-73823796a873`::ref 0 aborting False >Thread-1761::DEBUG::2012-12-02 23:00:42,039::task::568::TaskManager.Task::(_updateState) Task=`fce4ae4f-c288-4cc1-86ce-e68f74ce7e23`::moving from state init -> state preparing >Thread-1761::INFO::2012-12-02 23:00:42,039::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1761::INFO::2012-12-02 23:00:42,039::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482033.0132351, 'code': 200, 'valid': False}} >Thread-1761::DEBUG::2012-12-02 23:00:42,039::task::1151::TaskManager.Task::(prepare) Task=`fce4ae4f-c288-4cc1-86ce-e68f74ce7e23`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482033.0132351, 'code': 200, 'valid': False}} >Thread-1761::DEBUG::2012-12-02 23:00:42,039::task::568::TaskManager.Task::(_updateState) Task=`fce4ae4f-c288-4cc1-86ce-e68f74ce7e23`::moving from state preparing -> state finished >Thread-1761::DEBUG::2012-12-02 23:00:42,039::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1761::DEBUG::2012-12-02 23:00:42,039::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1761::DEBUG::2012-12-02 23:00:42,040::task::957::TaskManager.Task::(_decref) Task=`fce4ae4f-c288-4cc1-86ce-e68f74ce7e23`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:00:43,018::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1763::DEBUG::2012-12-02 23:00:44,263::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1763::DEBUG::2012-12-02 23:00:44,264::task::568::TaskManager.Task::(_updateState) Task=`26d1756a-7c22-4ecc-b062-6fb8530b9aa3`::moving from state init -> state preparing >Thread-1763::INFO::2012-12-02 23:00:44,264::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1763::INFO::2012-12-02 23:00:44,264::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1763::DEBUG::2012-12-02 23:00:44,264::task::1151::TaskManager.Task::(prepare) Task=`26d1756a-7c22-4ecc-b062-6fb8530b9aa3`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1763::DEBUG::2012-12-02 23:00:44,264::task::568::TaskManager.Task::(_updateState) Task=`26d1756a-7c22-4ecc-b062-6fb8530b9aa3`::moving from state preparing -> state finished >Thread-1763::DEBUG::2012-12-02 23:00:44,264::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1763::DEBUG::2012-12-02 23:00:44,265::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1763::DEBUG::2012-12-02 23:00:44,265::task::957::TaskManager.Task::(_decref) Task=`26d1756a-7c22-4ecc-b062-6fb8530b9aa3`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:00:53,024::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1768::DEBUG::2012-12-02 23:00:53,097::task::568::TaskManager.Task::(_updateState) Task=`b690a7cf-b0dc-48d2-abb8-ea38ea343e73`::moving from state init -> state preparing >Thread-1768::INFO::2012-12-02 23:00:53,097::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1768::INFO::2012-12-02 23:00:53,097::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482053.025245, 'code': 200, 'valid': False}} >Thread-1768::DEBUG::2012-12-02 23:00:53,097::task::1151::TaskManager.Task::(prepare) Task=`b690a7cf-b0dc-48d2-abb8-ea38ea343e73`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482053.025245, 'code': 200, 'valid': False}} >Thread-1768::DEBUG::2012-12-02 23:00:53,097::task::568::TaskManager.Task::(_updateState) Task=`b690a7cf-b0dc-48d2-abb8-ea38ea343e73`::moving from state preparing -> state finished >Thread-1768::DEBUG::2012-12-02 23:00:53,097::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1768::DEBUG::2012-12-02 23:00:53,097::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1768::DEBUG::2012-12-02 23:00:53,098::task::957::TaskManager.Task::(_decref) Task=`b690a7cf-b0dc-48d2-abb8-ea38ea343e73`::ref 0 aborting False >Thread-1770::DEBUG::2012-12-02 23:00:54,749::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1770::DEBUG::2012-12-02 23:00:54,749::task::568::TaskManager.Task::(_updateState) Task=`124ccb0c-4dae-438c-90c7-cc9616df5dc0`::moving from state init -> state preparing >Thread-1770::INFO::2012-12-02 23:00:54,749::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1770::INFO::2012-12-02 23:00:54,749::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1770::DEBUG::2012-12-02 23:00:54,750::task::1151::TaskManager.Task::(prepare) Task=`124ccb0c-4dae-438c-90c7-cc9616df5dc0`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1770::DEBUG::2012-12-02 23:00:54,750::task::568::TaskManager.Task::(_updateState) Task=`124ccb0c-4dae-438c-90c7-cc9616df5dc0`::moving from state preparing -> state finished >Thread-1770::DEBUG::2012-12-02 23:00:54,750::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1770::DEBUG::2012-12-02 23:00:54,750::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1770::DEBUG::2012-12-02 23:00:54,750::task::957::TaskManager.Task::(_decref) Task=`124ccb0c-4dae-438c-90c7-cc9616df5dc0`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:01:03,032::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1775::DEBUG::2012-12-02 23:01:04,041::task::568::TaskManager.Task::(_updateState) Task=`e388039c-b910-49c5-83f6-7466e6ff8721`::moving from state init -> state preparing >Thread-1775::INFO::2012-12-02 23:01:04,044::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1775::INFO::2012-12-02 23:01:04,044::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482063.0360329, 'code': 200, 'valid': False}} >Thread-1775::DEBUG::2012-12-02 23:01:04,044::task::1151::TaskManager.Task::(prepare) Task=`e388039c-b910-49c5-83f6-7466e6ff8721`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482063.0360329, 'code': 200, 'valid': False}} >Thread-1775::DEBUG::2012-12-02 23:01:04,044::task::568::TaskManager.Task::(_updateState) Task=`e388039c-b910-49c5-83f6-7466e6ff8721`::moving from state preparing -> state finished >Thread-1775::DEBUG::2012-12-02 23:01:04,044::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1775::DEBUG::2012-12-02 23:01:04,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1775::DEBUG::2012-12-02 23:01:04,045::task::957::TaskManager.Task::(_decref) Task=`e388039c-b910-49c5-83f6-7466e6ff8721`::ref 0 aborting False >Thread-1777::DEBUG::2012-12-02 23:01:05,208::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1777::DEBUG::2012-12-02 23:01:05,209::task::568::TaskManager.Task::(_updateState) Task=`85196025-6343-489f-b1b9-2d22dda61815`::moving from state init -> state preparing >Thread-1777::INFO::2012-12-02 23:01:05,209::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1777::INFO::2012-12-02 23:01:05,209::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1777::DEBUG::2012-12-02 23:01:05,209::task::1151::TaskManager.Task::(prepare) Task=`85196025-6343-489f-b1b9-2d22dda61815`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1777::DEBUG::2012-12-02 23:01:05,209::task::568::TaskManager.Task::(_updateState) Task=`85196025-6343-489f-b1b9-2d22dda61815`::moving from state preparing -> state finished >Thread-1777::DEBUG::2012-12-02 23:01:05,210::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1777::DEBUG::2012-12-02 23:01:05,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1777::DEBUG::2012-12-02 23:01:05,210::task::957::TaskManager.Task::(_decref) Task=`85196025-6343-489f-b1b9-2d22dda61815`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:01:13,043::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1782::DEBUG::2012-12-02 23:01:14,976::task::568::TaskManager.Task::(_updateState) Task=`58ad2d56-6695-4a75-980d-95bb80205a69`::moving from state init -> state preparing >Thread-1782::INFO::2012-12-02 23:01:14,976::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1782::INFO::2012-12-02 23:01:14,976::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482073.0441101, 'code': 200, 'valid': False}} >Thread-1782::DEBUG::2012-12-02 23:01:14,976::task::1151::TaskManager.Task::(prepare) Task=`58ad2d56-6695-4a75-980d-95bb80205a69`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482073.0441101, 'code': 200, 'valid': False}} >Thread-1782::DEBUG::2012-12-02 23:01:14,976::task::568::TaskManager.Task::(_updateState) Task=`58ad2d56-6695-4a75-980d-95bb80205a69`::moving from state preparing -> state finished >Thread-1782::DEBUG::2012-12-02 23:01:14,976::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1782::DEBUG::2012-12-02 23:01:14,977::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1782::DEBUG::2012-12-02 23:01:14,977::task::957::TaskManager.Task::(_decref) Task=`58ad2d56-6695-4a75-980d-95bb80205a69`::ref 0 aborting False >Thread-1784::DEBUG::2012-12-02 23:01:15,685::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1784::DEBUG::2012-12-02 23:01:15,686::task::568::TaskManager.Task::(_updateState) Task=`ce2ef35b-3dee-4700-932c-cf393dbf2904`::moving from state init -> state preparing >Thread-1784::INFO::2012-12-02 23:01:15,686::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1784::INFO::2012-12-02 23:01:15,686::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1784::DEBUG::2012-12-02 23:01:15,686::task::1151::TaskManager.Task::(prepare) Task=`ce2ef35b-3dee-4700-932c-cf393dbf2904`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1784::DEBUG::2012-12-02 23:01:15,686::task::568::TaskManager.Task::(_updateState) Task=`ce2ef35b-3dee-4700-932c-cf393dbf2904`::moving from state preparing -> state finished >Thread-1784::DEBUG::2012-12-02 23:01:15,686::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1784::DEBUG::2012-12-02 23:01:15,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1784::DEBUG::2012-12-02 23:01:15,687::task::957::TaskManager.Task::(_decref) Task=`ce2ef35b-3dee-4700-932c-cf393dbf2904`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:01:23,048::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1789::DEBUG::2012-12-02 23:01:25,907::task::568::TaskManager.Task::(_updateState) Task=`22d9976c-a401-473c-befa-8c09bdf3a50f`::moving from state init -> state preparing >Thread-1789::INFO::2012-12-02 23:01:25,907::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1789::INFO::2012-12-02 23:01:25,907::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482083.0491791, 'code': 200, 'valid': False}} >Thread-1789::DEBUG::2012-12-02 23:01:25,907::task::1151::TaskManager.Task::(prepare) Task=`22d9976c-a401-473c-befa-8c09bdf3a50f`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482083.0491791, 'code': 200, 'valid': False}} >Thread-1789::DEBUG::2012-12-02 23:01:25,907::task::568::TaskManager.Task::(_updateState) Task=`22d9976c-a401-473c-befa-8c09bdf3a50f`::moving from state preparing -> state finished >Thread-1789::DEBUG::2012-12-02 23:01:25,907::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1789::DEBUG::2012-12-02 23:01:25,908::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1789::DEBUG::2012-12-02 23:01:25,908::task::957::TaskManager.Task::(_decref) Task=`22d9976c-a401-473c-befa-8c09bdf3a50f`::ref 0 aborting False >Thread-1791::DEBUG::2012-12-02 23:01:26,174::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1791::DEBUG::2012-12-02 23:01:26,175::task::568::TaskManager.Task::(_updateState) Task=`7115ef2d-7f60-4258-8387-6a9e6acf55bf`::moving from state init -> state preparing >Thread-1791::INFO::2012-12-02 23:01:26,175::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1791::INFO::2012-12-02 23:01:26,175::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1791::DEBUG::2012-12-02 23:01:26,175::task::1151::TaskManager.Task::(prepare) Task=`7115ef2d-7f60-4258-8387-6a9e6acf55bf`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1791::DEBUG::2012-12-02 23:01:26,176::task::568::TaskManager.Task::(_updateState) Task=`7115ef2d-7f60-4258-8387-6a9e6acf55bf`::moving from state preparing -> state finished >Thread-1791::DEBUG::2012-12-02 23:01:26,176::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1791::DEBUG::2012-12-02 23:01:26,176::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1791::DEBUG::2012-12-02 23:01:26,176::task::957::TaskManager.Task::(_decref) Task=`7115ef2d-7f60-4258-8387-6a9e6acf55bf`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:01:33,055::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1796::DEBUG::2012-12-02 23:01:36,657::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1796::DEBUG::2012-12-02 23:01:36,658::task::568::TaskManager.Task::(_updateState) Task=`11f1200c-f27c-4bc4-92c2-802f0a48e0fb`::moving from state init -> state preparing >Thread-1796::INFO::2012-12-02 23:01:36,658::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1796::INFO::2012-12-02 23:01:36,658::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1796::DEBUG::2012-12-02 23:01:36,658::task::1151::TaskManager.Task::(prepare) Task=`11f1200c-f27c-4bc4-92c2-802f0a48e0fb`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1796::DEBUG::2012-12-02 23:01:36,658::task::568::TaskManager.Task::(_updateState) Task=`11f1200c-f27c-4bc4-92c2-802f0a48e0fb`::moving from state preparing -> state finished >Thread-1796::DEBUG::2012-12-02 23:01:36,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1796::DEBUG::2012-12-02 23:01:36,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1796::DEBUG::2012-12-02 23:01:36,659::task::957::TaskManager.Task::(_decref) Task=`11f1200c-f27c-4bc4-92c2-802f0a48e0fb`::ref 0 aborting False >Thread-1797::DEBUG::2012-12-02 23:01:36,851::task::568::TaskManager.Task::(_updateState) Task=`983a3d2d-ed1a-4d88-88f1-810a6e8344ac`::moving from state init -> state preparing >Thread-1797::INFO::2012-12-02 23:01:36,851::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1797::INFO::2012-12-02 23:01:36,851::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482093.0558131, 'code': 200, 'valid': False}} >Thread-1797::DEBUG::2012-12-02 23:01:36,851::task::1151::TaskManager.Task::(prepare) Task=`983a3d2d-ed1a-4d88-88f1-810a6e8344ac`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482093.0558131, 'code': 200, 'valid': False}} >Thread-1797::DEBUG::2012-12-02 23:01:36,852::task::568::TaskManager.Task::(_updateState) Task=`983a3d2d-ed1a-4d88-88f1-810a6e8344ac`::moving from state preparing -> state finished >Thread-1797::DEBUG::2012-12-02 23:01:36,852::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1797::DEBUG::2012-12-02 23:01:36,852::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1797::DEBUG::2012-12-02 23:01:36,852::task::957::TaskManager.Task::(_decref) Task=`983a3d2d-ed1a-4d88-88f1-810a6e8344ac`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:01:43,062::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1803::DEBUG::2012-12-02 23:01:47,139::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1803::DEBUG::2012-12-02 23:01:47,140::task::568::TaskManager.Task::(_updateState) Task=`ddaecdbc-0bb0-426d-8801-d0c52392c97b`::moving from state init -> state preparing >Thread-1803::INFO::2012-12-02 23:01:47,140::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1803::INFO::2012-12-02 23:01:47,140::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1803::DEBUG::2012-12-02 23:01:47,140::task::1151::TaskManager.Task::(prepare) Task=`ddaecdbc-0bb0-426d-8801-d0c52392c97b`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1803::DEBUG::2012-12-02 23:01:47,140::task::568::TaskManager.Task::(_updateState) Task=`ddaecdbc-0bb0-426d-8801-d0c52392c97b`::moving from state preparing -> state finished >Thread-1803::DEBUG::2012-12-02 23:01:47,140::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1803::DEBUG::2012-12-02 23:01:47,141::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1803::DEBUG::2012-12-02 23:01:47,141::task::957::TaskManager.Task::(_decref) Task=`ddaecdbc-0bb0-426d-8801-d0c52392c97b`::ref 0 aborting False >Thread-1804::DEBUG::2012-12-02 23:01:47,787::task::568::TaskManager.Task::(_updateState) Task=`098de472-6f63-4bc6-a11f-58b158e4233c`::moving from state init -> state preparing >Thread-1804::INFO::2012-12-02 23:01:47,787::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1804::INFO::2012-12-02 23:01:47,787::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482103.0624919, 'code': 200, 'valid': False}} >Thread-1804::DEBUG::2012-12-02 23:01:47,787::task::1151::TaskManager.Task::(prepare) Task=`098de472-6f63-4bc6-a11f-58b158e4233c`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482103.0624919, 'code': 200, 'valid': False}} >Thread-1804::DEBUG::2012-12-02 23:01:47,787::task::568::TaskManager.Task::(_updateState) Task=`098de472-6f63-4bc6-a11f-58b158e4233c`::moving from state preparing -> state finished >Thread-1804::DEBUG::2012-12-02 23:01:47,787::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1804::DEBUG::2012-12-02 23:01:47,788::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1804::DEBUG::2012-12-02 23:01:47,788::task::957::TaskManager.Task::(_decref) Task=`098de472-6f63-4bc6-a11f-58b158e4233c`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:01:53,067::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1810::DEBUG::2012-12-02 23:01:57,677::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1810::DEBUG::2012-12-02 23:01:57,678::task::568::TaskManager.Task::(_updateState) Task=`7a2b0b4a-021a-4206-86a6-845f3b10f1e1`::moving from state init -> state preparing >Thread-1810::INFO::2012-12-02 23:01:57,678::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1810::INFO::2012-12-02 23:01:57,678::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1810::DEBUG::2012-12-02 23:01:57,678::task::1151::TaskManager.Task::(prepare) Task=`7a2b0b4a-021a-4206-86a6-845f3b10f1e1`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1810::DEBUG::2012-12-02 23:01:57,678::task::568::TaskManager.Task::(_updateState) Task=`7a2b0b4a-021a-4206-86a6-845f3b10f1e1`::moving from state preparing -> state finished >Thread-1810::DEBUG::2012-12-02 23:01:57,678::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1810::DEBUG::2012-12-02 23:01:57,679::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1810::DEBUG::2012-12-02 23:01:57,679::task::957::TaskManager.Task::(_decref) Task=`7a2b0b4a-021a-4206-86a6-845f3b10f1e1`::ref 0 aborting False >Thread-1811::DEBUG::2012-12-02 23:01:58,786::task::568::TaskManager.Task::(_updateState) Task=`0cabc216-55b8-4d53-ab4e-4dc8789acb04`::moving from state init -> state preparing >Thread-1811::INFO::2012-12-02 23:01:58,787::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1811::INFO::2012-12-02 23:01:58,787::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482113.0680709, 'code': 200, 'valid': False}} >Thread-1811::DEBUG::2012-12-02 23:01:58,787::task::1151::TaskManager.Task::(prepare) Task=`0cabc216-55b8-4d53-ab4e-4dc8789acb04`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482113.0680709, 'code': 200, 'valid': False}} >Thread-1811::DEBUG::2012-12-02 23:01:58,787::task::568::TaskManager.Task::(_updateState) Task=`0cabc216-55b8-4d53-ab4e-4dc8789acb04`::moving from state preparing -> state finished >Thread-1811::DEBUG::2012-12-02 23:01:58,787::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1811::DEBUG::2012-12-02 23:01:58,787::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1811::DEBUG::2012-12-02 23:01:58,787::task::957::TaskManager.Task::(_decref) Task=`0cabc216-55b8-4d53-ab4e-4dc8789acb04`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:02:03,075::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1817::DEBUG::2012-12-02 23:02:08,156::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1817::DEBUG::2012-12-02 23:02:08,156::task::568::TaskManager.Task::(_updateState) Task=`b05b3508-aa46-4eca-a957-0bec9d08f790`::moving from state init -> state preparing >Thread-1817::INFO::2012-12-02 23:02:08,156::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1817::INFO::2012-12-02 23:02:08,157::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1817::DEBUG::2012-12-02 23:02:08,157::task::1151::TaskManager.Task::(prepare) Task=`b05b3508-aa46-4eca-a957-0bec9d08f790`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1817::DEBUG::2012-12-02 23:02:08,157::task::568::TaskManager.Task::(_updateState) Task=`b05b3508-aa46-4eca-a957-0bec9d08f790`::moving from state preparing -> state finished >Thread-1817::DEBUG::2012-12-02 23:02:08,157::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1817::DEBUG::2012-12-02 23:02:08,157::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1817::DEBUG::2012-12-02 23:02:08,157::task::957::TaskManager.Task::(_decref) Task=`b05b3508-aa46-4eca-a957-0bec9d08f790`::ref 0 aborting False >Thread-1818::DEBUG::2012-12-02 23:02:09,789::task::568::TaskManager.Task::(_updateState) Task=`b703d2b8-dc0f-4174-b813-b22df36f7a4b`::moving from state init -> state preparing >Thread-1818::INFO::2012-12-02 23:02:09,790::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1818::INFO::2012-12-02 23:02:09,790::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482123.0759239, 'code': 200, 'valid': False}} >Thread-1818::DEBUG::2012-12-02 23:02:09,790::task::1151::TaskManager.Task::(prepare) Task=`b703d2b8-dc0f-4174-b813-b22df36f7a4b`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482123.0759239, 'code': 200, 'valid': False}} >Thread-1818::DEBUG::2012-12-02 23:02:09,790::task::568::TaskManager.Task::(_updateState) Task=`b703d2b8-dc0f-4174-b813-b22df36f7a4b`::moving from state preparing -> state finished >Thread-1818::DEBUG::2012-12-02 23:02:09,790::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1818::DEBUG::2012-12-02 23:02:09,790::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1818::DEBUG::2012-12-02 23:02:09,791::task::957::TaskManager.Task::(_decref) Task=`b703d2b8-dc0f-4174-b813-b22df36f7a4b`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:02:13,084::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1824::DEBUG::2012-12-02 23:02:18,622::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1824::DEBUG::2012-12-02 23:02:18,623::task::568::TaskManager.Task::(_updateState) Task=`cc069dee-2fc9-445e-9545-20cc3ca43e04`::moving from state init -> state preparing >Thread-1824::INFO::2012-12-02 23:02:18,623::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1824::INFO::2012-12-02 23:02:18,623::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1824::DEBUG::2012-12-02 23:02:18,623::task::1151::TaskManager.Task::(prepare) Task=`cc069dee-2fc9-445e-9545-20cc3ca43e04`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1824::DEBUG::2012-12-02 23:02:18,623::task::568::TaskManager.Task::(_updateState) Task=`cc069dee-2fc9-445e-9545-20cc3ca43e04`::moving from state preparing -> state finished >Thread-1824::DEBUG::2012-12-02 23:02:18,623::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1824::DEBUG::2012-12-02 23:02:18,624::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1824::DEBUG::2012-12-02 23:02:18,624::task::957::TaskManager.Task::(_decref) Task=`cc069dee-2fc9-445e-9545-20cc3ca43e04`::ref 0 aborting False >Thread-1825::DEBUG::2012-12-02 23:02:20,721::task::568::TaskManager.Task::(_updateState) Task=`ed4e1d99-0ff2-43eb-bd77-a7e22d85ff50`::moving from state init -> state preparing >Thread-1825::INFO::2012-12-02 23:02:20,721::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1825::INFO::2012-12-02 23:02:20,721::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482133.084415, 'code': 200, 'valid': False}} >Thread-1825::DEBUG::2012-12-02 23:02:20,721::task::1151::TaskManager.Task::(prepare) Task=`ed4e1d99-0ff2-43eb-bd77-a7e22d85ff50`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482133.084415, 'code': 200, 'valid': False}} >Thread-1825::DEBUG::2012-12-02 23:02:20,722::task::568::TaskManager.Task::(_updateState) Task=`ed4e1d99-0ff2-43eb-bd77-a7e22d85ff50`::moving from state preparing -> state finished >Thread-1825::DEBUG::2012-12-02 23:02:20,722::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1825::DEBUG::2012-12-02 23:02:20,722::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1825::DEBUG::2012-12-02 23:02:20,722::task::957::TaskManager.Task::(_decref) Task=`ed4e1d99-0ff2-43eb-bd77-a7e22d85ff50`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:02:23,089::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1830::DEBUG::2012-12-02 23:02:29,097::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1830::DEBUG::2012-12-02 23:02:29,098::task::568::TaskManager.Task::(_updateState) Task=`6b4294d9-2bee-4e8f-b3c5-38af9fa86268`::moving from state init -> state preparing >Thread-1830::INFO::2012-12-02 23:02:29,098::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1830::INFO::2012-12-02 23:02:29,098::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1830::DEBUG::2012-12-02 23:02:29,098::task::1151::TaskManager.Task::(prepare) Task=`6b4294d9-2bee-4e8f-b3c5-38af9fa86268`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1830::DEBUG::2012-12-02 23:02:29,098::task::568::TaskManager.Task::(_updateState) Task=`6b4294d9-2bee-4e8f-b3c5-38af9fa86268`::moving from state preparing -> state finished >Thread-1830::DEBUG::2012-12-02 23:02:29,099::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1830::DEBUG::2012-12-02 23:02:29,099::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1830::DEBUG::2012-12-02 23:02:29,099::task::957::TaskManager.Task::(_decref) Task=`6b4294d9-2bee-4e8f-b3c5-38af9fa86268`::ref 0 aborting False >Thread-1832::DEBUG::2012-12-02 23:02:31,672::task::568::TaskManager.Task::(_updateState) Task=`2cf50389-e348-4d44-aff4-dc77ab19fd2d`::moving from state init -> state preparing >Thread-1832::INFO::2012-12-02 23:02:31,672::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1832::INFO::2012-12-02 23:02:31,672::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482143.089792, 'code': 200, 'valid': False}} >Thread-1832::DEBUG::2012-12-02 23:02:31,672::task::1151::TaskManager.Task::(prepare) Task=`2cf50389-e348-4d44-aff4-dc77ab19fd2d`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482143.089792, 'code': 200, 'valid': False}} >Thread-1832::DEBUG::2012-12-02 23:02:31,673::task::568::TaskManager.Task::(_updateState) Task=`2cf50389-e348-4d44-aff4-dc77ab19fd2d`::moving from state preparing -> state finished >Thread-1832::DEBUG::2012-12-02 23:02:31,673::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1832::DEBUG::2012-12-02 23:02:31,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1832::DEBUG::2012-12-02 23:02:31,673::task::957::TaskManager.Task::(_decref) Task=`2cf50389-e348-4d44-aff4-dc77ab19fd2d`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:02:33,096::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1837::DEBUG::2012-12-02 23:02:39,721::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1837::DEBUG::2012-12-02 23:02:39,722::task::568::TaskManager.Task::(_updateState) Task=`2b380e28-a8ad-482e-98ab-8dbd515a59bb`::moving from state init -> state preparing >Thread-1837::INFO::2012-12-02 23:02:39,722::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1837::INFO::2012-12-02 23:02:39,722::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1837::DEBUG::2012-12-02 23:02:39,722::task::1151::TaskManager.Task::(prepare) Task=`2b380e28-a8ad-482e-98ab-8dbd515a59bb`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1837::DEBUG::2012-12-02 23:02:39,722::task::568::TaskManager.Task::(_updateState) Task=`2b380e28-a8ad-482e-98ab-8dbd515a59bb`::moving from state preparing -> state finished >Thread-1837::DEBUG::2012-12-02 23:02:39,723::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1837::DEBUG::2012-12-02 23:02:39,723::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1837::DEBUG::2012-12-02 23:02:39,723::task::957::TaskManager.Task::(_decref) Task=`2b380e28-a8ad-482e-98ab-8dbd515a59bb`::ref 0 aborting False >Thread-1839::DEBUG::2012-12-02 23:02:42,629::task::568::TaskManager.Task::(_updateState) Task=`55c76a00-5e12-4d50-8e12-ad2f33f2ea3e`::moving from state init -> state preparing >Thread-1839::INFO::2012-12-02 23:02:42,629::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1839::INFO::2012-12-02 23:02:42,629::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482153.0962889, 'code': 200, 'valid': False}} >Thread-1839::DEBUG::2012-12-02 23:02:42,629::task::1151::TaskManager.Task::(prepare) Task=`55c76a00-5e12-4d50-8e12-ad2f33f2ea3e`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482153.0962889, 'code': 200, 'valid': False}} >Thread-1839::DEBUG::2012-12-02 23:02:42,630::task::568::TaskManager.Task::(_updateState) Task=`55c76a00-5e12-4d50-8e12-ad2f33f2ea3e`::moving from state preparing -> state finished >Thread-1839::DEBUG::2012-12-02 23:02:42,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1839::DEBUG::2012-12-02 23:02:42,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1839::DEBUG::2012-12-02 23:02:42,630::task::957::TaskManager.Task::(_decref) Task=`55c76a00-5e12-4d50-8e12-ad2f33f2ea3e`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:02:43,102::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1844::DEBUG::2012-12-02 23:02:50,320::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1844::DEBUG::2012-12-02 23:02:50,320::task::568::TaskManager.Task::(_updateState) Task=`e07370b7-e155-45b0-b2d5-63f9228e4c88`::moving from state init -> state preparing >Thread-1844::INFO::2012-12-02 23:02:50,321::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1844::INFO::2012-12-02 23:02:50,321::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1844::DEBUG::2012-12-02 23:02:50,321::task::1151::TaskManager.Task::(prepare) Task=`e07370b7-e155-45b0-b2d5-63f9228e4c88`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1844::DEBUG::2012-12-02 23:02:50,321::task::568::TaskManager.Task::(_updateState) Task=`e07370b7-e155-45b0-b2d5-63f9228e4c88`::moving from state preparing -> state finished >Thread-1844::DEBUG::2012-12-02 23:02:50,321::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1844::DEBUG::2012-12-02 23:02:50,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1844::DEBUG::2012-12-02 23:02:50,321::task::957::TaskManager.Task::(_decref) Task=`e07370b7-e155-45b0-b2d5-63f9228e4c88`::ref 0 aborting False >Thread-1252::ERROR::2012-12-02 23:02:53,108::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251/2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-1846::DEBUG::2012-12-02 23:02:53,545::task::568::TaskManager.Task::(_updateState) Task=`c0500464-ccf0-456f-ac0b-e2708215a880`::moving from state init -> state preparing >Thread-1846::INFO::2012-12-02 23:02:53,545::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1846::INFO::2012-12-02 23:02:53,546::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-1846::DEBUG::2012-12-02 23:02:53,546::task::1151::TaskManager.Task::(prepare) Task=`c0500464-ccf0-456f-ac0b-e2708215a880`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-1846::DEBUG::2012-12-02 23:02:53,546::task::568::TaskManager.Task::(_updateState) Task=`c0500464-ccf0-456f-ac0b-e2708215a880`::moving from state preparing -> state finished >Thread-1846::DEBUG::2012-12-02 23:02:53,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1846::DEBUG::2012-12-02 23:02:53,546::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1846::DEBUG::2012-12-02 23:02:53,546::task::957::TaskManager.Task::(_decref) Task=`c0500464-ccf0-456f-ac0b-e2708215a880`::ref 0 aborting False >Thread-1851::DEBUG::2012-12-02 23:03:00,931::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1851::DEBUG::2012-12-02 23:03:00,932::task::568::TaskManager.Task::(_updateState) Task=`6f739cdf-5e1e-48c5-86ec-8101b8ff7d63`::moving from state init -> state preparing >Thread-1851::INFO::2012-12-02 23:03:00,932::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1851::INFO::2012-12-02 23:03:00,932::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1851::DEBUG::2012-12-02 23:03:00,933::task::1151::TaskManager.Task::(prepare) Task=`6f739cdf-5e1e-48c5-86ec-8101b8ff7d63`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1851::DEBUG::2012-12-02 23:03:00,933::task::568::TaskManager.Task::(_updateState) Task=`6f739cdf-5e1e-48c5-86ec-8101b8ff7d63`::moving from state preparing -> state finished >Thread-1851::DEBUG::2012-12-02 23:03:00,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1851::DEBUG::2012-12-02 23:03:00,933::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1851::DEBUG::2012-12-02 23:03:00,933::task::957::TaskManager.Task::(_decref) Task=`6f739cdf-5e1e-48c5-86ec-8101b8ff7d63`::ref 0 aborting False >Thread-1252::DEBUG::2012-12-02 23:03:03,108::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 2ee486e4-abfd-4f11-9b29-42bfc74d7064 >Thread-1252::DEBUG::2012-12-02 23:03:03,108::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1252::DEBUG::2012-12-02 23:03:03,109::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1252::DEBUG::2012-12-02 23:03:03,338::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1252::WARNING::2012-12-02 23:03:03,341::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1252::DEBUG::2012-12-02 23:03:03,341::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1252::DEBUG::2012-12-02 23:03:03,376::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1252::DEBUG::2012-12-02 23:03:03,377::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1252::DEBUG::2012-12-02 23:03:03,587::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1252::WARNING::2012-12-02 23:03:03,588::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1252::DEBUG::2012-12-02 23:03:03,588::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1975::DEBUG::2012-12-02 23:03:04,497::task::568::TaskManager.Task::(_updateState) Task=`dfb0280f-8a33-42cf-895c-4845a2ba47d0`::moving from state init -> state preparing >Thread-1975::INFO::2012-12-02 23:03:04,497::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1975::INFO::2012-12-02 23:03:04,497::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-1975::DEBUG::2012-12-02 23:03:04,498::task::1151::TaskManager.Task::(prepare) Task=`dfb0280f-8a33-42cf-895c-4845a2ba47d0`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-1975::DEBUG::2012-12-02 23:03:04,498::task::568::TaskManager.Task::(_updateState) Task=`dfb0280f-8a33-42cf-895c-4845a2ba47d0`::moving from state preparing -> state finished >Thread-1975::DEBUG::2012-12-02 23:03:04,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1975::DEBUG::2012-12-02 23:03:04,498::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1975::DEBUG::2012-12-02 23:03:04,498::task::957::TaskManager.Task::(_decref) Task=`dfb0280f-8a33-42cf-895c-4845a2ba47d0`::ref 0 aborting False >Thread-1980::DEBUG::2012-12-02 23:03:11,527::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1980::DEBUG::2012-12-02 23:03:11,528::task::568::TaskManager.Task::(_updateState) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::moving from state init -> state preparing >Thread-1980::INFO::2012-12-02 23:03:11,528::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1980::DEBUG::2012-12-02 23:03:11,528::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1980::DEBUG::2012-12-02 23:03:11,529::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1980::DEBUG::2012-12-02 23:03:11,737::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1980::WARNING::2012-12-02 23:03:11,738::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1980::DEBUG::2012-12-02 23:03:11,738::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1980::ERROR::2012-12-02 23:03:11,767::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-1980::ERROR::2012-12-02 23:03:11,768::task::833::TaskManager.Task::(_setError) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-1980::DEBUG::2012-12-02 23:03:11,768::task::852::TaskManager.Task::(_run) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::Task._run: 99ff9368-7a8e-42ae-808e-95fc55e58594 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1980::DEBUG::2012-12-02 23:03:11,768::task::1177::TaskManager.Task::(stop) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::stopping in state preparing (force False) >Thread-1980::DEBUG::2012-12-02 23:03:11,768::task::957::TaskManager.Task::(_decref) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::ref 1 aborting True >Thread-1980::INFO::2012-12-02 23:03:11,768::task::1134::TaskManager.Task::(prepare) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-1980::DEBUG::2012-12-02 23:03:11,768::task::1139::TaskManager.Task::(prepare) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::Prepare: aborted: Error validating master storage domain >Thread-1980::DEBUG::2012-12-02 23:03:11,769::task::957::TaskManager.Task::(_decref) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::ref 0 aborting True >Thread-1980::DEBUG::2012-12-02 23:03:11,769::task::892::TaskManager.Task::(_doAbort) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::Task._doAbort: force False >Thread-1980::DEBUG::2012-12-02 23:03:11,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1980::DEBUG::2012-12-02 23:03:11,769::task::568::TaskManager.Task::(_updateState) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::moving from state preparing -> state aborting >Thread-1980::DEBUG::2012-12-02 23:03:11,769::task::523::TaskManager.Task::(__state_aborting) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::_aborting: recover policy none >Thread-1980::DEBUG::2012-12-02 23:03:11,769::task::568::TaskManager.Task::(_updateState) Task=`99ff9368-7a8e-42ae-808e-95fc55e58594`::moving from state aborting -> state failed >Thread-1980::DEBUG::2012-12-02 23:03:11,769::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1980::DEBUG::2012-12-02 23:03:11,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1980::ERROR::2012-12-02 23:03:11,770::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2043::DEBUG::2012-12-02 23:03:15,440::task::568::TaskManager.Task::(_updateState) Task=`7fe8704f-58a8-4e04-aac3-9cd8e7b804a8`::moving from state init -> state preparing >Thread-2043::INFO::2012-12-02 23:03:15,441::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2043::INFO::2012-12-02 23:03:15,441::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2043::DEBUG::2012-12-02 23:03:15,441::task::1151::TaskManager.Task::(prepare) Task=`7fe8704f-58a8-4e04-aac3-9cd8e7b804a8`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2043::DEBUG::2012-12-02 23:03:15,441::task::568::TaskManager.Task::(_updateState) Task=`7fe8704f-58a8-4e04-aac3-9cd8e7b804a8`::moving from state preparing -> state finished >Thread-2043::DEBUG::2012-12-02 23:03:15,441::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2043::DEBUG::2012-12-02 23:03:15,441::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2043::DEBUG::2012-12-02 23:03:15,441::task::957::TaskManager.Task::(_decref) Task=`7fe8704f-58a8-4e04-aac3-9cd8e7b804a8`::ref 0 aborting False >Thread-2047::DEBUG::2012-12-02 23:03:21,969::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2047::DEBUG::2012-12-02 23:03:21,969::task::568::TaskManager.Task::(_updateState) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::moving from state init -> state preparing >Thread-2047::INFO::2012-12-02 23:03:21,969::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2047::DEBUG::2012-12-02 23:03:21,969::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2047::DEBUG::2012-12-02 23:03:21,970::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2047::DEBUG::2012-12-02 23:03:22,177::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2047::WARNING::2012-12-02 23:03:22,178::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2047::DEBUG::2012-12-02 23:03:22,178::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2047::ERROR::2012-12-02 23:03:22,208::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2047::ERROR::2012-12-02 23:03:22,209::task::833::TaskManager.Task::(_setError) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2047::DEBUG::2012-12-02 23:03:22,209::task::852::TaskManager.Task::(_run) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::Task._run: 5caf0044-f17e-4f0a-aaec-7292f3f00bc8 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2047::DEBUG::2012-12-02 23:03:22,209::task::1177::TaskManager.Task::(stop) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::stopping in state preparing (force False) >Thread-2047::DEBUG::2012-12-02 23:03:22,209::task::957::TaskManager.Task::(_decref) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::ref 1 aborting True >Thread-2047::INFO::2012-12-02 23:03:22,209::task::1134::TaskManager.Task::(prepare) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2047::DEBUG::2012-12-02 23:03:22,210::task::1139::TaskManager.Task::(prepare) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::Prepare: aborted: Error validating master storage domain >Thread-2047::DEBUG::2012-12-02 23:03:22,210::task::957::TaskManager.Task::(_decref) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::ref 0 aborting True >Thread-2047::DEBUG::2012-12-02 23:03:22,210::task::892::TaskManager.Task::(_doAbort) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::Task._doAbort: force False >Thread-2047::DEBUG::2012-12-02 23:03:22,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2047::DEBUG::2012-12-02 23:03:22,210::task::568::TaskManager.Task::(_updateState) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::moving from state preparing -> state aborting >Thread-2047::DEBUG::2012-12-02 23:03:22,210::task::523::TaskManager.Task::(__state_aborting) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::_aborting: recover policy none >Thread-2047::DEBUG::2012-12-02 23:03:22,210::task::568::TaskManager.Task::(_updateState) Task=`5caf0044-f17e-4f0a-aaec-7292f3f00bc8`::moving from state aborting -> state failed >Thread-2047::DEBUG::2012-12-02 23:03:22,211::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2047::DEBUG::2012-12-02 23:03:22,211::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2047::ERROR::2012-12-02 23:03:22,211::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2111::DEBUG::2012-12-02 23:03:26,375::task::568::TaskManager.Task::(_updateState) Task=`2a3fe17d-e98b-49b2-b0c6-41486ecb73cb`::moving from state init -> state preparing >Thread-2111::INFO::2012-12-02 23:03:26,375::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2111::INFO::2012-12-02 23:03:26,375::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2111::DEBUG::2012-12-02 23:03:26,375::task::1151::TaskManager.Task::(prepare) Task=`2a3fe17d-e98b-49b2-b0c6-41486ecb73cb`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2111::DEBUG::2012-12-02 23:03:26,375::task::568::TaskManager.Task::(_updateState) Task=`2a3fe17d-e98b-49b2-b0c6-41486ecb73cb`::moving from state preparing -> state finished >Thread-2111::DEBUG::2012-12-02 23:03:26,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2111::DEBUG::2012-12-02 23:03:26,376::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2111::DEBUG::2012-12-02 23:03:26,376::task::957::TaskManager.Task::(_decref) Task=`2a3fe17d-e98b-49b2-b0c6-41486ecb73cb`::ref 0 aborting False >Thread-2115::DEBUG::2012-12-02 23:03:32,371::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2115::DEBUG::2012-12-02 23:03:32,371::task::568::TaskManager.Task::(_updateState) Task=`32279101-3924-4de4-8d60-05a819a6657a`::moving from state init -> state preparing >Thread-2115::INFO::2012-12-02 23:03:32,371::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2115::DEBUG::2012-12-02 23:03:32,372::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2115::DEBUG::2012-12-02 23:03:32,372::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2115::DEBUG::2012-12-02 23:03:32,580::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2115::WARNING::2012-12-02 23:03:32,581::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2115::DEBUG::2012-12-02 23:03:32,581::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2115::ERROR::2012-12-02 23:03:32,612::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2115::ERROR::2012-12-02 23:03:32,612::task::833::TaskManager.Task::(_setError) Task=`32279101-3924-4de4-8d60-05a819a6657a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2115::DEBUG::2012-12-02 23:03:32,612::task::852::TaskManager.Task::(_run) Task=`32279101-3924-4de4-8d60-05a819a6657a`::Task._run: 32279101-3924-4de4-8d60-05a819a6657a ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2115::DEBUG::2012-12-02 23:03:32,612::task::1177::TaskManager.Task::(stop) Task=`32279101-3924-4de4-8d60-05a819a6657a`::stopping in state preparing (force False) >Thread-2115::DEBUG::2012-12-02 23:03:32,613::task::957::TaskManager.Task::(_decref) Task=`32279101-3924-4de4-8d60-05a819a6657a`::ref 1 aborting True >Thread-2115::INFO::2012-12-02 23:03:32,613::task::1134::TaskManager.Task::(prepare) Task=`32279101-3924-4de4-8d60-05a819a6657a`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2115::DEBUG::2012-12-02 23:03:32,613::task::1139::TaskManager.Task::(prepare) Task=`32279101-3924-4de4-8d60-05a819a6657a`::Prepare: aborted: Error validating master storage domain >Thread-2115::DEBUG::2012-12-02 23:03:32,613::task::957::TaskManager.Task::(_decref) Task=`32279101-3924-4de4-8d60-05a819a6657a`::ref 0 aborting True >Thread-2115::DEBUG::2012-12-02 23:03:32,613::task::892::TaskManager.Task::(_doAbort) Task=`32279101-3924-4de4-8d60-05a819a6657a`::Task._doAbort: force False >Thread-2115::DEBUG::2012-12-02 23:03:32,613::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2115::DEBUG::2012-12-02 23:03:32,613::task::568::TaskManager.Task::(_updateState) Task=`32279101-3924-4de4-8d60-05a819a6657a`::moving from state preparing -> state aborting >Thread-2115::DEBUG::2012-12-02 23:03:32,614::task::523::TaskManager.Task::(__state_aborting) Task=`32279101-3924-4de4-8d60-05a819a6657a`::_aborting: recover policy none >Thread-2115::DEBUG::2012-12-02 23:03:32,614::task::568::TaskManager.Task::(_updateState) Task=`32279101-3924-4de4-8d60-05a819a6657a`::moving from state aborting -> state failed >Thread-2115::DEBUG::2012-12-02 23:03:32,614::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2115::DEBUG::2012-12-02 23:03:32,614::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2115::ERROR::2012-12-02 23:03:32,614::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2179::DEBUG::2012-12-02 23:03:37,271::task::568::TaskManager.Task::(_updateState) Task=`c0247e5e-9ea7-48f3-b0ec-507c94039580`::moving from state init -> state preparing >Thread-2179::INFO::2012-12-02 23:03:37,271::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2179::INFO::2012-12-02 23:03:37,271::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2179::DEBUG::2012-12-02 23:03:37,271::task::1151::TaskManager.Task::(prepare) Task=`c0247e5e-9ea7-48f3-b0ec-507c94039580`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2179::DEBUG::2012-12-02 23:03:37,271::task::568::TaskManager.Task::(_updateState) Task=`c0247e5e-9ea7-48f3-b0ec-507c94039580`::moving from state preparing -> state finished >Thread-2179::DEBUG::2012-12-02 23:03:37,272::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2179::DEBUG::2012-12-02 23:03:37,272::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2179::DEBUG::2012-12-02 23:03:37,272::task::957::TaskManager.Task::(_decref) Task=`c0247e5e-9ea7-48f3-b0ec-507c94039580`::ref 0 aborting False >Thread-2183::DEBUG::2012-12-02 23:03:42,785::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2183::DEBUG::2012-12-02 23:03:42,785::task::568::TaskManager.Task::(_updateState) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::moving from state init -> state preparing >Thread-2183::INFO::2012-12-02 23:03:42,785::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2183::DEBUG::2012-12-02 23:03:42,786::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2183::DEBUG::2012-12-02 23:03:42,786::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2183::DEBUG::2012-12-02 23:03:42,997::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2183::WARNING::2012-12-02 23:03:42,998::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2183::DEBUG::2012-12-02 23:03:42,998::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2183::ERROR::2012-12-02 23:03:43,027::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2183::ERROR::2012-12-02 23:03:43,028::task::833::TaskManager.Task::(_setError) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2183::DEBUG::2012-12-02 23:03:43,028::task::852::TaskManager.Task::(_run) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::Task._run: d14ea0ec-99a6-4150-831b-cb42bde32a87 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2183::DEBUG::2012-12-02 23:03:43,028::task::1177::TaskManager.Task::(stop) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::stopping in state preparing (force False) >Thread-2183::DEBUG::2012-12-02 23:03:43,028::task::957::TaskManager.Task::(_decref) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::ref 1 aborting True >Thread-2183::INFO::2012-12-02 23:03:43,029::task::1134::TaskManager.Task::(prepare) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2183::DEBUG::2012-12-02 23:03:43,029::task::1139::TaskManager.Task::(prepare) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::Prepare: aborted: Error validating master storage domain >Thread-2183::DEBUG::2012-12-02 23:03:43,029::task::957::TaskManager.Task::(_decref) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::ref 0 aborting True >Thread-2183::DEBUG::2012-12-02 23:03:43,029::task::892::TaskManager.Task::(_doAbort) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::Task._doAbort: force False >Thread-2183::DEBUG::2012-12-02 23:03:43,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2183::DEBUG::2012-12-02 23:03:43,029::task::568::TaskManager.Task::(_updateState) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::moving from state preparing -> state aborting >Thread-2183::DEBUG::2012-12-02 23:03:43,029::task::523::TaskManager.Task::(__state_aborting) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::_aborting: recover policy none >Thread-2183::DEBUG::2012-12-02 23:03:43,030::task::568::TaskManager.Task::(_updateState) Task=`d14ea0ec-99a6-4150-831b-cb42bde32a87`::moving from state aborting -> state failed >Thread-2183::DEBUG::2012-12-02 23:03:43,030::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2183::DEBUG::2012-12-02 23:03:43,030::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2183::ERROR::2012-12-02 23:03:43,030::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2247::DEBUG::2012-12-02 23:03:48,213::task::568::TaskManager.Task::(_updateState) Task=`a27d8e75-5b3b-41ef-aa2f-fe4a76ab0e08`::moving from state init -> state preparing >Thread-2247::INFO::2012-12-02 23:03:48,213::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2247::INFO::2012-12-02 23:03:48,214::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2247::DEBUG::2012-12-02 23:03:48,214::task::1151::TaskManager.Task::(prepare) Task=`a27d8e75-5b3b-41ef-aa2f-fe4a76ab0e08`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2247::DEBUG::2012-12-02 23:03:48,214::task::568::TaskManager.Task::(_updateState) Task=`a27d8e75-5b3b-41ef-aa2f-fe4a76ab0e08`::moving from state preparing -> state finished >Thread-2247::DEBUG::2012-12-02 23:03:48,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2247::DEBUG::2012-12-02 23:03:48,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2247::DEBUG::2012-12-02 23:03:48,215::task::957::TaskManager.Task::(_decref) Task=`a27d8e75-5b3b-41ef-aa2f-fe4a76ab0e08`::ref 0 aborting False >Thread-2251::DEBUG::2012-12-02 23:03:53,202::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2251::DEBUG::2012-12-02 23:03:53,203::task::568::TaskManager.Task::(_updateState) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::moving from state init -> state preparing >Thread-2251::INFO::2012-12-02 23:03:53,203::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2251::DEBUG::2012-12-02 23:03:53,203::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2251::DEBUG::2012-12-02 23:03:53,203::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2251::DEBUG::2012-12-02 23:03:53,407::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2251::WARNING::2012-12-02 23:03:53,408::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2251::DEBUG::2012-12-02 23:03:53,408::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2251::ERROR::2012-12-02 23:03:53,437::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2251::ERROR::2012-12-02 23:03:53,437::task::833::TaskManager.Task::(_setError) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2251::DEBUG::2012-12-02 23:03:53,437::task::852::TaskManager.Task::(_run) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::Task._run: 1635879e-0567-4f7f-892f-e81f3fb9862a ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2251::DEBUG::2012-12-02 23:03:53,438::task::1177::TaskManager.Task::(stop) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::stopping in state preparing (force False) >Thread-2251::DEBUG::2012-12-02 23:03:53,438::task::957::TaskManager.Task::(_decref) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::ref 1 aborting True >Thread-2251::INFO::2012-12-02 23:03:53,438::task::1134::TaskManager.Task::(prepare) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2251::DEBUG::2012-12-02 23:03:53,438::task::1139::TaskManager.Task::(prepare) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::Prepare: aborted: Error validating master storage domain >Thread-2251::DEBUG::2012-12-02 23:03:53,438::task::957::TaskManager.Task::(_decref) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::ref 0 aborting True >Thread-2251::DEBUG::2012-12-02 23:03:53,438::task::892::TaskManager.Task::(_doAbort) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::Task._doAbort: force False >Thread-2251::DEBUG::2012-12-02 23:03:53,438::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2251::DEBUG::2012-12-02 23:03:53,438::task::568::TaskManager.Task::(_updateState) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::moving from state preparing -> state aborting >Thread-2251::DEBUG::2012-12-02 23:03:53,439::task::523::TaskManager.Task::(__state_aborting) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::_aborting: recover policy none >Thread-2251::DEBUG::2012-12-02 23:03:53,439::task::568::TaskManager.Task::(_updateState) Task=`1635879e-0567-4f7f-892f-e81f3fb9862a`::moving from state aborting -> state failed >Thread-2251::DEBUG::2012-12-02 23:03:53,439::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2251::DEBUG::2012-12-02 23:03:53,439::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2251::ERROR::2012-12-02 23:03:53,439::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2315::DEBUG::2012-12-02 23:03:59,224::task::568::TaskManager.Task::(_updateState) Task=`8a13c150-0507-4747-be87-a3304d7204af`::moving from state init -> state preparing >Thread-2315::INFO::2012-12-02 23:03:59,224::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2315::INFO::2012-12-02 23:03:59,224::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2315::DEBUG::2012-12-02 23:03:59,225::task::1151::TaskManager.Task::(prepare) Task=`8a13c150-0507-4747-be87-a3304d7204af`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2315::DEBUG::2012-12-02 23:03:59,225::task::568::TaskManager.Task::(_updateState) Task=`8a13c150-0507-4747-be87-a3304d7204af`::moving from state preparing -> state finished >Thread-2315::DEBUG::2012-12-02 23:03:59,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2315::DEBUG::2012-12-02 23:03:59,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2315::DEBUG::2012-12-02 23:03:59,225::task::957::TaskManager.Task::(_decref) Task=`8a13c150-0507-4747-be87-a3304d7204af`::ref 0 aborting False >Thread-2318::DEBUG::2012-12-02 23:04:03,600::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2318::DEBUG::2012-12-02 23:04:03,600::task::568::TaskManager.Task::(_updateState) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::moving from state init -> state preparing >Thread-2318::INFO::2012-12-02 23:04:03,600::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2318::DEBUG::2012-12-02 23:04:03,601::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2318::DEBUG::2012-12-02 23:04:03,601::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2318::DEBUG::2012-12-02 23:04:03,803::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2318::WARNING::2012-12-02 23:04:03,805::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2318::DEBUG::2012-12-02 23:04:03,805::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2318::ERROR::2012-12-02 23:04:03,838::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2318::ERROR::2012-12-02 23:04:03,838::task::833::TaskManager.Task::(_setError) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2318::DEBUG::2012-12-02 23:04:03,839::task::852::TaskManager.Task::(_run) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::Task._run: a0d90e7e-84b0-45a6-8cdb-c77df672fb65 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2318::DEBUG::2012-12-02 23:04:03,839::task::1177::TaskManager.Task::(stop) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::stopping in state preparing (force False) >Thread-2318::DEBUG::2012-12-02 23:04:03,839::task::957::TaskManager.Task::(_decref) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::ref 1 aborting True >Thread-2318::INFO::2012-12-02 23:04:03,839::task::1134::TaskManager.Task::(prepare) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2318::DEBUG::2012-12-02 23:04:03,839::task::1139::TaskManager.Task::(prepare) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::Prepare: aborted: Error validating master storage domain >Thread-2318::DEBUG::2012-12-02 23:04:03,839::task::957::TaskManager.Task::(_decref) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::ref 0 aborting True >Thread-2318::DEBUG::2012-12-02 23:04:03,839::task::892::TaskManager.Task::(_doAbort) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::Task._doAbort: force False >Thread-2318::DEBUG::2012-12-02 23:04:03,840::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2318::DEBUG::2012-12-02 23:04:03,840::task::568::TaskManager.Task::(_updateState) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::moving from state preparing -> state aborting >Thread-2318::DEBUG::2012-12-02 23:04:03,840::task::523::TaskManager.Task::(__state_aborting) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::_aborting: recover policy none >Thread-2318::DEBUG::2012-12-02 23:04:03,840::task::568::TaskManager.Task::(_updateState) Task=`a0d90e7e-84b0-45a6-8cdb-c77df672fb65`::moving from state aborting -> state failed >Thread-2318::DEBUG::2012-12-02 23:04:03,840::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2318::DEBUG::2012-12-02 23:04:03,840::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2318::ERROR::2012-12-02 23:04:03,840::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2383::DEBUG::2012-12-02 23:04:10,208::task::568::TaskManager.Task::(_updateState) Task=`56ce62b2-aa02-4c38-8bc8-ebf5f5e0fe40`::moving from state init -> state preparing >Thread-2383::INFO::2012-12-02 23:04:10,208::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2383::INFO::2012-12-02 23:04:10,209::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2383::DEBUG::2012-12-02 23:04:10,209::task::1151::TaskManager.Task::(prepare) Task=`56ce62b2-aa02-4c38-8bc8-ebf5f5e0fe40`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2383::DEBUG::2012-12-02 23:04:10,209::task::568::TaskManager.Task::(_updateState) Task=`56ce62b2-aa02-4c38-8bc8-ebf5f5e0fe40`::moving from state preparing -> state finished >Thread-2383::DEBUG::2012-12-02 23:04:10,209::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2383::DEBUG::2012-12-02 23:04:10,209::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2383::DEBUG::2012-12-02 23:04:10,209::task::957::TaskManager.Task::(_decref) Task=`56ce62b2-aa02-4c38-8bc8-ebf5f5e0fe40`::ref 0 aborting False >Thread-2386::DEBUG::2012-12-02 23:04:14,007::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2386::DEBUG::2012-12-02 23:04:14,007::task::568::TaskManager.Task::(_updateState) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::moving from state init -> state preparing >Thread-2386::INFO::2012-12-02 23:04:14,008::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2386::DEBUG::2012-12-02 23:04:14,008::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2386::DEBUG::2012-12-02 23:04:14,008::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2386::DEBUG::2012-12-02 23:04:14,210::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2386::WARNING::2012-12-02 23:04:14,211::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2386::DEBUG::2012-12-02 23:04:14,211::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2386::ERROR::2012-12-02 23:04:14,239::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2386::ERROR::2012-12-02 23:04:14,239::task::833::TaskManager.Task::(_setError) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2386::DEBUG::2012-12-02 23:04:14,240::task::852::TaskManager.Task::(_run) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::Task._run: fb58460b-c4b0-4a08-b89b-bd57f652c596 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2386::DEBUG::2012-12-02 23:04:14,240::task::1177::TaskManager.Task::(stop) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::stopping in state preparing (force False) >Thread-2386::DEBUG::2012-12-02 23:04:14,240::task::957::TaskManager.Task::(_decref) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::ref 1 aborting True >Thread-2386::INFO::2012-12-02 23:04:14,240::task::1134::TaskManager.Task::(prepare) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2386::DEBUG::2012-12-02 23:04:14,240::task::1139::TaskManager.Task::(prepare) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::Prepare: aborted: Error validating master storage domain >Thread-2386::DEBUG::2012-12-02 23:04:14,240::task::957::TaskManager.Task::(_decref) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::ref 0 aborting True >Thread-2386::DEBUG::2012-12-02 23:04:14,240::task::892::TaskManager.Task::(_doAbort) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::Task._doAbort: force False >Thread-2386::DEBUG::2012-12-02 23:04:14,240::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2386::DEBUG::2012-12-02 23:04:14,241::task::568::TaskManager.Task::(_updateState) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::moving from state preparing -> state aborting >Thread-2386::DEBUG::2012-12-02 23:04:14,241::task::523::TaskManager.Task::(__state_aborting) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::_aborting: recover policy none >Thread-2386::DEBUG::2012-12-02 23:04:14,241::task::568::TaskManager.Task::(_updateState) Task=`fb58460b-c4b0-4a08-b89b-bd57f652c596`::moving from state aborting -> state failed >Thread-2386::DEBUG::2012-12-02 23:04:14,241::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2386::DEBUG::2012-12-02 23:04:14,241::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2386::ERROR::2012-12-02 23:04:14,241::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2451::DEBUG::2012-12-02 23:04:21,131::task::568::TaskManager.Task::(_updateState) Task=`0bf24322-4c98-418e-bda3-554c72fc4853`::moving from state init -> state preparing >Thread-2451::INFO::2012-12-02 23:04:21,131::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2451::INFO::2012-12-02 23:04:21,132::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2451::DEBUG::2012-12-02 23:04:21,132::task::1151::TaskManager.Task::(prepare) Task=`0bf24322-4c98-418e-bda3-554c72fc4853`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2451::DEBUG::2012-12-02 23:04:21,132::task::568::TaskManager.Task::(_updateState) Task=`0bf24322-4c98-418e-bda3-554c72fc4853`::moving from state preparing -> state finished >Thread-2451::DEBUG::2012-12-02 23:04:21,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2451::DEBUG::2012-12-02 23:04:21,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2451::DEBUG::2012-12-02 23:04:21,132::task::957::TaskManager.Task::(_decref) Task=`0bf24322-4c98-418e-bda3-554c72fc4853`::ref 0 aborting False >Thread-2454::DEBUG::2012-12-02 23:04:24,408::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2454::DEBUG::2012-12-02 23:04:24,408::task::568::TaskManager.Task::(_updateState) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::moving from state init -> state preparing >Thread-2454::INFO::2012-12-02 23:04:24,408::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2454::DEBUG::2012-12-02 23:04:24,408::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2454::DEBUG::2012-12-02 23:04:24,409::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2454::DEBUG::2012-12-02 23:04:24,611::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2454::WARNING::2012-12-02 23:04:24,612::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2454::DEBUG::2012-12-02 23:04:24,613::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2454::ERROR::2012-12-02 23:04:24,642::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2454::ERROR::2012-12-02 23:04:24,643::task::833::TaskManager.Task::(_setError) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2454::DEBUG::2012-12-02 23:04:24,643::task::852::TaskManager.Task::(_run) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::Task._run: 961102fb-e0a4-46b5-8252-039aa5a5e47d ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2454::DEBUG::2012-12-02 23:04:24,643::task::1177::TaskManager.Task::(stop) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::stopping in state preparing (force False) >Thread-2454::DEBUG::2012-12-02 23:04:24,643::task::957::TaskManager.Task::(_decref) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::ref 1 aborting True >Thread-2454::INFO::2012-12-02 23:04:24,644::task::1134::TaskManager.Task::(prepare) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2454::DEBUG::2012-12-02 23:04:24,644::task::1139::TaskManager.Task::(prepare) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::Prepare: aborted: Error validating master storage domain >Thread-2454::DEBUG::2012-12-02 23:04:24,644::task::957::TaskManager.Task::(_decref) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::ref 0 aborting True >Thread-2454::DEBUG::2012-12-02 23:04:24,644::task::892::TaskManager.Task::(_doAbort) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::Task._doAbort: force False >Thread-2454::DEBUG::2012-12-02 23:04:24,644::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2454::DEBUG::2012-12-02 23:04:24,644::task::568::TaskManager.Task::(_updateState) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::moving from state preparing -> state aborting >Thread-2454::DEBUG::2012-12-02 23:04:24,644::task::523::TaskManager.Task::(__state_aborting) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::_aborting: recover policy none >Thread-2454::DEBUG::2012-12-02 23:04:24,645::task::568::TaskManager.Task::(_updateState) Task=`961102fb-e0a4-46b5-8252-039aa5a5e47d`::moving from state aborting -> state failed >Thread-2454::DEBUG::2012-12-02 23:04:24,645::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2454::DEBUG::2012-12-02 23:04:24,645::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2454::ERROR::2012-12-02 23:04:24,645::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2519::DEBUG::2012-12-02 23:04:32,120::task::568::TaskManager.Task::(_updateState) Task=`2adc84f3-9cbe-44ac-b45c-40dc4393e40a`::moving from state init -> state preparing >Thread-2519::INFO::2012-12-02 23:04:32,120::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2519::INFO::2012-12-02 23:04:32,120::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2519::DEBUG::2012-12-02 23:04:32,120::task::1151::TaskManager.Task::(prepare) Task=`2adc84f3-9cbe-44ac-b45c-40dc4393e40a`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2519::DEBUG::2012-12-02 23:04:32,120::task::568::TaskManager.Task::(_updateState) Task=`2adc84f3-9cbe-44ac-b45c-40dc4393e40a`::moving from state preparing -> state finished >Thread-2519::DEBUG::2012-12-02 23:04:32,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2519::DEBUG::2012-12-02 23:04:32,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2519::DEBUG::2012-12-02 23:04:32,121::task::957::TaskManager.Task::(_decref) Task=`2adc84f3-9cbe-44ac-b45c-40dc4393e40a`::ref 0 aborting False >Thread-2522::DEBUG::2012-12-02 23:04:34,803::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2522::DEBUG::2012-12-02 23:04:34,804::task::568::TaskManager.Task::(_updateState) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::moving from state init -> state preparing >Thread-2522::INFO::2012-12-02 23:04:34,804::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2522::DEBUG::2012-12-02 23:04:34,804::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2522::DEBUG::2012-12-02 23:04:34,804::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2522::DEBUG::2012-12-02 23:04:35,012::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2522::WARNING::2012-12-02 23:04:35,013::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2522::DEBUG::2012-12-02 23:04:35,013::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2522::ERROR::2012-12-02 23:04:35,043::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2522::ERROR::2012-12-02 23:04:35,043::task::833::TaskManager.Task::(_setError) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2522::DEBUG::2012-12-02 23:04:35,043::task::852::TaskManager.Task::(_run) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::Task._run: 5605e4bf-2af8-4442-9d15-4438f54d1ead ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2522::DEBUG::2012-12-02 23:04:35,044::task::1177::TaskManager.Task::(stop) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::stopping in state preparing (force False) >Thread-2522::DEBUG::2012-12-02 23:04:35,044::task::957::TaskManager.Task::(_decref) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::ref 1 aborting True >Thread-2522::INFO::2012-12-02 23:04:35,044::task::1134::TaskManager.Task::(prepare) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2522::DEBUG::2012-12-02 23:04:35,044::task::1139::TaskManager.Task::(prepare) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::Prepare: aborted: Error validating master storage domain >Thread-2522::DEBUG::2012-12-02 23:04:35,044::task::957::TaskManager.Task::(_decref) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::ref 0 aborting True >Thread-2522::DEBUG::2012-12-02 23:04:35,044::task::892::TaskManager.Task::(_doAbort) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::Task._doAbort: force False >Thread-2522::DEBUG::2012-12-02 23:04:35,044::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2522::DEBUG::2012-12-02 23:04:35,045::task::568::TaskManager.Task::(_updateState) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::moving from state preparing -> state aborting >Thread-2522::DEBUG::2012-12-02 23:04:35,045::task::523::TaskManager.Task::(__state_aborting) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::_aborting: recover policy none >Thread-2522::DEBUG::2012-12-02 23:04:35,045::task::568::TaskManager.Task::(_updateState) Task=`5605e4bf-2af8-4442-9d15-4438f54d1ead`::moving from state aborting -> state failed >Thread-2522::DEBUG::2012-12-02 23:04:35,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2522::DEBUG::2012-12-02 23:04:35,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2522::ERROR::2012-12-02 23:04:35,045::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2587::DEBUG::2012-12-02 23:04:43,056::task::568::TaskManager.Task::(_updateState) Task=`7ce5a091-3778-40f1-9d07-dc26661bdfe7`::moving from state init -> state preparing >Thread-2587::INFO::2012-12-02 23:04:43,057::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2587::INFO::2012-12-02 23:04:43,057::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2587::DEBUG::2012-12-02 23:04:43,057::task::1151::TaskManager.Task::(prepare) Task=`7ce5a091-3778-40f1-9d07-dc26661bdfe7`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2587::DEBUG::2012-12-02 23:04:43,057::task::568::TaskManager.Task::(_updateState) Task=`7ce5a091-3778-40f1-9d07-dc26661bdfe7`::moving from state preparing -> state finished >Thread-2587::DEBUG::2012-12-02 23:04:43,057::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2587::DEBUG::2012-12-02 23:04:43,057::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2587::DEBUG::2012-12-02 23:04:43,057::task::957::TaskManager.Task::(_decref) Task=`7ce5a091-3778-40f1-9d07-dc26661bdfe7`::ref 0 aborting False >Thread-2589::DEBUG::2012-12-02 23:04:45,234::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2589::DEBUG::2012-12-02 23:04:45,234::task::568::TaskManager.Task::(_updateState) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::moving from state init -> state preparing >Thread-2589::INFO::2012-12-02 23:04:45,234::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2589::DEBUG::2012-12-02 23:04:45,235::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2589::DEBUG::2012-12-02 23:04:45,235::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2589::DEBUG::2012-12-02 23:04:45,438::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2589::WARNING::2012-12-02 23:04:45,439::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2589::DEBUG::2012-12-02 23:04:45,440::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2589::ERROR::2012-12-02 23:04:45,469::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2589::ERROR::2012-12-02 23:04:45,469::task::833::TaskManager.Task::(_setError) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2589::DEBUG::2012-12-02 23:04:45,469::task::852::TaskManager.Task::(_run) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::Task._run: 3bb04eec-377b-4654-89b5-0611c9fd875d ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2589::DEBUG::2012-12-02 23:04:45,470::task::1177::TaskManager.Task::(stop) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::stopping in state preparing (force False) >Thread-2589::DEBUG::2012-12-02 23:04:45,470::task::957::TaskManager.Task::(_decref) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::ref 1 aborting True >Thread-2589::INFO::2012-12-02 23:04:45,470::task::1134::TaskManager.Task::(prepare) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2589::DEBUG::2012-12-02 23:04:45,470::task::1139::TaskManager.Task::(prepare) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::Prepare: aborted: Error validating master storage domain >Thread-2589::DEBUG::2012-12-02 23:04:45,470::task::957::TaskManager.Task::(_decref) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::ref 0 aborting True >Thread-2589::DEBUG::2012-12-02 23:04:45,470::task::892::TaskManager.Task::(_doAbort) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::Task._doAbort: force False >Thread-2589::DEBUG::2012-12-02 23:04:45,470::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2589::DEBUG::2012-12-02 23:04:45,471::task::568::TaskManager.Task::(_updateState) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::moving from state preparing -> state aborting >Thread-2589::DEBUG::2012-12-02 23:04:45,471::task::523::TaskManager.Task::(__state_aborting) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::_aborting: recover policy none >Thread-2589::DEBUG::2012-12-02 23:04:45,471::task::568::TaskManager.Task::(_updateState) Task=`3bb04eec-377b-4654-89b5-0611c9fd875d`::moving from state aborting -> state failed >Thread-2589::DEBUG::2012-12-02 23:04:45,471::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2589::DEBUG::2012-12-02 23:04:45,471::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2589::ERROR::2012-12-02 23:04:45,471::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2655::DEBUG::2012-12-02 23:04:53,977::task::568::TaskManager.Task::(_updateState) Task=`34401875-f612-4dba-8624-5c9d44e9ff08`::moving from state init -> state preparing >Thread-2655::INFO::2012-12-02 23:04:53,977::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2655::INFO::2012-12-02 23:04:53,978::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2655::DEBUG::2012-12-02 23:04:53,978::task::1151::TaskManager.Task::(prepare) Task=`34401875-f612-4dba-8624-5c9d44e9ff08`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2655::DEBUG::2012-12-02 23:04:53,978::task::568::TaskManager.Task::(_updateState) Task=`34401875-f612-4dba-8624-5c9d44e9ff08`::moving from state preparing -> state finished >Thread-2655::DEBUG::2012-12-02 23:04:53,978::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2655::DEBUG::2012-12-02 23:04:53,978::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2655::DEBUG::2012-12-02 23:04:53,978::task::957::TaskManager.Task::(_decref) Task=`34401875-f612-4dba-8624-5c9d44e9ff08`::ref 0 aborting False >Thread-2657::DEBUG::2012-12-02 23:04:55,642::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2657::DEBUG::2012-12-02 23:04:55,642::task::568::TaskManager.Task::(_updateState) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::moving from state init -> state preparing >Thread-2657::INFO::2012-12-02 23:04:55,643::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2657::DEBUG::2012-12-02 23:04:55,643::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2657::DEBUG::2012-12-02 23:04:55,643::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2657::DEBUG::2012-12-02 23:04:55,845::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2657::WARNING::2012-12-02 23:04:55,846::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2657::DEBUG::2012-12-02 23:04:55,846::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2657::ERROR::2012-12-02 23:04:55,875::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2657::ERROR::2012-12-02 23:04:55,875::task::833::TaskManager.Task::(_setError) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2657::DEBUG::2012-12-02 23:04:55,876::task::852::TaskManager.Task::(_run) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::Task._run: 6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2657::DEBUG::2012-12-02 23:04:55,876::task::1177::TaskManager.Task::(stop) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::stopping in state preparing (force False) >Thread-2657::DEBUG::2012-12-02 23:04:55,876::task::957::TaskManager.Task::(_decref) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::ref 1 aborting True >Thread-2657::INFO::2012-12-02 23:04:55,876::task::1134::TaskManager.Task::(prepare) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2657::DEBUG::2012-12-02 23:04:55,876::task::1139::TaskManager.Task::(prepare) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::Prepare: aborted: Error validating master storage domain >Thread-2657::DEBUG::2012-12-02 23:04:55,876::task::957::TaskManager.Task::(_decref) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::ref 0 aborting True >Thread-2657::DEBUG::2012-12-02 23:04:55,877::task::892::TaskManager.Task::(_doAbort) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::Task._doAbort: force False >Thread-2657::DEBUG::2012-12-02 23:04:55,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2657::DEBUG::2012-12-02 23:04:55,877::task::568::TaskManager.Task::(_updateState) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::moving from state preparing -> state aborting >Thread-2657::DEBUG::2012-12-02 23:04:55,877::task::523::TaskManager.Task::(__state_aborting) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::_aborting: recover policy none >Thread-2657::DEBUG::2012-12-02 23:04:55,877::task::568::TaskManager.Task::(_updateState) Task=`6c3ac8ca-59e2-4d9b-bea8-7179f846b3b4`::moving from state aborting -> state failed >Thread-2657::DEBUG::2012-12-02 23:04:55,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2657::DEBUG::2012-12-02 23:04:55,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2657::ERROR::2012-12-02 23:04:55,877::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2723::DEBUG::2012-12-02 23:05:04,944::task::568::TaskManager.Task::(_updateState) Task=`03bb8325-b31f-4018-a501-819997ec8be9`::moving from state init -> state preparing >Thread-2723::INFO::2012-12-02 23:05:04,944::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2723::INFO::2012-12-02 23:05:04,944::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2723::DEBUG::2012-12-02 23:05:04,944::task::1151::TaskManager.Task::(prepare) Task=`03bb8325-b31f-4018-a501-819997ec8be9`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2723::DEBUG::2012-12-02 23:05:04,944::task::568::TaskManager.Task::(_updateState) Task=`03bb8325-b31f-4018-a501-819997ec8be9`::moving from state preparing -> state finished >Thread-2723::DEBUG::2012-12-02 23:05:04,945::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2723::DEBUG::2012-12-02 23:05:04,945::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2723::DEBUG::2012-12-02 23:05:04,945::task::957::TaskManager.Task::(_decref) Task=`03bb8325-b31f-4018-a501-819997ec8be9`::ref 0 aborting False >Thread-2725::DEBUG::2012-12-02 23:05:06,043::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2725::DEBUG::2012-12-02 23:05:06,044::task::568::TaskManager.Task::(_updateState) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::moving from state init -> state preparing >Thread-2725::INFO::2012-12-02 23:05:06,044::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2725::DEBUG::2012-12-02 23:05:06,044::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2725::DEBUG::2012-12-02 23:05:06,045::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2725::DEBUG::2012-12-02 23:05:06,250::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2725::WARNING::2012-12-02 23:05:06,251::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2725::DEBUG::2012-12-02 23:05:06,252::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2725::ERROR::2012-12-02 23:05:06,286::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2725::ERROR::2012-12-02 23:05:06,287::task::833::TaskManager.Task::(_setError) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2725::DEBUG::2012-12-02 23:05:06,287::task::852::TaskManager.Task::(_run) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::Task._run: 245160b4-9af0-40f1-a082-0fc10c47d38b ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2725::DEBUG::2012-12-02 23:05:06,287::task::1177::TaskManager.Task::(stop) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::stopping in state preparing (force False) >Thread-2725::DEBUG::2012-12-02 23:05:06,287::task::957::TaskManager.Task::(_decref) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::ref 1 aborting True >Thread-2725::INFO::2012-12-02 23:05:06,287::task::1134::TaskManager.Task::(prepare) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2725::DEBUG::2012-12-02 23:05:06,288::task::1139::TaskManager.Task::(prepare) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::Prepare: aborted: Error validating master storage domain >Thread-2725::DEBUG::2012-12-02 23:05:06,288::task::957::TaskManager.Task::(_decref) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::ref 0 aborting True >Thread-2725::DEBUG::2012-12-02 23:05:06,288::task::892::TaskManager.Task::(_doAbort) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::Task._doAbort: force False >Thread-2725::DEBUG::2012-12-02 23:05:06,288::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2725::DEBUG::2012-12-02 23:05:06,288::task::568::TaskManager.Task::(_updateState) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::moving from state preparing -> state aborting >Thread-2725::DEBUG::2012-12-02 23:05:06,288::task::523::TaskManager.Task::(__state_aborting) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::_aborting: recover policy none >Thread-2725::DEBUG::2012-12-02 23:05:06,288::task::568::TaskManager.Task::(_updateState) Task=`245160b4-9af0-40f1-a082-0fc10c47d38b`::moving from state aborting -> state failed >Thread-2725::DEBUG::2012-12-02 23:05:06,288::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2725::DEBUG::2012-12-02 23:05:06,289::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2725::ERROR::2012-12-02 23:05:06,289::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2791::DEBUG::2012-12-02 23:05:15,901::task::568::TaskManager.Task::(_updateState) Task=`c4fcdd4a-c0f3-49ac-99e2-65e50d86d6ec`::moving from state init -> state preparing >Thread-2791::INFO::2012-12-02 23:05:15,901::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2791::INFO::2012-12-02 23:05:15,901::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2791::DEBUG::2012-12-02 23:05:15,901::task::1151::TaskManager.Task::(prepare) Task=`c4fcdd4a-c0f3-49ac-99e2-65e50d86d6ec`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2791::DEBUG::2012-12-02 23:05:15,901::task::568::TaskManager.Task::(_updateState) Task=`c4fcdd4a-c0f3-49ac-99e2-65e50d86d6ec`::moving from state preparing -> state finished >Thread-2791::DEBUG::2012-12-02 23:05:15,902::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2791::DEBUG::2012-12-02 23:05:15,902::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2791::DEBUG::2012-12-02 23:05:15,902::task::957::TaskManager.Task::(_decref) Task=`c4fcdd4a-c0f3-49ac-99e2-65e50d86d6ec`::ref 0 aborting False >Thread-2793::DEBUG::2012-12-02 23:05:16,451::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2793::DEBUG::2012-12-02 23:05:16,451::task::568::TaskManager.Task::(_updateState) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::moving from state init -> state preparing >Thread-2793::INFO::2012-12-02 23:05:16,451::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2793::DEBUG::2012-12-02 23:05:16,452::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2793::DEBUG::2012-12-02 23:05:16,452::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2793::DEBUG::2012-12-02 23:05:16,654::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2793::WARNING::2012-12-02 23:05:16,655::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2793::DEBUG::2012-12-02 23:05:16,656::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2793::ERROR::2012-12-02 23:05:16,684::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2793::ERROR::2012-12-02 23:05:16,685::task::833::TaskManager.Task::(_setError) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2793::DEBUG::2012-12-02 23:05:16,685::task::852::TaskManager.Task::(_run) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::Task._run: f99da64c-7ebf-49dc-971c-524efd070d8e ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2793::DEBUG::2012-12-02 23:05:16,685::task::1177::TaskManager.Task::(stop) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::stopping in state preparing (force False) >Thread-2793::DEBUG::2012-12-02 23:05:16,685::task::957::TaskManager.Task::(_decref) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::ref 1 aborting True >Thread-2793::INFO::2012-12-02 23:05:16,685::task::1134::TaskManager.Task::(prepare) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2793::DEBUG::2012-12-02 23:05:16,686::task::1139::TaskManager.Task::(prepare) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::Prepare: aborted: Error validating master storage domain >Thread-2793::DEBUG::2012-12-02 23:05:16,686::task::957::TaskManager.Task::(_decref) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::ref 0 aborting True >Thread-2793::DEBUG::2012-12-02 23:05:16,686::task::892::TaskManager.Task::(_doAbort) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::Task._doAbort: force False >Thread-2793::DEBUG::2012-12-02 23:05:16,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2793::DEBUG::2012-12-02 23:05:16,686::task::568::TaskManager.Task::(_updateState) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::moving from state preparing -> state aborting >Thread-2793::DEBUG::2012-12-02 23:05:16,686::task::523::TaskManager.Task::(__state_aborting) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::_aborting: recover policy none >Thread-2793::DEBUG::2012-12-02 23:05:16,686::task::568::TaskManager.Task::(_updateState) Task=`f99da64c-7ebf-49dc-971c-524efd070d8e`::moving from state aborting -> state failed >Thread-2793::DEBUG::2012-12-02 23:05:16,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2793::DEBUG::2012-12-02 23:05:16,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2793::ERROR::2012-12-02 23:05:16,687::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2859::DEBUG::2012-12-02 23:05:26,837::task::568::TaskManager.Task::(_updateState) Task=`4fc06fd8-9803-4c4c-97d9-fb69e279e5c8`::moving from state init -> state preparing >Thread-2859::INFO::2012-12-02 23:05:26,838::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2859::INFO::2012-12-02 23:05:26,838::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2859::DEBUG::2012-12-02 23:05:26,838::task::1151::TaskManager.Task::(prepare) Task=`4fc06fd8-9803-4c4c-97d9-fb69e279e5c8`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2859::DEBUG::2012-12-02 23:05:26,838::task::568::TaskManager.Task::(_updateState) Task=`4fc06fd8-9803-4c4c-97d9-fb69e279e5c8`::moving from state preparing -> state finished >Thread-2859::DEBUG::2012-12-02 23:05:26,838::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2859::DEBUG::2012-12-02 23:05:26,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2859::DEBUG::2012-12-02 23:05:26,838::task::957::TaskManager.Task::(_decref) Task=`4fc06fd8-9803-4c4c-97d9-fb69e279e5c8`::ref 0 aborting False >Thread-2860::DEBUG::2012-12-02 23:05:26,844::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2860::DEBUG::2012-12-02 23:05:26,845::task::568::TaskManager.Task::(_updateState) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::moving from state init -> state preparing >Thread-2860::INFO::2012-12-02 23:05:26,845::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2860::DEBUG::2012-12-02 23:05:26,845::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2860::DEBUG::2012-12-02 23:05:26,845::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2860::DEBUG::2012-12-02 23:05:27,050::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2860::WARNING::2012-12-02 23:05:27,052::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2860::DEBUG::2012-12-02 23:05:27,052::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2860::ERROR::2012-12-02 23:05:27,082::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2860::ERROR::2012-12-02 23:05:27,083::task::833::TaskManager.Task::(_setError) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2860::DEBUG::2012-12-02 23:05:27,083::task::852::TaskManager.Task::(_run) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::Task._run: bc7f1daa-de58-4da1-9141-f1a867a3d3c9 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2860::DEBUG::2012-12-02 23:05:27,083::task::1177::TaskManager.Task::(stop) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::stopping in state preparing (force False) >Thread-2860::DEBUG::2012-12-02 23:05:27,083::task::957::TaskManager.Task::(_decref) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::ref 1 aborting True >Thread-2860::INFO::2012-12-02 23:05:27,083::task::1134::TaskManager.Task::(prepare) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2860::DEBUG::2012-12-02 23:05:27,083::task::1139::TaskManager.Task::(prepare) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::Prepare: aborted: Error validating master storage domain >Thread-2860::DEBUG::2012-12-02 23:05:27,084::task::957::TaskManager.Task::(_decref) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::ref 0 aborting True >Thread-2860::DEBUG::2012-12-02 23:05:27,084::task::892::TaskManager.Task::(_doAbort) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::Task._doAbort: force False >Thread-2860::DEBUG::2012-12-02 23:05:27,084::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2860::DEBUG::2012-12-02 23:05:27,084::task::568::TaskManager.Task::(_updateState) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::moving from state preparing -> state aborting >Thread-2860::DEBUG::2012-12-02 23:05:27,084::task::523::TaskManager.Task::(__state_aborting) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::_aborting: recover policy none >Thread-2860::DEBUG::2012-12-02 23:05:27,084::task::568::TaskManager.Task::(_updateState) Task=`bc7f1daa-de58-4da1-9141-f1a867a3d3c9`::moving from state aborting -> state failed >Thread-2860::DEBUG::2012-12-02 23:05:27,084::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2860::DEBUG::2012-12-02 23:05:27,084::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2860::ERROR::2012-12-02 23:05:27,085::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2927::DEBUG::2012-12-02 23:05:37,233::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2927::DEBUG::2012-12-02 23:05:37,233::task::568::TaskManager.Task::(_updateState) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::moving from state init -> state preparing >Thread-2927::INFO::2012-12-02 23:05:37,234::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2927::DEBUG::2012-12-02 23:05:37,234::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2927::DEBUG::2012-12-02 23:05:37,234::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2927::DEBUG::2012-12-02 23:05:37,439::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2927::WARNING::2012-12-02 23:05:37,440::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2927::DEBUG::2012-12-02 23:05:37,440::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2927::ERROR::2012-12-02 23:05:37,470::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2927::ERROR::2012-12-02 23:05:37,470::task::833::TaskManager.Task::(_setError) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2927::DEBUG::2012-12-02 23:05:37,470::task::852::TaskManager.Task::(_run) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::Task._run: ec46f0aa-801f-4099-9100-22ecc1f91de0 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2927::DEBUG::2012-12-02 23:05:37,470::task::1177::TaskManager.Task::(stop) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::stopping in state preparing (force False) >Thread-2927::DEBUG::2012-12-02 23:05:37,471::task::957::TaskManager.Task::(_decref) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::ref 1 aborting True >Thread-2927::INFO::2012-12-02 23:05:37,471::task::1134::TaskManager.Task::(prepare) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2927::DEBUG::2012-12-02 23:05:37,471::task::1139::TaskManager.Task::(prepare) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::Prepare: aborted: Error validating master storage domain >Thread-2927::DEBUG::2012-12-02 23:05:37,471::task::957::TaskManager.Task::(_decref) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::ref 0 aborting True >Thread-2927::DEBUG::2012-12-02 23:05:37,471::task::892::TaskManager.Task::(_doAbort) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::Task._doAbort: force False >Thread-2927::DEBUG::2012-12-02 23:05:37,471::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2927::DEBUG::2012-12-02 23:05:37,471::task::568::TaskManager.Task::(_updateState) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::moving from state preparing -> state aborting >Thread-2927::DEBUG::2012-12-02 23:05:37,471::task::523::TaskManager.Task::(__state_aborting) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::_aborting: recover policy none >Thread-2927::DEBUG::2012-12-02 23:05:37,472::task::568::TaskManager.Task::(_updateState) Task=`ec46f0aa-801f-4099-9100-22ecc1f91de0`::moving from state aborting -> state failed >Thread-2927::DEBUG::2012-12-02 23:05:37,472::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2927::DEBUG::2012-12-02 23:05:37,472::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2927::ERROR::2012-12-02 23:05:37,472::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-2989::DEBUG::2012-12-02 23:05:37,746::task::568::TaskManager.Task::(_updateState) Task=`a06692e6-367f-46a8-b3ec-410376e8a19b`::moving from state init -> state preparing >Thread-2989::INFO::2012-12-02 23:05:37,747::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2989::INFO::2012-12-02 23:05:37,747::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2989::DEBUG::2012-12-02 23:05:37,747::task::1151::TaskManager.Task::(prepare) Task=`a06692e6-367f-46a8-b3ec-410376e8a19b`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-2989::DEBUG::2012-12-02 23:05:37,747::task::568::TaskManager.Task::(_updateState) Task=`a06692e6-367f-46a8-b3ec-410376e8a19b`::moving from state preparing -> state finished >Thread-2989::DEBUG::2012-12-02 23:05:37,747::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2989::DEBUG::2012-12-02 23:05:37,747::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2989::DEBUG::2012-12-02 23:05:37,747::task::957::TaskManager.Task::(_decref) Task=`a06692e6-367f-46a8-b3ec-410376e8a19b`::ref 0 aborting False >Thread-2995::DEBUG::2012-12-02 23:05:47,625::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2995::DEBUG::2012-12-02 23:05:47,625::task::568::TaskManager.Task::(_updateState) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::moving from state init -> state preparing >Thread-2995::INFO::2012-12-02 23:05:47,625::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2995::DEBUG::2012-12-02 23:05:47,625::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2995::DEBUG::2012-12-02 23:05:47,626::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2995::DEBUG::2012-12-02 23:05:47,827::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2995::WARNING::2012-12-02 23:05:47,828::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2995::DEBUG::2012-12-02 23:05:47,829::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2995::ERROR::2012-12-02 23:05:47,858::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2995::ERROR::2012-12-02 23:05:47,859::task::833::TaskManager.Task::(_setError) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-2995::DEBUG::2012-12-02 23:05:47,859::task::852::TaskManager.Task::(_run) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::Task._run: 700a05e4-2968-409a-870b-1f9cd0ee3717 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2995::DEBUG::2012-12-02 23:05:47,859::task::1177::TaskManager.Task::(stop) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::stopping in state preparing (force False) >Thread-2995::DEBUG::2012-12-02 23:05:47,859::task::957::TaskManager.Task::(_decref) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::ref 1 aborting True >Thread-2995::INFO::2012-12-02 23:05:47,859::task::1134::TaskManager.Task::(prepare) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-2995::DEBUG::2012-12-02 23:05:47,859::task::1139::TaskManager.Task::(prepare) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::Prepare: aborted: Error validating master storage domain >Thread-2995::DEBUG::2012-12-02 23:05:47,859::task::957::TaskManager.Task::(_decref) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::ref 0 aborting True >Thread-2995::DEBUG::2012-12-02 23:05:47,860::task::892::TaskManager.Task::(_doAbort) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::Task._doAbort: force False >Thread-2995::DEBUG::2012-12-02 23:05:47,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2995::DEBUG::2012-12-02 23:05:47,860::task::568::TaskManager.Task::(_updateState) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::moving from state preparing -> state aborting >Thread-2995::DEBUG::2012-12-02 23:05:47,860::task::523::TaskManager.Task::(__state_aborting) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::_aborting: recover policy none >Thread-2995::DEBUG::2012-12-02 23:05:47,860::task::568::TaskManager.Task::(_updateState) Task=`700a05e4-2968-409a-870b-1f9cd0ee3717`::moving from state aborting -> state failed >Thread-2995::DEBUG::2012-12-02 23:05:47,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2995::DEBUG::2012-12-02 23:05:47,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2995::ERROR::2012-12-02 23:05:47,860::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3057::DEBUG::2012-12-02 23:05:48,650::task::568::TaskManager.Task::(_updateState) Task=`1250d05d-8a29-4004-9a51-feb9904f5141`::moving from state init -> state preparing >Thread-3057::INFO::2012-12-02 23:05:48,651::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3057::INFO::2012-12-02 23:05:48,651::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3057::DEBUG::2012-12-02 23:05:48,651::task::1151::TaskManager.Task::(prepare) Task=`1250d05d-8a29-4004-9a51-feb9904f5141`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3057::DEBUG::2012-12-02 23:05:48,651::task::568::TaskManager.Task::(_updateState) Task=`1250d05d-8a29-4004-9a51-feb9904f5141`::moving from state preparing -> state finished >Thread-3057::DEBUG::2012-12-02 23:05:48,651::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3057::DEBUG::2012-12-02 23:05:48,651::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3057::DEBUG::2012-12-02 23:05:48,651::task::957::TaskManager.Task::(_decref) Task=`1250d05d-8a29-4004-9a51-feb9904f5141`::ref 0 aborting False >Thread-3063::DEBUG::2012-12-02 23:05:58,015::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3063::DEBUG::2012-12-02 23:05:58,015::task::568::TaskManager.Task::(_updateState) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::moving from state init -> state preparing >Thread-3063::INFO::2012-12-02 23:05:58,016::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3063::DEBUG::2012-12-02 23:05:58,016::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3063::DEBUG::2012-12-02 23:05:58,016::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3063::DEBUG::2012-12-02 23:05:58,220::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3063::WARNING::2012-12-02 23:05:58,222::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3063::DEBUG::2012-12-02 23:05:58,222::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3063::ERROR::2012-12-02 23:05:58,250::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3063::ERROR::2012-12-02 23:05:58,250::task::833::TaskManager.Task::(_setError) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3063::DEBUG::2012-12-02 23:05:58,251::task::852::TaskManager.Task::(_run) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::Task._run: ef4d86af-c716-4516-af9d-5d9883bf5165 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3063::DEBUG::2012-12-02 23:05:58,251::task::1177::TaskManager.Task::(stop) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::stopping in state preparing (force False) >Thread-3063::DEBUG::2012-12-02 23:05:58,251::task::957::TaskManager.Task::(_decref) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::ref 1 aborting True >Thread-3063::INFO::2012-12-02 23:05:58,251::task::1134::TaskManager.Task::(prepare) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3063::DEBUG::2012-12-02 23:05:58,251::task::1139::TaskManager.Task::(prepare) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::Prepare: aborted: Error validating master storage domain >Thread-3063::DEBUG::2012-12-02 23:05:58,251::task::957::TaskManager.Task::(_decref) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::ref 0 aborting True >Thread-3063::DEBUG::2012-12-02 23:05:58,252::task::892::TaskManager.Task::(_doAbort) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::Task._doAbort: force False >Thread-3063::DEBUG::2012-12-02 23:05:58,252::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3063::DEBUG::2012-12-02 23:05:58,252::task::568::TaskManager.Task::(_updateState) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::moving from state preparing -> state aborting >Thread-3063::DEBUG::2012-12-02 23:05:58,252::task::523::TaskManager.Task::(__state_aborting) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::_aborting: recover policy none >Thread-3063::DEBUG::2012-12-02 23:05:58,252::task::568::TaskManager.Task::(_updateState) Task=`ef4d86af-c716-4516-af9d-5d9883bf5165`::moving from state aborting -> state failed >Thread-3063::DEBUG::2012-12-02 23:05:58,252::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3063::DEBUG::2012-12-02 23:05:58,252::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3063::ERROR::2012-12-02 23:05:58,252::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3125::DEBUG::2012-12-02 23:05:59,567::task::568::TaskManager.Task::(_updateState) Task=`8775da3f-5051-479e-8ed0-b8f890740d83`::moving from state init -> state preparing >Thread-3125::INFO::2012-12-02 23:05:59,567::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3125::INFO::2012-12-02 23:05:59,568::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3125::DEBUG::2012-12-02 23:05:59,568::task::1151::TaskManager.Task::(prepare) Task=`8775da3f-5051-479e-8ed0-b8f890740d83`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3125::DEBUG::2012-12-02 23:05:59,568::task::568::TaskManager.Task::(_updateState) Task=`8775da3f-5051-479e-8ed0-b8f890740d83`::moving from state preparing -> state finished >Thread-3125::DEBUG::2012-12-02 23:05:59,568::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3125::DEBUG::2012-12-02 23:05:59,568::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3125::DEBUG::2012-12-02 23:05:59,568::task::957::TaskManager.Task::(_decref) Task=`8775da3f-5051-479e-8ed0-b8f890740d83`::ref 0 aborting False >Thread-3131::DEBUG::2012-12-02 23:06:08,402::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3131::DEBUG::2012-12-02 23:06:08,402::task::568::TaskManager.Task::(_updateState) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::moving from state init -> state preparing >Thread-3131::INFO::2012-12-02 23:06:08,402::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3131::DEBUG::2012-12-02 23:06:08,402::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3131::DEBUG::2012-12-02 23:06:08,403::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3131::DEBUG::2012-12-02 23:06:08,606::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3131::WARNING::2012-12-02 23:06:08,607::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3131::DEBUG::2012-12-02 23:06:08,608::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3131::ERROR::2012-12-02 23:06:08,641::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3131::ERROR::2012-12-02 23:06:08,642::task::833::TaskManager.Task::(_setError) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3131::DEBUG::2012-12-02 23:06:08,642::task::852::TaskManager.Task::(_run) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::Task._run: 3ebda6d6-fb20-44da-8059-a3c59b5b345e ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3131::DEBUG::2012-12-02 23:06:08,642::task::1177::TaskManager.Task::(stop) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::stopping in state preparing (force False) >Thread-3131::DEBUG::2012-12-02 23:06:08,642::task::957::TaskManager.Task::(_decref) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::ref 1 aborting True >Thread-3131::INFO::2012-12-02 23:06:08,642::task::1134::TaskManager.Task::(prepare) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3131::DEBUG::2012-12-02 23:06:08,643::task::1139::TaskManager.Task::(prepare) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::Prepare: aborted: Error validating master storage domain >Thread-3131::DEBUG::2012-12-02 23:06:08,643::task::957::TaskManager.Task::(_decref) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::ref 0 aborting True >Thread-3131::DEBUG::2012-12-02 23:06:08,643::task::892::TaskManager.Task::(_doAbort) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::Task._doAbort: force False >Thread-3131::DEBUG::2012-12-02 23:06:08,643::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3131::DEBUG::2012-12-02 23:06:08,643::task::568::TaskManager.Task::(_updateState) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::moving from state preparing -> state aborting >Thread-3131::DEBUG::2012-12-02 23:06:08,643::task::523::TaskManager.Task::(__state_aborting) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::_aborting: recover policy none >Thread-3131::DEBUG::2012-12-02 23:06:08,643::task::568::TaskManager.Task::(_updateState) Task=`3ebda6d6-fb20-44da-8059-a3c59b5b345e`::moving from state aborting -> state failed >Thread-3131::DEBUG::2012-12-02 23:06:08,643::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3131::DEBUG::2012-12-02 23:06:08,644::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3131::ERROR::2012-12-02 23:06:08,644::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3193::DEBUG::2012-12-02 23:06:10,480::task::568::TaskManager.Task::(_updateState) Task=`ca712266-75b4-408a-bec7-d3c49d964520`::moving from state init -> state preparing >Thread-3193::INFO::2012-12-02 23:06:10,480::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3193::INFO::2012-12-02 23:06:10,480::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3193::DEBUG::2012-12-02 23:06:10,480::task::1151::TaskManager.Task::(prepare) Task=`ca712266-75b4-408a-bec7-d3c49d964520`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3193::DEBUG::2012-12-02 23:06:10,480::task::568::TaskManager.Task::(_updateState) Task=`ca712266-75b4-408a-bec7-d3c49d964520`::moving from state preparing -> state finished >Thread-3193::DEBUG::2012-12-02 23:06:10,480::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3193::DEBUG::2012-12-02 23:06:10,481::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3193::DEBUG::2012-12-02 23:06:10,481::task::957::TaskManager.Task::(_decref) Task=`ca712266-75b4-408a-bec7-d3c49d964520`::ref 0 aborting False >Thread-3198::DEBUG::2012-12-02 23:06:18,790::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3198::DEBUG::2012-12-02 23:06:18,791::task::568::TaskManager.Task::(_updateState) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::moving from state init -> state preparing >Thread-3198::INFO::2012-12-02 23:06:18,791::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3198::DEBUG::2012-12-02 23:06:18,791::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3198::DEBUG::2012-12-02 23:06:18,792::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3198::DEBUG::2012-12-02 23:06:18,995::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3198::WARNING::2012-12-02 23:06:18,996::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3198::DEBUG::2012-12-02 23:06:18,996::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3198::ERROR::2012-12-02 23:06:19,027::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3198::ERROR::2012-12-02 23:06:19,027::task::833::TaskManager.Task::(_setError) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3198::DEBUG::2012-12-02 23:06:19,027::task::852::TaskManager.Task::(_run) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::Task._run: a905c81d-71d0-48b0-b2bf-7d98a0105e0e ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3198::DEBUG::2012-12-02 23:06:19,028::task::1177::TaskManager.Task::(stop) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::stopping in state preparing (force False) >Thread-3198::DEBUG::2012-12-02 23:06:19,028::task::957::TaskManager.Task::(_decref) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::ref 1 aborting True >Thread-3198::INFO::2012-12-02 23:06:19,028::task::1134::TaskManager.Task::(prepare) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3198::DEBUG::2012-12-02 23:06:19,028::task::1139::TaskManager.Task::(prepare) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::Prepare: aborted: Error validating master storage domain >Thread-3198::DEBUG::2012-12-02 23:06:19,028::task::957::TaskManager.Task::(_decref) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::ref 0 aborting True >Thread-3198::DEBUG::2012-12-02 23:06:19,028::task::892::TaskManager.Task::(_doAbort) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::Task._doAbort: force False >Thread-3198::DEBUG::2012-12-02 23:06:19,028::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3198::DEBUG::2012-12-02 23:06:19,028::task::568::TaskManager.Task::(_updateState) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::moving from state preparing -> state aborting >Thread-3198::DEBUG::2012-12-02 23:06:19,029::task::523::TaskManager.Task::(__state_aborting) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::_aborting: recover policy none >Thread-3198::DEBUG::2012-12-02 23:06:19,029::task::568::TaskManager.Task::(_updateState) Task=`a905c81d-71d0-48b0-b2bf-7d98a0105e0e`::moving from state aborting -> state failed >Thread-3198::DEBUG::2012-12-02 23:06:19,029::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3198::DEBUG::2012-12-02 23:06:19,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3198::ERROR::2012-12-02 23:06:19,029::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3261::DEBUG::2012-12-02 23:06:21,585::task::568::TaskManager.Task::(_updateState) Task=`c41da080-1e49-439f-aec4-4b24b817fbe6`::moving from state init -> state preparing >Thread-3261::INFO::2012-12-02 23:06:21,586::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3261::INFO::2012-12-02 23:06:21,586::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3261::DEBUG::2012-12-02 23:06:21,586::task::1151::TaskManager.Task::(prepare) Task=`c41da080-1e49-439f-aec4-4b24b817fbe6`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3261::DEBUG::2012-12-02 23:06:21,586::task::568::TaskManager.Task::(_updateState) Task=`c41da080-1e49-439f-aec4-4b24b817fbe6`::moving from state preparing -> state finished >Thread-3261::DEBUG::2012-12-02 23:06:21,586::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3261::DEBUG::2012-12-02 23:06:21,586::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3261::DEBUG::2012-12-02 23:06:21,587::task::957::TaskManager.Task::(_decref) Task=`c41da080-1e49-439f-aec4-4b24b817fbe6`::ref 0 aborting False >Thread-3266::DEBUG::2012-12-02 23:06:29,193::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3266::DEBUG::2012-12-02 23:06:29,193::task::568::TaskManager.Task::(_updateState) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::moving from state init -> state preparing >Thread-3266::INFO::2012-12-02 23:06:29,194::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3266::DEBUG::2012-12-02 23:06:29,194::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3266::DEBUG::2012-12-02 23:06:29,194::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3266::DEBUG::2012-12-02 23:06:29,400::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3266::WARNING::2012-12-02 23:06:29,402::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3266::DEBUG::2012-12-02 23:06:29,402::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3266::ERROR::2012-12-02 23:06:29,430::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3266::ERROR::2012-12-02 23:06:29,431::task::833::TaskManager.Task::(_setError) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3266::DEBUG::2012-12-02 23:06:29,431::task::852::TaskManager.Task::(_run) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::Task._run: d9d4a28f-84b5-44ef-8a22-ecf02a1540ad ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3266::DEBUG::2012-12-02 23:06:29,431::task::1177::TaskManager.Task::(stop) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::stopping in state preparing (force False) >Thread-3266::DEBUG::2012-12-02 23:06:29,431::task::957::TaskManager.Task::(_decref) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::ref 1 aborting True >Thread-3266::INFO::2012-12-02 23:06:29,431::task::1134::TaskManager.Task::(prepare) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3266::DEBUG::2012-12-02 23:06:29,432::task::1139::TaskManager.Task::(prepare) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::Prepare: aborted: Error validating master storage domain >Thread-3266::DEBUG::2012-12-02 23:06:29,432::task::957::TaskManager.Task::(_decref) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::ref 0 aborting True >Thread-3266::DEBUG::2012-12-02 23:06:29,432::task::892::TaskManager.Task::(_doAbort) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::Task._doAbort: force False >Thread-3266::DEBUG::2012-12-02 23:06:29,432::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3266::DEBUG::2012-12-02 23:06:29,432::task::568::TaskManager.Task::(_updateState) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::moving from state preparing -> state aborting >Thread-3266::DEBUG::2012-12-02 23:06:29,432::task::523::TaskManager.Task::(__state_aborting) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::_aborting: recover policy none >Thread-3266::DEBUG::2012-12-02 23:06:29,432::task::568::TaskManager.Task::(_updateState) Task=`d9d4a28f-84b5-44ef-8a22-ecf02a1540ad`::moving from state aborting -> state failed >Thread-3266::DEBUG::2012-12-02 23:06:29,432::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3266::DEBUG::2012-12-02 23:06:29,433::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3266::ERROR::2012-12-02 23:06:29,433::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3329::DEBUG::2012-12-02 23:06:32,525::task::568::TaskManager.Task::(_updateState) Task=`777eba12-55da-4073-ac37-a0493af957e2`::moving from state init -> state preparing >Thread-3329::INFO::2012-12-02 23:06:32,526::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3329::INFO::2012-12-02 23:06:32,526::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3329::DEBUG::2012-12-02 23:06:32,526::task::1151::TaskManager.Task::(prepare) Task=`777eba12-55da-4073-ac37-a0493af957e2`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3329::DEBUG::2012-12-02 23:06:32,526::task::568::TaskManager.Task::(_updateState) Task=`777eba12-55da-4073-ac37-a0493af957e2`::moving from state preparing -> state finished >Thread-3329::DEBUG::2012-12-02 23:06:32,526::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3329::DEBUG::2012-12-02 23:06:32,526::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3329::DEBUG::2012-12-02 23:06:32,526::task::957::TaskManager.Task::(_decref) Task=`777eba12-55da-4073-ac37-a0493af957e2`::ref 0 aborting False >Thread-3334::DEBUG::2012-12-02 23:06:39,580::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3334::DEBUG::2012-12-02 23:06:39,581::task::568::TaskManager.Task::(_updateState) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::moving from state init -> state preparing >Thread-3334::INFO::2012-12-02 23:06:39,581::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3334::DEBUG::2012-12-02 23:06:39,581::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3334::DEBUG::2012-12-02 23:06:39,581::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3334::DEBUG::2012-12-02 23:06:39,784::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3334::WARNING::2012-12-02 23:06:39,786::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3334::DEBUG::2012-12-02 23:06:39,786::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3334::ERROR::2012-12-02 23:06:39,815::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3334::ERROR::2012-12-02 23:06:39,815::task::833::TaskManager.Task::(_setError) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3334::DEBUG::2012-12-02 23:06:39,816::task::852::TaskManager.Task::(_run) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::Task._run: bb78cf84-1877-44f7-9263-6d8332018706 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3334::DEBUG::2012-12-02 23:06:39,816::task::1177::TaskManager.Task::(stop) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::stopping in state preparing (force False) >Thread-3334::DEBUG::2012-12-02 23:06:39,816::task::957::TaskManager.Task::(_decref) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::ref 1 aborting True >Thread-3334::INFO::2012-12-02 23:06:39,816::task::1134::TaskManager.Task::(prepare) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3334::DEBUG::2012-12-02 23:06:39,816::task::1139::TaskManager.Task::(prepare) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::Prepare: aborted: Error validating master storage domain >Thread-3334::DEBUG::2012-12-02 23:06:39,816::task::957::TaskManager.Task::(_decref) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::ref 0 aborting True >Thread-3334::DEBUG::2012-12-02 23:06:39,816::task::892::TaskManager.Task::(_doAbort) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::Task._doAbort: force False >Thread-3334::DEBUG::2012-12-02 23:06:39,816::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3334::DEBUG::2012-12-02 23:06:39,817::task::568::TaskManager.Task::(_updateState) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::moving from state preparing -> state aborting >Thread-3334::DEBUG::2012-12-02 23:06:39,817::task::523::TaskManager.Task::(__state_aborting) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::_aborting: recover policy none >Thread-3334::DEBUG::2012-12-02 23:06:39,817::task::568::TaskManager.Task::(_updateState) Task=`bb78cf84-1877-44f7-9263-6d8332018706`::moving from state aborting -> state failed >Thread-3334::DEBUG::2012-12-02 23:06:39,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3334::DEBUG::2012-12-02 23:06:39,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3334::ERROR::2012-12-02 23:06:39,817::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3397::DEBUG::2012-12-02 23:06:43,425::task::568::TaskManager.Task::(_updateState) Task=`d28dc6a0-83c7-44f4-8fd1-0083b18c2280`::moving from state init -> state preparing >Thread-3397::INFO::2012-12-02 23:06:43,425::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3397::INFO::2012-12-02 23:06:43,425::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3397::DEBUG::2012-12-02 23:06:43,426::task::1151::TaskManager.Task::(prepare) Task=`d28dc6a0-83c7-44f4-8fd1-0083b18c2280`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3397::DEBUG::2012-12-02 23:06:43,426::task::568::TaskManager.Task::(_updateState) Task=`d28dc6a0-83c7-44f4-8fd1-0083b18c2280`::moving from state preparing -> state finished >Thread-3397::DEBUG::2012-12-02 23:06:43,426::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3397::DEBUG::2012-12-02 23:06:43,426::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3397::DEBUG::2012-12-02 23:06:43,426::task::957::TaskManager.Task::(_decref) Task=`d28dc6a0-83c7-44f4-8fd1-0083b18c2280`::ref 0 aborting False >Thread-3401::DEBUG::2012-12-02 23:06:49,988::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3401::DEBUG::2012-12-02 23:06:49,989::task::568::TaskManager.Task::(_updateState) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::moving from state init -> state preparing >Thread-3401::INFO::2012-12-02 23:06:49,989::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3401::DEBUG::2012-12-02 23:06:49,989::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3401::DEBUG::2012-12-02 23:06:49,989::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3401::DEBUG::2012-12-02 23:06:50,194::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3401::WARNING::2012-12-02 23:06:50,196::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3401::DEBUG::2012-12-02 23:06:50,196::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3401::ERROR::2012-12-02 23:06:50,227::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3401::ERROR::2012-12-02 23:06:50,227::task::833::TaskManager.Task::(_setError) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3401::DEBUG::2012-12-02 23:06:50,227::task::852::TaskManager.Task::(_run) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::Task._run: 1b3056cf-3f4a-4516-904a-f9ea1fbe5461 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3401::DEBUG::2012-12-02 23:06:50,227::task::1177::TaskManager.Task::(stop) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::stopping in state preparing (force False) >Thread-3401::DEBUG::2012-12-02 23:06:50,228::task::957::TaskManager.Task::(_decref) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::ref 1 aborting True >Thread-3401::INFO::2012-12-02 23:06:50,228::task::1134::TaskManager.Task::(prepare) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3401::DEBUG::2012-12-02 23:06:50,228::task::1139::TaskManager.Task::(prepare) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::Prepare: aborted: Error validating master storage domain >Thread-3401::DEBUG::2012-12-02 23:06:50,228::task::957::TaskManager.Task::(_decref) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::ref 0 aborting True >Thread-3401::DEBUG::2012-12-02 23:06:50,228::task::892::TaskManager.Task::(_doAbort) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::Task._doAbort: force False >Thread-3401::DEBUG::2012-12-02 23:06:50,228::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3401::DEBUG::2012-12-02 23:06:50,228::task::568::TaskManager.Task::(_updateState) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::moving from state preparing -> state aborting >Thread-3401::DEBUG::2012-12-02 23:06:50,228::task::523::TaskManager.Task::(__state_aborting) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::_aborting: recover policy none >Thread-3401::DEBUG::2012-12-02 23:06:50,229::task::568::TaskManager.Task::(_updateState) Task=`1b3056cf-3f4a-4516-904a-f9ea1fbe5461`::moving from state aborting -> state failed >Thread-3401::DEBUG::2012-12-02 23:06:50,229::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3401::DEBUG::2012-12-02 23:06:50,229::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3401::ERROR::2012-12-02 23:06:50,229::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3465::DEBUG::2012-12-02 23:06:54,348::task::568::TaskManager.Task::(_updateState) Task=`65b69cdf-eb81-44f1-9519-0a7a0249d83c`::moving from state init -> state preparing >Thread-3465::INFO::2012-12-02 23:06:54,348::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3465::INFO::2012-12-02 23:06:54,349::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3465::DEBUG::2012-12-02 23:06:54,349::task::1151::TaskManager.Task::(prepare) Task=`65b69cdf-eb81-44f1-9519-0a7a0249d83c`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3465::DEBUG::2012-12-02 23:06:54,349::task::568::TaskManager.Task::(_updateState) Task=`65b69cdf-eb81-44f1-9519-0a7a0249d83c`::moving from state preparing -> state finished >Thread-3465::DEBUG::2012-12-02 23:06:54,349::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3465::DEBUG::2012-12-02 23:06:54,349::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3465::DEBUG::2012-12-02 23:06:54,349::task::957::TaskManager.Task::(_decref) Task=`65b69cdf-eb81-44f1-9519-0a7a0249d83c`::ref 0 aborting False >Thread-3469::DEBUG::2012-12-02 23:07:00,386::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3469::DEBUG::2012-12-02 23:07:00,387::task::568::TaskManager.Task::(_updateState) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::moving from state init -> state preparing >Thread-3469::INFO::2012-12-02 23:07:00,387::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3469::DEBUG::2012-12-02 23:07:00,387::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3469::DEBUG::2012-12-02 23:07:00,388::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3469::DEBUG::2012-12-02 23:07:00,592::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3469::WARNING::2012-12-02 23:07:00,594::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3469::DEBUG::2012-12-02 23:07:00,594::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3469::ERROR::2012-12-02 23:07:00,624::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3469::ERROR::2012-12-02 23:07:00,624::task::833::TaskManager.Task::(_setError) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3469::DEBUG::2012-12-02 23:07:00,624::task::852::TaskManager.Task::(_run) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::Task._run: 35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3469::DEBUG::2012-12-02 23:07:00,624::task::1177::TaskManager.Task::(stop) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::stopping in state preparing (force False) >Thread-3469::DEBUG::2012-12-02 23:07:00,625::task::957::TaskManager.Task::(_decref) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::ref 1 aborting True >Thread-3469::INFO::2012-12-02 23:07:00,625::task::1134::TaskManager.Task::(prepare) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3469::DEBUG::2012-12-02 23:07:00,625::task::1139::TaskManager.Task::(prepare) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::Prepare: aborted: Error validating master storage domain >Thread-3469::DEBUG::2012-12-02 23:07:00,625::task::957::TaskManager.Task::(_decref) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::ref 0 aborting True >Thread-3469::DEBUG::2012-12-02 23:07:00,625::task::892::TaskManager.Task::(_doAbort) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::Task._doAbort: force False >Thread-3469::DEBUG::2012-12-02 23:07:00,625::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3469::DEBUG::2012-12-02 23:07:00,625::task::568::TaskManager.Task::(_updateState) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::moving from state preparing -> state aborting >Thread-3469::DEBUG::2012-12-02 23:07:00,626::task::523::TaskManager.Task::(__state_aborting) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::_aborting: recover policy none >Thread-3469::DEBUG::2012-12-02 23:07:00,626::task::568::TaskManager.Task::(_updateState) Task=`35d0de8f-b8d8-4bfd-8632-8ad9abb7e91c`::moving from state aborting -> state failed >Thread-3469::DEBUG::2012-12-02 23:07:00,626::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3469::DEBUG::2012-12-02 23:07:00,626::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3469::ERROR::2012-12-02 23:07:00,626::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3533::DEBUG::2012-12-02 23:07:05,266::task::568::TaskManager.Task::(_updateState) Task=`2a916ead-f7b6-4551-ab8b-419db53047b4`::moving from state init -> state preparing >Thread-3533::INFO::2012-12-02 23:07:05,266::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3533::INFO::2012-12-02 23:07:05,266::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3533::DEBUG::2012-12-02 23:07:05,267::task::1151::TaskManager.Task::(prepare) Task=`2a916ead-f7b6-4551-ab8b-419db53047b4`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3533::DEBUG::2012-12-02 23:07:05,267::task::568::TaskManager.Task::(_updateState) Task=`2a916ead-f7b6-4551-ab8b-419db53047b4`::moving from state preparing -> state finished >Thread-3533::DEBUG::2012-12-02 23:07:05,267::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3533::DEBUG::2012-12-02 23:07:05,267::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3533::DEBUG::2012-12-02 23:07:05,267::task::957::TaskManager.Task::(_decref) Task=`2a916ead-f7b6-4551-ab8b-419db53047b4`::ref 0 aborting False >Thread-3537::DEBUG::2012-12-02 23:07:10,807::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3537::DEBUG::2012-12-02 23:07:10,808::task::568::TaskManager.Task::(_updateState) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::moving from state init -> state preparing >Thread-3537::INFO::2012-12-02 23:07:10,808::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3537::DEBUG::2012-12-02 23:07:10,808::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3537::DEBUG::2012-12-02 23:07:10,808::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3537::DEBUG::2012-12-02 23:07:11,012::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3537::WARNING::2012-12-02 23:07:11,014::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3537::DEBUG::2012-12-02 23:07:11,014::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3537::ERROR::2012-12-02 23:07:11,047::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3537::ERROR::2012-12-02 23:07:11,047::task::833::TaskManager.Task::(_setError) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3537::DEBUG::2012-12-02 23:07:11,048::task::852::TaskManager.Task::(_run) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::Task._run: 7455e7cd-bb5b-4b50-9820-e7470b5fe5d0 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3537::DEBUG::2012-12-02 23:07:11,048::task::1177::TaskManager.Task::(stop) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::stopping in state preparing (force False) >Thread-3537::DEBUG::2012-12-02 23:07:11,048::task::957::TaskManager.Task::(_decref) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::ref 1 aborting True >Thread-3537::INFO::2012-12-02 23:07:11,048::task::1134::TaskManager.Task::(prepare) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3537::DEBUG::2012-12-02 23:07:11,048::task::1139::TaskManager.Task::(prepare) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::Prepare: aborted: Error validating master storage domain >Thread-3537::DEBUG::2012-12-02 23:07:11,048::task::957::TaskManager.Task::(_decref) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::ref 0 aborting True >Thread-3537::DEBUG::2012-12-02 23:07:11,049::task::892::TaskManager.Task::(_doAbort) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::Task._doAbort: force False >Thread-3537::DEBUG::2012-12-02 23:07:11,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3537::DEBUG::2012-12-02 23:07:11,049::task::568::TaskManager.Task::(_updateState) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::moving from state preparing -> state aborting >Thread-3537::DEBUG::2012-12-02 23:07:11,049::task::523::TaskManager.Task::(__state_aborting) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::_aborting: recover policy none >Thread-3537::DEBUG::2012-12-02 23:07:11,049::task::568::TaskManager.Task::(_updateState) Task=`7455e7cd-bb5b-4b50-9820-e7470b5fe5d0`::moving from state aborting -> state failed >Thread-3537::DEBUG::2012-12-02 23:07:11,049::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3537::DEBUG::2012-12-02 23:07:11,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3537::ERROR::2012-12-02 23:07:11,049::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3601::DEBUG::2012-12-02 23:07:16,267::task::568::TaskManager.Task::(_updateState) Task=`453284e4-6794-4d79-b6f2-8f74e81698ff`::moving from state init -> state preparing >Thread-3601::INFO::2012-12-02 23:07:16,267::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3601::INFO::2012-12-02 23:07:16,267::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3601::DEBUG::2012-12-02 23:07:16,267::task::1151::TaskManager.Task::(prepare) Task=`453284e4-6794-4d79-b6f2-8f74e81698ff`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3601::DEBUG::2012-12-02 23:07:16,267::task::568::TaskManager.Task::(_updateState) Task=`453284e4-6794-4d79-b6f2-8f74e81698ff`::moving from state preparing -> state finished >Thread-3601::DEBUG::2012-12-02 23:07:16,268::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3601::DEBUG::2012-12-02 23:07:16,268::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3601::DEBUG::2012-12-02 23:07:16,268::task::957::TaskManager.Task::(_decref) Task=`453284e4-6794-4d79-b6f2-8f74e81698ff`::ref 0 aborting False >Thread-3605::DEBUG::2012-12-02 23:07:21,216::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3605::DEBUG::2012-12-02 23:07:21,217::task::568::TaskManager.Task::(_updateState) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::moving from state init -> state preparing >Thread-3605::INFO::2012-12-02 23:07:21,217::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3605::DEBUG::2012-12-02 23:07:21,217::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3605::DEBUG::2012-12-02 23:07:21,217::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3605::DEBUG::2012-12-02 23:07:21,422::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3605::WARNING::2012-12-02 23:07:21,423::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3605::DEBUG::2012-12-02 23:07:21,424::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3605::ERROR::2012-12-02 23:07:21,452::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3605::ERROR::2012-12-02 23:07:21,452::task::833::TaskManager.Task::(_setError) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3605::DEBUG::2012-12-02 23:07:21,453::task::852::TaskManager.Task::(_run) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::Task._run: 3d1856b7-b0be-4f05-a545-4b1f61cab927 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3605::DEBUG::2012-12-02 23:07:21,453::task::1177::TaskManager.Task::(stop) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::stopping in state preparing (force False) >Thread-3605::DEBUG::2012-12-02 23:07:21,453::task::957::TaskManager.Task::(_decref) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::ref 1 aborting True >Thread-3605::INFO::2012-12-02 23:07:21,453::task::1134::TaskManager.Task::(prepare) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3605::DEBUG::2012-12-02 23:07:21,453::task::1139::TaskManager.Task::(prepare) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::Prepare: aborted: Error validating master storage domain >Thread-3605::DEBUG::2012-12-02 23:07:21,453::task::957::TaskManager.Task::(_decref) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::ref 0 aborting True >Thread-3605::DEBUG::2012-12-02 23:07:21,454::task::892::TaskManager.Task::(_doAbort) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::Task._doAbort: force False >Thread-3605::DEBUG::2012-12-02 23:07:21,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3605::DEBUG::2012-12-02 23:07:21,454::task::568::TaskManager.Task::(_updateState) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::moving from state preparing -> state aborting >Thread-3605::DEBUG::2012-12-02 23:07:21,454::task::523::TaskManager.Task::(__state_aborting) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::_aborting: recover policy none >Thread-3605::DEBUG::2012-12-02 23:07:21,454::task::568::TaskManager.Task::(_updateState) Task=`3d1856b7-b0be-4f05-a545-4b1f61cab927`::moving from state aborting -> state failed >Thread-3605::DEBUG::2012-12-02 23:07:21,454::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3605::DEBUG::2012-12-02 23:07:21,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3605::ERROR::2012-12-02 23:07:21,454::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3669::DEBUG::2012-12-02 23:07:27,171::task::568::TaskManager.Task::(_updateState) Task=`d87cbf61-e2a8-460b-9635-2333fb52f655`::moving from state init -> state preparing >Thread-3669::INFO::2012-12-02 23:07:27,171::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3669::INFO::2012-12-02 23:07:27,172::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3669::DEBUG::2012-12-02 23:07:27,172::task::1151::TaskManager.Task::(prepare) Task=`d87cbf61-e2a8-460b-9635-2333fb52f655`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3669::DEBUG::2012-12-02 23:07:27,172::task::568::TaskManager.Task::(_updateState) Task=`d87cbf61-e2a8-460b-9635-2333fb52f655`::moving from state preparing -> state finished >Thread-3669::DEBUG::2012-12-02 23:07:27,172::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3669::DEBUG::2012-12-02 23:07:27,172::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3669::DEBUG::2012-12-02 23:07:27,172::task::957::TaskManager.Task::(_decref) Task=`d87cbf61-e2a8-460b-9635-2333fb52f655`::ref 0 aborting False >Thread-3672::DEBUG::2012-12-02 23:07:31,614::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3672::DEBUG::2012-12-02 23:07:31,614::task::568::TaskManager.Task::(_updateState) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::moving from state init -> state preparing >Thread-3672::INFO::2012-12-02 23:07:31,615::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3672::DEBUG::2012-12-02 23:07:31,615::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3672::DEBUG::2012-12-02 23:07:31,615::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3672::DEBUG::2012-12-02 23:07:31,821::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3672::WARNING::2012-12-02 23:07:31,823::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3672::DEBUG::2012-12-02 23:07:31,823::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3672::ERROR::2012-12-02 23:07:31,853::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3672::ERROR::2012-12-02 23:07:31,853::task::833::TaskManager.Task::(_setError) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3672::DEBUG::2012-12-02 23:07:31,853::task::852::TaskManager.Task::(_run) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::Task._run: 9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3672::DEBUG::2012-12-02 23:07:31,854::task::1177::TaskManager.Task::(stop) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::stopping in state preparing (force False) >Thread-3672::DEBUG::2012-12-02 23:07:31,854::task::957::TaskManager.Task::(_decref) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::ref 1 aborting True >Thread-3672::INFO::2012-12-02 23:07:31,854::task::1134::TaskManager.Task::(prepare) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3672::DEBUG::2012-12-02 23:07:31,854::task::1139::TaskManager.Task::(prepare) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::Prepare: aborted: Error validating master storage domain >Thread-3672::DEBUG::2012-12-02 23:07:31,854::task::957::TaskManager.Task::(_decref) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::ref 0 aborting True >Thread-3672::DEBUG::2012-12-02 23:07:31,854::task::892::TaskManager.Task::(_doAbort) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::Task._doAbort: force False >Thread-3672::DEBUG::2012-12-02 23:07:31,854::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3672::DEBUG::2012-12-02 23:07:31,855::task::568::TaskManager.Task::(_updateState) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::moving from state preparing -> state aborting >Thread-3672::DEBUG::2012-12-02 23:07:31,855::task::523::TaskManager.Task::(__state_aborting) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::_aborting: recover policy none >Thread-3672::DEBUG::2012-12-02 23:07:31,855::task::568::TaskManager.Task::(_updateState) Task=`9236a9b4-0fcc-49fc-9b7b-6197e3ac53fc`::moving from state aborting -> state failed >Thread-3672::DEBUG::2012-12-02 23:07:31,855::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3672::DEBUG::2012-12-02 23:07:31,855::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3672::ERROR::2012-12-02 23:07:31,855::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3737::DEBUG::2012-12-02 23:07:38,103::task::568::TaskManager.Task::(_updateState) Task=`0e7b973d-be56-4750-a723-51f46d37c5ca`::moving from state init -> state preparing >Thread-3737::INFO::2012-12-02 23:07:38,103::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3737::INFO::2012-12-02 23:07:38,103::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3737::DEBUG::2012-12-02 23:07:38,103::task::1151::TaskManager.Task::(prepare) Task=`0e7b973d-be56-4750-a723-51f46d37c5ca`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3737::DEBUG::2012-12-02 23:07:38,103::task::568::TaskManager.Task::(_updateState) Task=`0e7b973d-be56-4750-a723-51f46d37c5ca`::moving from state preparing -> state finished >Thread-3737::DEBUG::2012-12-02 23:07:38,104::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3737::DEBUG::2012-12-02 23:07:38,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3737::DEBUG::2012-12-02 23:07:38,104::task::957::TaskManager.Task::(_decref) Task=`0e7b973d-be56-4750-a723-51f46d37c5ca`::ref 0 aborting False >Thread-3740::DEBUG::2012-12-02 23:07:42,010::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3740::DEBUG::2012-12-02 23:07:42,011::task::568::TaskManager.Task::(_updateState) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::moving from state init -> state preparing >Thread-3740::INFO::2012-12-02 23:07:42,011::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3740::DEBUG::2012-12-02 23:07:42,011::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3740::DEBUG::2012-12-02 23:07:42,012::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3740::DEBUG::2012-12-02 23:07:42,215::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3740::WARNING::2012-12-02 23:07:42,217::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3740::DEBUG::2012-12-02 23:07:42,217::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3740::ERROR::2012-12-02 23:07:42,247::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3740::ERROR::2012-12-02 23:07:42,247::task::833::TaskManager.Task::(_setError) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3740::DEBUG::2012-12-02 23:07:42,248::task::852::TaskManager.Task::(_run) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::Task._run: cb245b0f-887b-449b-bafa-18a924a12f8e ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3740::DEBUG::2012-12-02 23:07:42,248::task::1177::TaskManager.Task::(stop) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::stopping in state preparing (force False) >Thread-3740::DEBUG::2012-12-02 23:07:42,248::task::957::TaskManager.Task::(_decref) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::ref 1 aborting True >Thread-3740::INFO::2012-12-02 23:07:42,248::task::1134::TaskManager.Task::(prepare) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3740::DEBUG::2012-12-02 23:07:42,248::task::1139::TaskManager.Task::(prepare) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::Prepare: aborted: Error validating master storage domain >Thread-3740::DEBUG::2012-12-02 23:07:42,248::task::957::TaskManager.Task::(_decref) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::ref 0 aborting True >Thread-3740::DEBUG::2012-12-02 23:07:42,248::task::892::TaskManager.Task::(_doAbort) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::Task._doAbort: force False >Thread-3740::DEBUG::2012-12-02 23:07:42,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3740::DEBUG::2012-12-02 23:07:42,249::task::568::TaskManager.Task::(_updateState) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::moving from state preparing -> state aborting >Thread-3740::DEBUG::2012-12-02 23:07:42,249::task::523::TaskManager.Task::(__state_aborting) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::_aborting: recover policy none >Thread-3740::DEBUG::2012-12-02 23:07:42,249::task::568::TaskManager.Task::(_updateState) Task=`cb245b0f-887b-449b-bafa-18a924a12f8e`::moving from state aborting -> state failed >Thread-3740::DEBUG::2012-12-02 23:07:42,249::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3740::DEBUG::2012-12-02 23:07:42,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3740::ERROR::2012-12-02 23:07:42,249::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3805::DEBUG::2012-12-02 23:07:49,044::task::568::TaskManager.Task::(_updateState) Task=`b9b07279-5119-4992-a6b5-32fc85bd7bb0`::moving from state init -> state preparing >Thread-3805::INFO::2012-12-02 23:07:49,044::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3805::INFO::2012-12-02 23:07:49,044::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3805::DEBUG::2012-12-02 23:07:49,044::task::1151::TaskManager.Task::(prepare) Task=`b9b07279-5119-4992-a6b5-32fc85bd7bb0`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3805::DEBUG::2012-12-02 23:07:49,044::task::568::TaskManager.Task::(_updateState) Task=`b9b07279-5119-4992-a6b5-32fc85bd7bb0`::moving from state preparing -> state finished >Thread-3805::DEBUG::2012-12-02 23:07:49,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3805::DEBUG::2012-12-02 23:07:49,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3805::DEBUG::2012-12-02 23:07:49,045::task::957::TaskManager.Task::(_decref) Task=`b9b07279-5119-4992-a6b5-32fc85bd7bb0`::ref 0 aborting False >Thread-3808::DEBUG::2012-12-02 23:07:52,420::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3808::DEBUG::2012-12-02 23:07:52,421::task::568::TaskManager.Task::(_updateState) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::moving from state init -> state preparing >Thread-3808::INFO::2012-12-02 23:07:52,455::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3808::DEBUG::2012-12-02 23:07:52,455::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3808::DEBUG::2012-12-02 23:07:52,456::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3808::DEBUG::2012-12-02 23:07:52,661::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3808::WARNING::2012-12-02 23:07:52,663::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3808::DEBUG::2012-12-02 23:07:52,663::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3808::ERROR::2012-12-02 23:07:52,692::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3808::ERROR::2012-12-02 23:07:52,693::task::833::TaskManager.Task::(_setError) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3808::DEBUG::2012-12-02 23:07:52,693::task::852::TaskManager.Task::(_run) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::Task._run: fc0007b1-4c93-4b6d-a33b-0fc2f13accc3 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3808::DEBUG::2012-12-02 23:07:52,693::task::1177::TaskManager.Task::(stop) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::stopping in state preparing (force False) >Thread-3808::DEBUG::2012-12-02 23:07:52,693::task::957::TaskManager.Task::(_decref) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::ref 1 aborting True >Thread-3808::INFO::2012-12-02 23:07:52,693::task::1134::TaskManager.Task::(prepare) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3808::DEBUG::2012-12-02 23:07:52,694::task::1139::TaskManager.Task::(prepare) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::Prepare: aborted: Error validating master storage domain >Thread-3808::DEBUG::2012-12-02 23:07:52,694::task::957::TaskManager.Task::(_decref) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::ref 0 aborting True >Thread-3808::DEBUG::2012-12-02 23:07:52,694::task::892::TaskManager.Task::(_doAbort) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::Task._doAbort: force False >Thread-3808::DEBUG::2012-12-02 23:07:52,694::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3808::DEBUG::2012-12-02 23:07:52,694::task::568::TaskManager.Task::(_updateState) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::moving from state preparing -> state aborting >Thread-3808::DEBUG::2012-12-02 23:07:52,694::task::523::TaskManager.Task::(__state_aborting) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::_aborting: recover policy none >Thread-3808::DEBUG::2012-12-02 23:07:52,694::task::568::TaskManager.Task::(_updateState) Task=`fc0007b1-4c93-4b6d-a33b-0fc2f13accc3`::moving from state aborting -> state failed >Thread-3808::DEBUG::2012-12-02 23:07:52,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3808::DEBUG::2012-12-02 23:07:52,695::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3808::ERROR::2012-12-02 23:07:52,695::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3873::DEBUG::2012-12-02 23:07:59,941::task::568::TaskManager.Task::(_updateState) Task=`dfba2b03-10be-4d17-a0ac-79f93642f7bf`::moving from state init -> state preparing >Thread-3873::INFO::2012-12-02 23:07:59,942::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3873::INFO::2012-12-02 23:07:59,942::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3873::DEBUG::2012-12-02 23:07:59,942::task::1151::TaskManager.Task::(prepare) Task=`dfba2b03-10be-4d17-a0ac-79f93642f7bf`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3873::DEBUG::2012-12-02 23:07:59,942::task::568::TaskManager.Task::(_updateState) Task=`dfba2b03-10be-4d17-a0ac-79f93642f7bf`::moving from state preparing -> state finished >Thread-3873::DEBUG::2012-12-02 23:07:59,942::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3873::DEBUG::2012-12-02 23:07:59,942::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3873::DEBUG::2012-12-02 23:07:59,942::task::957::TaskManager.Task::(_decref) Task=`dfba2b03-10be-4d17-a0ac-79f93642f7bf`::ref 0 aborting False >Thread-3876::DEBUG::2012-12-02 23:08:02,854::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3876::DEBUG::2012-12-02 23:08:02,854::task::568::TaskManager.Task::(_updateState) Task=`791b4926-50b4-4145-a682-469689a682cd`::moving from state init -> state preparing >Thread-3876::INFO::2012-12-02 23:08:02,854::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3876::DEBUG::2012-12-02 23:08:02,854::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3876::DEBUG::2012-12-02 23:08:02,855::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3876::DEBUG::2012-12-02 23:08:03,058::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3876::WARNING::2012-12-02 23:08:03,060::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3876::DEBUG::2012-12-02 23:08:03,060::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3876::ERROR::2012-12-02 23:08:03,089::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3876::ERROR::2012-12-02 23:08:03,089::task::833::TaskManager.Task::(_setError) Task=`791b4926-50b4-4145-a682-469689a682cd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3876::DEBUG::2012-12-02 23:08:03,089::task::852::TaskManager.Task::(_run) Task=`791b4926-50b4-4145-a682-469689a682cd`::Task._run: 791b4926-50b4-4145-a682-469689a682cd ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3876::DEBUG::2012-12-02 23:08:03,090::task::1177::TaskManager.Task::(stop) Task=`791b4926-50b4-4145-a682-469689a682cd`::stopping in state preparing (force False) >Thread-3876::DEBUG::2012-12-02 23:08:03,090::task::957::TaskManager.Task::(_decref) Task=`791b4926-50b4-4145-a682-469689a682cd`::ref 1 aborting True >Thread-3876::INFO::2012-12-02 23:08:03,090::task::1134::TaskManager.Task::(prepare) Task=`791b4926-50b4-4145-a682-469689a682cd`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3876::DEBUG::2012-12-02 23:08:03,090::task::1139::TaskManager.Task::(prepare) Task=`791b4926-50b4-4145-a682-469689a682cd`::Prepare: aborted: Error validating master storage domain >Thread-3876::DEBUG::2012-12-02 23:08:03,090::task::957::TaskManager.Task::(_decref) Task=`791b4926-50b4-4145-a682-469689a682cd`::ref 0 aborting True >Thread-3876::DEBUG::2012-12-02 23:08:03,090::task::892::TaskManager.Task::(_doAbort) Task=`791b4926-50b4-4145-a682-469689a682cd`::Task._doAbort: force False >Thread-3876::DEBUG::2012-12-02 23:08:03,090::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3876::DEBUG::2012-12-02 23:08:03,090::task::568::TaskManager.Task::(_updateState) Task=`791b4926-50b4-4145-a682-469689a682cd`::moving from state preparing -> state aborting >Thread-3876::DEBUG::2012-12-02 23:08:03,091::task::523::TaskManager.Task::(__state_aborting) Task=`791b4926-50b4-4145-a682-469689a682cd`::_aborting: recover policy none >Thread-3876::DEBUG::2012-12-02 23:08:03,091::task::568::TaskManager.Task::(_updateState) Task=`791b4926-50b4-4145-a682-469689a682cd`::moving from state aborting -> state failed >Thread-3876::DEBUG::2012-12-02 23:08:03,091::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3876::DEBUG::2012-12-02 23:08:03,091::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3876::ERROR::2012-12-02 23:08:03,091::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-3941::DEBUG::2012-12-02 23:08:10,935::task::568::TaskManager.Task::(_updateState) Task=`7e7c2d1c-1f2b-4b5f-bc49-9cd373383045`::moving from state init -> state preparing >Thread-3941::INFO::2012-12-02 23:08:10,935::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3941::INFO::2012-12-02 23:08:10,935::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3941::DEBUG::2012-12-02 23:08:10,935::task::1151::TaskManager.Task::(prepare) Task=`7e7c2d1c-1f2b-4b5f-bc49-9cd373383045`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-3941::DEBUG::2012-12-02 23:08:10,936::task::568::TaskManager.Task::(_updateState) Task=`7e7c2d1c-1f2b-4b5f-bc49-9cd373383045`::moving from state preparing -> state finished >Thread-3941::DEBUG::2012-12-02 23:08:10,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3941::DEBUG::2012-12-02 23:08:10,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3941::DEBUG::2012-12-02 23:08:10,936::task::957::TaskManager.Task::(_decref) Task=`7e7c2d1c-1f2b-4b5f-bc49-9cd373383045`::ref 0 aborting False >Thread-3943::DEBUG::2012-12-02 23:08:13,243::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3943::DEBUG::2012-12-02 23:08:13,244::task::568::TaskManager.Task::(_updateState) Task=`1346bd8e-6815-4a95-af07-614440963fec`::moving from state init -> state preparing >Thread-3943::INFO::2012-12-02 23:08:13,244::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3943::DEBUG::2012-12-02 23:08:13,244::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3943::DEBUG::2012-12-02 23:08:13,244::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3943::DEBUG::2012-12-02 23:08:13,450::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3943::WARNING::2012-12-02 23:08:13,451::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3943::DEBUG::2012-12-02 23:08:13,452::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3943::ERROR::2012-12-02 23:08:13,484::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3943::ERROR::2012-12-02 23:08:13,485::task::833::TaskManager.Task::(_setError) Task=`1346bd8e-6815-4a95-af07-614440963fec`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-3943::DEBUG::2012-12-02 23:08:13,485::task::852::TaskManager.Task::(_run) Task=`1346bd8e-6815-4a95-af07-614440963fec`::Task._run: 1346bd8e-6815-4a95-af07-614440963fec ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-3943::DEBUG::2012-12-02 23:08:13,485::task::1177::TaskManager.Task::(stop) Task=`1346bd8e-6815-4a95-af07-614440963fec`::stopping in state preparing (force False) >Thread-3943::DEBUG::2012-12-02 23:08:13,485::task::957::TaskManager.Task::(_decref) Task=`1346bd8e-6815-4a95-af07-614440963fec`::ref 1 aborting True >Thread-3943::INFO::2012-12-02 23:08:13,485::task::1134::TaskManager.Task::(prepare) Task=`1346bd8e-6815-4a95-af07-614440963fec`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-3943::DEBUG::2012-12-02 23:08:13,486::task::1139::TaskManager.Task::(prepare) Task=`1346bd8e-6815-4a95-af07-614440963fec`::Prepare: aborted: Error validating master storage domain >Thread-3943::DEBUG::2012-12-02 23:08:13,486::task::957::TaskManager.Task::(_decref) Task=`1346bd8e-6815-4a95-af07-614440963fec`::ref 0 aborting True >Thread-3943::DEBUG::2012-12-02 23:08:13,486::task::892::TaskManager.Task::(_doAbort) Task=`1346bd8e-6815-4a95-af07-614440963fec`::Task._doAbort: force False >Thread-3943::DEBUG::2012-12-02 23:08:13,486::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3943::DEBUG::2012-12-02 23:08:13,486::task::568::TaskManager.Task::(_updateState) Task=`1346bd8e-6815-4a95-af07-614440963fec`::moving from state preparing -> state aborting >Thread-3943::DEBUG::2012-12-02 23:08:13,486::task::523::TaskManager.Task::(__state_aborting) Task=`1346bd8e-6815-4a95-af07-614440963fec`::_aborting: recover policy none >Thread-3943::DEBUG::2012-12-02 23:08:13,486::task::568::TaskManager.Task::(_updateState) Task=`1346bd8e-6815-4a95-af07-614440963fec`::moving from state aborting -> state failed >Thread-3943::DEBUG::2012-12-02 23:08:13,486::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3943::DEBUG::2012-12-02 23:08:13,487::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3943::ERROR::2012-12-02 23:08:13,487::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4009::DEBUG::2012-12-02 23:08:21,879::task::568::TaskManager.Task::(_updateState) Task=`4e0803d4-4eb2-432e-8fc2-b7737aa1bcef`::moving from state init -> state preparing >Thread-4009::INFO::2012-12-02 23:08:21,879::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4009::INFO::2012-12-02 23:08:21,879::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4009::DEBUG::2012-12-02 23:08:21,879::task::1151::TaskManager.Task::(prepare) Task=`4e0803d4-4eb2-432e-8fc2-b7737aa1bcef`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4009::DEBUG::2012-12-02 23:08:21,879::task::568::TaskManager.Task::(_updateState) Task=`4e0803d4-4eb2-432e-8fc2-b7737aa1bcef`::moving from state preparing -> state finished >Thread-4009::DEBUG::2012-12-02 23:08:21,879::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4009::DEBUG::2012-12-02 23:08:21,879::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4009::DEBUG::2012-12-02 23:08:21,880::task::957::TaskManager.Task::(_decref) Task=`4e0803d4-4eb2-432e-8fc2-b7737aa1bcef`::ref 0 aborting False >Thread-4011::DEBUG::2012-12-02 23:08:23,647::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4011::DEBUG::2012-12-02 23:08:23,647::task::568::TaskManager.Task::(_updateState) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::moving from state init -> state preparing >Thread-4011::INFO::2012-12-02 23:08:23,647::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4011::DEBUG::2012-12-02 23:08:23,648::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4011::DEBUG::2012-12-02 23:08:23,648::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4011::DEBUG::2012-12-02 23:08:23,852::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4011::WARNING::2012-12-02 23:08:23,854::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4011::DEBUG::2012-12-02 23:08:23,854::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4011::ERROR::2012-12-02 23:08:23,883::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4011::ERROR::2012-12-02 23:08:23,884::task::833::TaskManager.Task::(_setError) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4011::DEBUG::2012-12-02 23:08:23,884::task::852::TaskManager.Task::(_run) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::Task._run: 8958adc4-e45e-4d1b-a707-29177b66c44a ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4011::DEBUG::2012-12-02 23:08:23,884::task::1177::TaskManager.Task::(stop) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::stopping in state preparing (force False) >Thread-4011::DEBUG::2012-12-02 23:08:23,884::task::957::TaskManager.Task::(_decref) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::ref 1 aborting True >Thread-4011::INFO::2012-12-02 23:08:23,885::task::1134::TaskManager.Task::(prepare) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4011::DEBUG::2012-12-02 23:08:23,885::task::1139::TaskManager.Task::(prepare) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::Prepare: aborted: Error validating master storage domain >Thread-4011::DEBUG::2012-12-02 23:08:23,885::task::957::TaskManager.Task::(_decref) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::ref 0 aborting True >Thread-4011::DEBUG::2012-12-02 23:08:23,885::task::892::TaskManager.Task::(_doAbort) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::Task._doAbort: force False >Thread-4011::DEBUG::2012-12-02 23:08:23,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4011::DEBUG::2012-12-02 23:08:23,885::task::568::TaskManager.Task::(_updateState) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::moving from state preparing -> state aborting >Thread-4011::DEBUG::2012-12-02 23:08:23,885::task::523::TaskManager.Task::(__state_aborting) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::_aborting: recover policy none >Thread-4011::DEBUG::2012-12-02 23:08:23,885::task::568::TaskManager.Task::(_updateState) Task=`8958adc4-e45e-4d1b-a707-29177b66c44a`::moving from state aborting -> state failed >Thread-4011::DEBUG::2012-12-02 23:08:23,885::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4011::DEBUG::2012-12-02 23:08:23,886::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4011::ERROR::2012-12-02 23:08:23,886::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4077::DEBUG::2012-12-02 23:08:32,910::task::568::TaskManager.Task::(_updateState) Task=`c8631196-ff28-4619-a0bd-4ff80dc30c37`::moving from state init -> state preparing >Thread-4077::INFO::2012-12-02 23:08:32,910::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4077::INFO::2012-12-02 23:08:32,910::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4077::DEBUG::2012-12-02 23:08:32,910::task::1151::TaskManager.Task::(prepare) Task=`c8631196-ff28-4619-a0bd-4ff80dc30c37`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4077::DEBUG::2012-12-02 23:08:32,910::task::568::TaskManager.Task::(_updateState) Task=`c8631196-ff28-4619-a0bd-4ff80dc30c37`::moving from state preparing -> state finished >Thread-4077::DEBUG::2012-12-02 23:08:32,910::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4077::DEBUG::2012-12-02 23:08:32,911::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4077::DEBUG::2012-12-02 23:08:32,911::task::957::TaskManager.Task::(_decref) Task=`c8631196-ff28-4619-a0bd-4ff80dc30c37`::ref 0 aborting False >Thread-4079::DEBUG::2012-12-02 23:08:34,050::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4079::DEBUG::2012-12-02 23:08:34,050::task::568::TaskManager.Task::(_updateState) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::moving from state init -> state preparing >Thread-4079::INFO::2012-12-02 23:08:34,051::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4079::DEBUG::2012-12-02 23:08:34,051::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4079::DEBUG::2012-12-02 23:08:34,051::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4079::DEBUG::2012-12-02 23:08:34,255::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4079::WARNING::2012-12-02 23:08:34,257::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4079::DEBUG::2012-12-02 23:08:34,257::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4079::ERROR::2012-12-02 23:08:34,285::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4079::ERROR::2012-12-02 23:08:34,286::task::833::TaskManager.Task::(_setError) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4079::DEBUG::2012-12-02 23:08:34,286::task::852::TaskManager.Task::(_run) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::Task._run: b2587b43-ee8f-45f1-b213-06273554223c ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4079::DEBUG::2012-12-02 23:08:34,286::task::1177::TaskManager.Task::(stop) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::stopping in state preparing (force False) >Thread-4079::DEBUG::2012-12-02 23:08:34,286::task::957::TaskManager.Task::(_decref) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::ref 1 aborting True >Thread-4079::INFO::2012-12-02 23:08:34,287::task::1134::TaskManager.Task::(prepare) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4079::DEBUG::2012-12-02 23:08:34,287::task::1139::TaskManager.Task::(prepare) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::Prepare: aborted: Error validating master storage domain >Thread-4079::DEBUG::2012-12-02 23:08:34,287::task::957::TaskManager.Task::(_decref) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::ref 0 aborting True >Thread-4079::DEBUG::2012-12-02 23:08:34,287::task::892::TaskManager.Task::(_doAbort) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::Task._doAbort: force False >Thread-4079::DEBUG::2012-12-02 23:08:34,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4079::DEBUG::2012-12-02 23:08:34,287::task::568::TaskManager.Task::(_updateState) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::moving from state preparing -> state aborting >Thread-4079::DEBUG::2012-12-02 23:08:34,287::task::523::TaskManager.Task::(__state_aborting) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::_aborting: recover policy none >Thread-4079::DEBUG::2012-12-02 23:08:34,287::task::568::TaskManager.Task::(_updateState) Task=`b2587b43-ee8f-45f1-b213-06273554223c`::moving from state aborting -> state failed >Thread-4079::DEBUG::2012-12-02 23:08:34,288::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4079::DEBUG::2012-12-02 23:08:34,288::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4079::ERROR::2012-12-02 23:08:34,288::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4145::DEBUG::2012-12-02 23:08:43,830::task::568::TaskManager.Task::(_updateState) Task=`d1d89a62-3393-43ea-b553-29bc30065754`::moving from state init -> state preparing >Thread-4145::INFO::2012-12-02 23:08:43,831::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4145::INFO::2012-12-02 23:08:43,831::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4145::DEBUG::2012-12-02 23:08:43,831::task::1151::TaskManager.Task::(prepare) Task=`d1d89a62-3393-43ea-b553-29bc30065754`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4145::DEBUG::2012-12-02 23:08:43,831::task::568::TaskManager.Task::(_updateState) Task=`d1d89a62-3393-43ea-b553-29bc30065754`::moving from state preparing -> state finished >Thread-4145::DEBUG::2012-12-02 23:08:43,831::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4145::DEBUG::2012-12-02 23:08:43,831::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4145::DEBUG::2012-12-02 23:08:43,831::task::957::TaskManager.Task::(_decref) Task=`d1d89a62-3393-43ea-b553-29bc30065754`::ref 0 aborting False >Thread-4147::DEBUG::2012-12-02 23:08:44,449::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4147::DEBUG::2012-12-02 23:08:44,449::task::568::TaskManager.Task::(_updateState) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::moving from state init -> state preparing >Thread-4147::INFO::2012-12-02 23:08:44,449::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4147::DEBUG::2012-12-02 23:08:44,450::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4147::DEBUG::2012-12-02 23:08:44,450::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4147::DEBUG::2012-12-02 23:08:44,653::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4147::WARNING::2012-12-02 23:08:44,654::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4147::DEBUG::2012-12-02 23:08:44,655::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4147::ERROR::2012-12-02 23:08:44,684::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4147::ERROR::2012-12-02 23:08:44,684::task::833::TaskManager.Task::(_setError) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4147::DEBUG::2012-12-02 23:08:44,685::task::852::TaskManager.Task::(_run) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::Task._run: f62e9791-9b93-45c0-9c66-35ff74d2ab31 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4147::DEBUG::2012-12-02 23:08:44,685::task::1177::TaskManager.Task::(stop) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::stopping in state preparing (force False) >Thread-4147::DEBUG::2012-12-02 23:08:44,685::task::957::TaskManager.Task::(_decref) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::ref 1 aborting True >Thread-4147::INFO::2012-12-02 23:08:44,685::task::1134::TaskManager.Task::(prepare) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4147::DEBUG::2012-12-02 23:08:44,685::task::1139::TaskManager.Task::(prepare) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::Prepare: aborted: Error validating master storage domain >Thread-4147::DEBUG::2012-12-02 23:08:44,685::task::957::TaskManager.Task::(_decref) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::ref 0 aborting True >Thread-4147::DEBUG::2012-12-02 23:08:44,685::task::892::TaskManager.Task::(_doAbort) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::Task._doAbort: force False >Thread-4147::DEBUG::2012-12-02 23:08:44,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4147::DEBUG::2012-12-02 23:08:44,686::task::568::TaskManager.Task::(_updateState) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::moving from state preparing -> state aborting >Thread-4147::DEBUG::2012-12-02 23:08:44,686::task::523::TaskManager.Task::(__state_aborting) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::_aborting: recover policy none >Thread-4147::DEBUG::2012-12-02 23:08:44,686::task::568::TaskManager.Task::(_updateState) Task=`f62e9791-9b93-45c0-9c66-35ff74d2ab31`::moving from state aborting -> state failed >Thread-4147::DEBUG::2012-12-02 23:08:44,686::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4147::DEBUG::2012-12-02 23:08:44,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4147::ERROR::2012-12-02 23:08:44,686::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4213::DEBUG::2012-12-02 23:08:54,790::task::568::TaskManager.Task::(_updateState) Task=`3fe357c8-a9a9-4aaa-a103-11b90f15440e`::moving from state init -> state preparing >Thread-4213::INFO::2012-12-02 23:08:54,790::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4213::INFO::2012-12-02 23:08:54,791::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4213::DEBUG::2012-12-02 23:08:54,791::task::1151::TaskManager.Task::(prepare) Task=`3fe357c8-a9a9-4aaa-a103-11b90f15440e`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4213::DEBUG::2012-12-02 23:08:54,791::task::568::TaskManager.Task::(_updateState) Task=`3fe357c8-a9a9-4aaa-a103-11b90f15440e`::moving from state preparing -> state finished >Thread-4213::DEBUG::2012-12-02 23:08:54,791::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4213::DEBUG::2012-12-02 23:08:54,791::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4213::DEBUG::2012-12-02 23:08:54,791::task::957::TaskManager.Task::(_decref) Task=`3fe357c8-a9a9-4aaa-a103-11b90f15440e`::ref 0 aborting False >Thread-4214::DEBUG::2012-12-02 23:08:54,851::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4214::DEBUG::2012-12-02 23:08:54,851::task::568::TaskManager.Task::(_updateState) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::moving from state init -> state preparing >Thread-4214::INFO::2012-12-02 23:08:54,852::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4214::DEBUG::2012-12-02 23:08:54,852::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4214::DEBUG::2012-12-02 23:08:54,852::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4214::DEBUG::2012-12-02 23:08:55,056::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4214::WARNING::2012-12-02 23:08:55,057::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4214::DEBUG::2012-12-02 23:08:55,057::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4214::ERROR::2012-12-02 23:08:55,086::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4214::ERROR::2012-12-02 23:08:55,086::task::833::TaskManager.Task::(_setError) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4214::DEBUG::2012-12-02 23:08:55,086::task::852::TaskManager.Task::(_run) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::Task._run: 1c9fb5a6-bd51-4459-addc-62bc380be619 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4214::DEBUG::2012-12-02 23:08:55,087::task::1177::TaskManager.Task::(stop) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::stopping in state preparing (force False) >Thread-4214::DEBUG::2012-12-02 23:08:55,087::task::957::TaskManager.Task::(_decref) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::ref 1 aborting True >Thread-4214::INFO::2012-12-02 23:08:55,087::task::1134::TaskManager.Task::(prepare) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4214::DEBUG::2012-12-02 23:08:55,087::task::1139::TaskManager.Task::(prepare) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::Prepare: aborted: Error validating master storage domain >Thread-4214::DEBUG::2012-12-02 23:08:55,087::task::957::TaskManager.Task::(_decref) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::ref 0 aborting True >Thread-4214::DEBUG::2012-12-02 23:08:55,087::task::892::TaskManager.Task::(_doAbort) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::Task._doAbort: force False >Thread-4214::DEBUG::2012-12-02 23:08:55,087::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4214::DEBUG::2012-12-02 23:08:55,088::task::568::TaskManager.Task::(_updateState) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::moving from state preparing -> state aborting >Thread-4214::DEBUG::2012-12-02 23:08:55,088::task::523::TaskManager.Task::(__state_aborting) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::_aborting: recover policy none >Thread-4214::DEBUG::2012-12-02 23:08:55,088::task::568::TaskManager.Task::(_updateState) Task=`1c9fb5a6-bd51-4459-addc-62bc380be619`::moving from state aborting -> state failed >Thread-4214::DEBUG::2012-12-02 23:08:55,088::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4214::DEBUG::2012-12-02 23:08:55,088::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4214::ERROR::2012-12-02 23:08:55,088::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4281::DEBUG::2012-12-02 23:09:05,239::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4281::DEBUG::2012-12-02 23:09:05,239::task::568::TaskManager.Task::(_updateState) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::moving from state init -> state preparing >Thread-4281::INFO::2012-12-02 23:09:05,240::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4281::DEBUG::2012-12-02 23:09:05,240::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4281::DEBUG::2012-12-02 23:09:05,240::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4281::DEBUG::2012-12-02 23:09:05,444::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4281::WARNING::2012-12-02 23:09:05,446::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4281::DEBUG::2012-12-02 23:09:05,446::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4281::ERROR::2012-12-02 23:09:05,475::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4281::ERROR::2012-12-02 23:09:05,476::task::833::TaskManager.Task::(_setError) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4281::DEBUG::2012-12-02 23:09:05,476::task::852::TaskManager.Task::(_run) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::Task._run: d6d2a193-2443-4338-a6c2-4490268d1340 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4281::DEBUG::2012-12-02 23:09:05,476::task::1177::TaskManager.Task::(stop) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::stopping in state preparing (force False) >Thread-4281::DEBUG::2012-12-02 23:09:05,476::task::957::TaskManager.Task::(_decref) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::ref 1 aborting True >Thread-4281::INFO::2012-12-02 23:09:05,476::task::1134::TaskManager.Task::(prepare) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4281::DEBUG::2012-12-02 23:09:05,476::task::1139::TaskManager.Task::(prepare) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::Prepare: aborted: Error validating master storage domain >Thread-4281::DEBUG::2012-12-02 23:09:05,477::task::957::TaskManager.Task::(_decref) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::ref 0 aborting True >Thread-4281::DEBUG::2012-12-02 23:09:05,477::task::892::TaskManager.Task::(_doAbort) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::Task._doAbort: force False >Thread-4281::DEBUG::2012-12-02 23:09:05,477::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4281::DEBUG::2012-12-02 23:09:05,477::task::568::TaskManager.Task::(_updateState) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::moving from state preparing -> state aborting >Thread-4281::DEBUG::2012-12-02 23:09:05,477::task::523::TaskManager.Task::(__state_aborting) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::_aborting: recover policy none >Thread-4281::DEBUG::2012-12-02 23:09:05,477::task::568::TaskManager.Task::(_updateState) Task=`d6d2a193-2443-4338-a6c2-4490268d1340`::moving from state aborting -> state failed >Thread-4281::DEBUG::2012-12-02 23:09:05,477::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4281::DEBUG::2012-12-02 23:09:05,477::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4281::ERROR::2012-12-02 23:09:05,478::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4343::DEBUG::2012-12-02 23:09:05,711::task::568::TaskManager.Task::(_updateState) Task=`f28ad7e2-019c-4404-9495-7f39b7e5b871`::moving from state init -> state preparing >Thread-4343::INFO::2012-12-02 23:09:05,711::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4343::INFO::2012-12-02 23:09:05,712::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4343::DEBUG::2012-12-02 23:09:05,712::task::1151::TaskManager.Task::(prepare) Task=`f28ad7e2-019c-4404-9495-7f39b7e5b871`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4343::DEBUG::2012-12-02 23:09:05,712::task::568::TaskManager.Task::(_updateState) Task=`f28ad7e2-019c-4404-9495-7f39b7e5b871`::moving from state preparing -> state finished >Thread-4343::DEBUG::2012-12-02 23:09:05,712::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4343::DEBUG::2012-12-02 23:09:05,712::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4343::DEBUG::2012-12-02 23:09:05,712::task::957::TaskManager.Task::(_decref) Task=`f28ad7e2-019c-4404-9495-7f39b7e5b871`::ref 0 aborting False >Thread-4349::DEBUG::2012-12-02 23:09:15,645::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4349::DEBUG::2012-12-02 23:09:15,645::task::568::TaskManager.Task::(_updateState) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::moving from state init -> state preparing >Thread-4349::INFO::2012-12-02 23:09:15,645::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4349::DEBUG::2012-12-02 23:09:15,646::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4349::DEBUG::2012-12-02 23:09:15,646::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4349::DEBUG::2012-12-02 23:09:15,848::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4349::WARNING::2012-12-02 23:09:15,849::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4349::DEBUG::2012-12-02 23:09:15,850::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4349::ERROR::2012-12-02 23:09:15,885::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4349::ERROR::2012-12-02 23:09:15,885::task::833::TaskManager.Task::(_setError) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4349::DEBUG::2012-12-02 23:09:15,886::task::852::TaskManager.Task::(_run) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::Task._run: 4482ae7f-9693-46d0-9acb-a7d55409dca1 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4349::DEBUG::2012-12-02 23:09:15,886::task::1177::TaskManager.Task::(stop) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::stopping in state preparing (force False) >Thread-4349::DEBUG::2012-12-02 23:09:15,886::task::957::TaskManager.Task::(_decref) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::ref 1 aborting True >Thread-4349::INFO::2012-12-02 23:09:15,886::task::1134::TaskManager.Task::(prepare) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4349::DEBUG::2012-12-02 23:09:15,886::task::1139::TaskManager.Task::(prepare) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::Prepare: aborted: Error validating master storage domain >Thread-4349::DEBUG::2012-12-02 23:09:15,886::task::957::TaskManager.Task::(_decref) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::ref 0 aborting True >Thread-4349::DEBUG::2012-12-02 23:09:15,886::task::892::TaskManager.Task::(_doAbort) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::Task._doAbort: force False >Thread-4349::DEBUG::2012-12-02 23:09:15,886::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4349::DEBUG::2012-12-02 23:09:15,887::task::568::TaskManager.Task::(_updateState) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::moving from state preparing -> state aborting >Thread-4349::DEBUG::2012-12-02 23:09:15,887::task::523::TaskManager.Task::(__state_aborting) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::_aborting: recover policy none >Thread-4349::DEBUG::2012-12-02 23:09:15,887::task::568::TaskManager.Task::(_updateState) Task=`4482ae7f-9693-46d0-9acb-a7d55409dca1`::moving from state aborting -> state failed >Thread-4349::DEBUG::2012-12-02 23:09:15,887::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4349::DEBUG::2012-12-02 23:09:15,887::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4349::ERROR::2012-12-02 23:09:15,887::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4411::DEBUG::2012-12-02 23:09:16,659::task::568::TaskManager.Task::(_updateState) Task=`ab76e60d-a0cb-47e5-aeb6-912d04f37444`::moving from state init -> state preparing >Thread-4411::INFO::2012-12-02 23:09:16,659::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4411::INFO::2012-12-02 23:09:16,659::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4411::DEBUG::2012-12-02 23:09:16,659::task::1151::TaskManager.Task::(prepare) Task=`ab76e60d-a0cb-47e5-aeb6-912d04f37444`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4411::DEBUG::2012-12-02 23:09:16,659::task::568::TaskManager.Task::(_updateState) Task=`ab76e60d-a0cb-47e5-aeb6-912d04f37444`::moving from state preparing -> state finished >Thread-4411::DEBUG::2012-12-02 23:09:16,660::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4411::DEBUG::2012-12-02 23:09:16,660::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4411::DEBUG::2012-12-02 23:09:16,660::task::957::TaskManager.Task::(_decref) Task=`ab76e60d-a0cb-47e5-aeb6-912d04f37444`::ref 0 aborting False >Thread-4417::DEBUG::2012-12-02 23:09:26,045::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4417::DEBUG::2012-12-02 23:09:26,045::task::568::TaskManager.Task::(_updateState) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::moving from state init -> state preparing >Thread-4417::INFO::2012-12-02 23:09:26,045::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4417::DEBUG::2012-12-02 23:09:26,045::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4417::DEBUG::2012-12-02 23:09:26,046::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4417::DEBUG::2012-12-02 23:09:26,250::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4417::WARNING::2012-12-02 23:09:26,251::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4417::DEBUG::2012-12-02 23:09:26,251::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4417::ERROR::2012-12-02 23:09:26,286::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4417::ERROR::2012-12-02 23:09:26,286::task::833::TaskManager.Task::(_setError) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4417::DEBUG::2012-12-02 23:09:26,286::task::852::TaskManager.Task::(_run) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::Task._run: e4f30db6-f573-4779-8095-2e91e95006bb ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4417::DEBUG::2012-12-02 23:09:26,287::task::1177::TaskManager.Task::(stop) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::stopping in state preparing (force False) >Thread-4417::DEBUG::2012-12-02 23:09:26,287::task::957::TaskManager.Task::(_decref) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::ref 1 aborting True >Thread-4417::INFO::2012-12-02 23:09:26,287::task::1134::TaskManager.Task::(prepare) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4417::DEBUG::2012-12-02 23:09:26,287::task::1139::TaskManager.Task::(prepare) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::Prepare: aborted: Error validating master storage domain >Thread-4417::DEBUG::2012-12-02 23:09:26,287::task::957::TaskManager.Task::(_decref) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::ref 0 aborting True >Thread-4417::DEBUG::2012-12-02 23:09:26,287::task::892::TaskManager.Task::(_doAbort) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::Task._doAbort: force False >Thread-4417::DEBUG::2012-12-02 23:09:26,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4417::DEBUG::2012-12-02 23:09:26,288::task::568::TaskManager.Task::(_updateState) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::moving from state preparing -> state aborting >Thread-4417::DEBUG::2012-12-02 23:09:26,288::task::523::TaskManager.Task::(__state_aborting) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::_aborting: recover policy none >Thread-4417::DEBUG::2012-12-02 23:09:26,288::task::568::TaskManager.Task::(_updateState) Task=`e4f30db6-f573-4779-8095-2e91e95006bb`::moving from state aborting -> state failed >Thread-4417::DEBUG::2012-12-02 23:09:26,288::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4417::DEBUG::2012-12-02 23:09:26,288::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4417::ERROR::2012-12-02 23:09:26,288::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4479::DEBUG::2012-12-02 23:09:27,576::task::568::TaskManager.Task::(_updateState) Task=`1614ff76-44e8-49e5-9af8-b9c6da01a3d9`::moving from state init -> state preparing >Thread-4479::INFO::2012-12-02 23:09:27,577::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4479::INFO::2012-12-02 23:09:27,577::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4479::DEBUG::2012-12-02 23:09:27,577::task::1151::TaskManager.Task::(prepare) Task=`1614ff76-44e8-49e5-9af8-b9c6da01a3d9`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4479::DEBUG::2012-12-02 23:09:27,577::task::568::TaskManager.Task::(_updateState) Task=`1614ff76-44e8-49e5-9af8-b9c6da01a3d9`::moving from state preparing -> state finished >Thread-4479::DEBUG::2012-12-02 23:09:27,577::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4479::DEBUG::2012-12-02 23:09:27,577::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4479::DEBUG::2012-12-02 23:09:27,577::task::957::TaskManager.Task::(_decref) Task=`1614ff76-44e8-49e5-9af8-b9c6da01a3d9`::ref 0 aborting False >Thread-4485::DEBUG::2012-12-02 23:09:36,449::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4485::DEBUG::2012-12-02 23:09:36,450::task::568::TaskManager.Task::(_updateState) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::moving from state init -> state preparing >Thread-4485::INFO::2012-12-02 23:09:36,450::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4485::DEBUG::2012-12-02 23:09:36,450::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4485::DEBUG::2012-12-02 23:09:36,450::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4485::DEBUG::2012-12-02 23:09:36,656::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4485::WARNING::2012-12-02 23:09:36,657::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4485::DEBUG::2012-12-02 23:09:36,657::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4485::ERROR::2012-12-02 23:09:36,685::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4485::ERROR::2012-12-02 23:09:36,686::task::833::TaskManager.Task::(_setError) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4485::DEBUG::2012-12-02 23:09:36,686::task::852::TaskManager.Task::(_run) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::Task._run: 16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4485::DEBUG::2012-12-02 23:09:36,686::task::1177::TaskManager.Task::(stop) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::stopping in state preparing (force False) >Thread-4485::DEBUG::2012-12-02 23:09:36,686::task::957::TaskManager.Task::(_decref) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::ref 1 aborting True >Thread-4485::INFO::2012-12-02 23:09:36,686::task::1134::TaskManager.Task::(prepare) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4485::DEBUG::2012-12-02 23:09:36,686::task::1139::TaskManager.Task::(prepare) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::Prepare: aborted: Error validating master storage domain >Thread-4485::DEBUG::2012-12-02 23:09:36,687::task::957::TaskManager.Task::(_decref) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::ref 0 aborting True >Thread-4485::DEBUG::2012-12-02 23:09:36,687::task::892::TaskManager.Task::(_doAbort) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::Task._doAbort: force False >Thread-4485::DEBUG::2012-12-02 23:09:36,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4485::DEBUG::2012-12-02 23:09:36,687::task::568::TaskManager.Task::(_updateState) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::moving from state preparing -> state aborting >Thread-4485::DEBUG::2012-12-02 23:09:36,687::task::523::TaskManager.Task::(__state_aborting) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::_aborting: recover policy none >Thread-4485::DEBUG::2012-12-02 23:09:36,687::task::568::TaskManager.Task::(_updateState) Task=`16251d2a-1a86-42d4-85a6-ccaa0e6d2a2d`::moving from state aborting -> state failed >Thread-4485::DEBUG::2012-12-02 23:09:36,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4485::DEBUG::2012-12-02 23:09:36,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4485::ERROR::2012-12-02 23:09:36,688::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4547::DEBUG::2012-12-02 23:09:38,550::task::568::TaskManager.Task::(_updateState) Task=`7f71873c-d10a-4096-a733-4dc83d73a2a0`::moving from state init -> state preparing >Thread-4547::INFO::2012-12-02 23:09:38,551::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4547::INFO::2012-12-02 23:09:38,551::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4547::DEBUG::2012-12-02 23:09:38,551::task::1151::TaskManager.Task::(prepare) Task=`7f71873c-d10a-4096-a733-4dc83d73a2a0`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4547::DEBUG::2012-12-02 23:09:38,551::task::568::TaskManager.Task::(_updateState) Task=`7f71873c-d10a-4096-a733-4dc83d73a2a0`::moving from state preparing -> state finished >Thread-4547::DEBUG::2012-12-02 23:09:38,551::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4547::DEBUG::2012-12-02 23:09:38,551::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4547::DEBUG::2012-12-02 23:09:38,551::task::957::TaskManager.Task::(_decref) Task=`7f71873c-d10a-4096-a733-4dc83d73a2a0`::ref 0 aborting False >Thread-4552::DEBUG::2012-12-02 23:09:46,854::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4552::DEBUG::2012-12-02 23:09:46,855::task::568::TaskManager.Task::(_updateState) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::moving from state init -> state preparing >Thread-4552::INFO::2012-12-02 23:09:46,855::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4552::DEBUG::2012-12-02 23:09:46,855::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4552::DEBUG::2012-12-02 23:09:46,855::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4552::DEBUG::2012-12-02 23:09:47,062::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4552::WARNING::2012-12-02 23:09:47,063::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4552::DEBUG::2012-12-02 23:09:47,063::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4552::ERROR::2012-12-02 23:09:47,093::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4552::ERROR::2012-12-02 23:09:47,093::task::833::TaskManager.Task::(_setError) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4552::DEBUG::2012-12-02 23:09:47,093::task::852::TaskManager.Task::(_run) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::Task._run: 33e923c6-b161-4cde-a87f-427e7da8e940 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4552::DEBUG::2012-12-02 23:09:47,094::task::1177::TaskManager.Task::(stop) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::stopping in state preparing (force False) >Thread-4552::DEBUG::2012-12-02 23:09:47,094::task::957::TaskManager.Task::(_decref) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::ref 1 aborting True >Thread-4552::INFO::2012-12-02 23:09:47,094::task::1134::TaskManager.Task::(prepare) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4552::DEBUG::2012-12-02 23:09:47,094::task::1139::TaskManager.Task::(prepare) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::Prepare: aborted: Error validating master storage domain >Thread-4552::DEBUG::2012-12-02 23:09:47,094::task::957::TaskManager.Task::(_decref) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::ref 0 aborting True >Thread-4552::DEBUG::2012-12-02 23:09:47,094::task::892::TaskManager.Task::(_doAbort) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::Task._doAbort: force False >Thread-4552::DEBUG::2012-12-02 23:09:47,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4552::DEBUG::2012-12-02 23:09:47,095::task::568::TaskManager.Task::(_updateState) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::moving from state preparing -> state aborting >Thread-4552::DEBUG::2012-12-02 23:09:47,095::task::523::TaskManager.Task::(__state_aborting) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::_aborting: recover policy none >Thread-4552::DEBUG::2012-12-02 23:09:47,095::task::568::TaskManager.Task::(_updateState) Task=`33e923c6-b161-4cde-a87f-427e7da8e940`::moving from state aborting -> state failed >Thread-4552::DEBUG::2012-12-02 23:09:47,095::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4552::DEBUG::2012-12-02 23:09:47,095::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4552::ERROR::2012-12-02 23:09:47,095::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4615::DEBUG::2012-12-02 23:09:49,525::task::568::TaskManager.Task::(_updateState) Task=`1032c59e-589d-431e-810a-45d18d770247`::moving from state init -> state preparing >Thread-4615::INFO::2012-12-02 23:09:49,525::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4615::INFO::2012-12-02 23:09:49,525::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4615::DEBUG::2012-12-02 23:09:49,525::task::1151::TaskManager.Task::(prepare) Task=`1032c59e-589d-431e-810a-45d18d770247`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4615::DEBUG::2012-12-02 23:09:49,525::task::568::TaskManager.Task::(_updateState) Task=`1032c59e-589d-431e-810a-45d18d770247`::moving from state preparing -> state finished >Thread-4615::DEBUG::2012-12-02 23:09:49,525::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4615::DEBUG::2012-12-02 23:09:49,526::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4615::DEBUG::2012-12-02 23:09:49,526::task::957::TaskManager.Task::(_decref) Task=`1032c59e-589d-431e-810a-45d18d770247`::ref 0 aborting False >Thread-4620::DEBUG::2012-12-02 23:09:57,250::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4620::DEBUG::2012-12-02 23:09:57,251::task::568::TaskManager.Task::(_updateState) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::moving from state init -> state preparing >Thread-4620::INFO::2012-12-02 23:09:57,251::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4620::DEBUG::2012-12-02 23:09:57,251::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4620::DEBUG::2012-12-02 23:09:57,252::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4620::DEBUG::2012-12-02 23:09:57,455::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4620::WARNING::2012-12-02 23:09:57,456::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4620::DEBUG::2012-12-02 23:09:57,456::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4620::ERROR::2012-12-02 23:09:57,483::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4620::ERROR::2012-12-02 23:09:57,483::task::833::TaskManager.Task::(_setError) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4620::DEBUG::2012-12-02 23:09:57,484::task::852::TaskManager.Task::(_run) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::Task._run: 95d91718-7f6a-4aaa-92bf-0e3a27129070 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4620::DEBUG::2012-12-02 23:09:57,484::task::1177::TaskManager.Task::(stop) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::stopping in state preparing (force False) >Thread-4620::DEBUG::2012-12-02 23:09:57,484::task::957::TaskManager.Task::(_decref) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::ref 1 aborting True >Thread-4620::INFO::2012-12-02 23:09:57,484::task::1134::TaskManager.Task::(prepare) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4620::DEBUG::2012-12-02 23:09:57,484::task::1139::TaskManager.Task::(prepare) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::Prepare: aborted: Error validating master storage domain >Thread-4620::DEBUG::2012-12-02 23:09:57,484::task::957::TaskManager.Task::(_decref) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::ref 0 aborting True >Thread-4620::DEBUG::2012-12-02 23:09:57,484::task::892::TaskManager.Task::(_doAbort) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::Task._doAbort: force False >Thread-4620::DEBUG::2012-12-02 23:09:57,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4620::DEBUG::2012-12-02 23:09:57,485::task::568::TaskManager.Task::(_updateState) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::moving from state preparing -> state aborting >Thread-4620::DEBUG::2012-12-02 23:09:57,485::task::523::TaskManager.Task::(__state_aborting) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::_aborting: recover policy none >Thread-4620::DEBUG::2012-12-02 23:09:57,485::task::568::TaskManager.Task::(_updateState) Task=`95d91718-7f6a-4aaa-92bf-0e3a27129070`::moving from state aborting -> state failed >Thread-4620::DEBUG::2012-12-02 23:09:57,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4620::DEBUG::2012-12-02 23:09:57,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4620::ERROR::2012-12-02 23:09:57,485::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4683::DEBUG::2012-12-02 23:10:00,405::task::568::TaskManager.Task::(_updateState) Task=`436731d3-5621-4fa8-abc4-c1b1d902061e`::moving from state init -> state preparing >Thread-4683::INFO::2012-12-02 23:10:00,405::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4683::INFO::2012-12-02 23:10:00,406::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4683::DEBUG::2012-12-02 23:10:00,406::task::1151::TaskManager.Task::(prepare) Task=`436731d3-5621-4fa8-abc4-c1b1d902061e`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4683::DEBUG::2012-12-02 23:10:00,406::task::568::TaskManager.Task::(_updateState) Task=`436731d3-5621-4fa8-abc4-c1b1d902061e`::moving from state preparing -> state finished >Thread-4683::DEBUG::2012-12-02 23:10:00,406::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4683::DEBUG::2012-12-02 23:10:00,406::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4683::DEBUG::2012-12-02 23:10:00,406::task::957::TaskManager.Task::(_decref) Task=`436731d3-5621-4fa8-abc4-c1b1d902061e`::ref 0 aborting False >Thread-4688::DEBUG::2012-12-02 23:10:07,638::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4688::DEBUG::2012-12-02 23:10:07,639::task::568::TaskManager.Task::(_updateState) Task=`d5732278-460b-4896-94aa-90b618b531fe`::moving from state init -> state preparing >Thread-4688::INFO::2012-12-02 23:10:07,639::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4688::DEBUG::2012-12-02 23:10:07,639::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4688::DEBUG::2012-12-02 23:10:07,639::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4688::DEBUG::2012-12-02 23:10:07,847::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4688::WARNING::2012-12-02 23:10:07,848::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4688::DEBUG::2012-12-02 23:10:07,849::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4688::ERROR::2012-12-02 23:10:07,879::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4688::ERROR::2012-12-02 23:10:07,879::task::833::TaskManager.Task::(_setError) Task=`d5732278-460b-4896-94aa-90b618b531fe`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4688::DEBUG::2012-12-02 23:10:07,879::task::852::TaskManager.Task::(_run) Task=`d5732278-460b-4896-94aa-90b618b531fe`::Task._run: d5732278-460b-4896-94aa-90b618b531fe ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4688::DEBUG::2012-12-02 23:10:07,879::task::1177::TaskManager.Task::(stop) Task=`d5732278-460b-4896-94aa-90b618b531fe`::stopping in state preparing (force False) >Thread-4688::DEBUG::2012-12-02 23:10:07,880::task::957::TaskManager.Task::(_decref) Task=`d5732278-460b-4896-94aa-90b618b531fe`::ref 1 aborting True >Thread-4688::INFO::2012-12-02 23:10:07,880::task::1134::TaskManager.Task::(prepare) Task=`d5732278-460b-4896-94aa-90b618b531fe`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4688::DEBUG::2012-12-02 23:10:07,880::task::1139::TaskManager.Task::(prepare) Task=`d5732278-460b-4896-94aa-90b618b531fe`::Prepare: aborted: Error validating master storage domain >Thread-4688::DEBUG::2012-12-02 23:10:07,880::task::957::TaskManager.Task::(_decref) Task=`d5732278-460b-4896-94aa-90b618b531fe`::ref 0 aborting True >Thread-4688::DEBUG::2012-12-02 23:10:07,880::task::892::TaskManager.Task::(_doAbort) Task=`d5732278-460b-4896-94aa-90b618b531fe`::Task._doAbort: force False >Thread-4688::DEBUG::2012-12-02 23:10:07,880::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4688::DEBUG::2012-12-02 23:10:07,880::task::568::TaskManager.Task::(_updateState) Task=`d5732278-460b-4896-94aa-90b618b531fe`::moving from state preparing -> state aborting >Thread-4688::DEBUG::2012-12-02 23:10:07,880::task::523::TaskManager.Task::(__state_aborting) Task=`d5732278-460b-4896-94aa-90b618b531fe`::_aborting: recover policy none >Thread-4688::DEBUG::2012-12-02 23:10:07,881::task::568::TaskManager.Task::(_updateState) Task=`d5732278-460b-4896-94aa-90b618b531fe`::moving from state aborting -> state failed >Thread-4688::DEBUG::2012-12-02 23:10:07,881::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4688::DEBUG::2012-12-02 23:10:07,881::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4688::ERROR::2012-12-02 23:10:07,881::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4751::DEBUG::2012-12-02 23:10:11,319::task::568::TaskManager.Task::(_updateState) Task=`5477cfc7-5c11-426f-921a-1c0a1198cde6`::moving from state init -> state preparing >Thread-4751::INFO::2012-12-02 23:10:11,320::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4751::INFO::2012-12-02 23:10:11,320::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4751::DEBUG::2012-12-02 23:10:11,320::task::1151::TaskManager.Task::(prepare) Task=`5477cfc7-5c11-426f-921a-1c0a1198cde6`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4751::DEBUG::2012-12-02 23:10:11,320::task::568::TaskManager.Task::(_updateState) Task=`5477cfc7-5c11-426f-921a-1c0a1198cde6`::moving from state preparing -> state finished >Thread-4751::DEBUG::2012-12-02 23:10:11,320::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4751::DEBUG::2012-12-02 23:10:11,320::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4751::DEBUG::2012-12-02 23:10:11,320::task::957::TaskManager.Task::(_decref) Task=`5477cfc7-5c11-426f-921a-1c0a1198cde6`::ref 0 aborting False >Thread-4756::DEBUG::2012-12-02 23:10:18,033::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4756::DEBUG::2012-12-02 23:10:18,034::task::568::TaskManager.Task::(_updateState) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::moving from state init -> state preparing >Thread-4756::INFO::2012-12-02 23:10:18,034::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4756::DEBUG::2012-12-02 23:10:18,034::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4756::DEBUG::2012-12-02 23:10:18,034::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4756::DEBUG::2012-12-02 23:10:18,238::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4756::WARNING::2012-12-02 23:10:18,239::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4756::DEBUG::2012-12-02 23:10:18,240::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4756::ERROR::2012-12-02 23:10:18,273::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4756::ERROR::2012-12-02 23:10:18,273::task::833::TaskManager.Task::(_setError) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-4756::DEBUG::2012-12-02 23:10:18,273::task::852::TaskManager.Task::(_run) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::Task._run: d1ce2373-90a7-4f2b-aee3-15de53ab63ec ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4756::DEBUG::2012-12-02 23:10:18,273::task::1177::TaskManager.Task::(stop) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::stopping in state preparing (force False) >Thread-4756::DEBUG::2012-12-02 23:10:18,274::task::957::TaskManager.Task::(_decref) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::ref 1 aborting True >Thread-4756::INFO::2012-12-02 23:10:18,274::task::1134::TaskManager.Task::(prepare) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-4756::DEBUG::2012-12-02 23:10:18,274::task::1139::TaskManager.Task::(prepare) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::Prepare: aborted: Error validating master storage domain >Thread-4756::DEBUG::2012-12-02 23:10:18,274::task::957::TaskManager.Task::(_decref) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::ref 0 aborting True >Thread-4756::DEBUG::2012-12-02 23:10:18,274::task::892::TaskManager.Task::(_doAbort) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::Task._doAbort: force False >Thread-4756::DEBUG::2012-12-02 23:10:18,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4756::DEBUG::2012-12-02 23:10:18,274::task::568::TaskManager.Task::(_updateState) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::moving from state preparing -> state aborting >Thread-4756::DEBUG::2012-12-02 23:10:18,275::task::523::TaskManager.Task::(__state_aborting) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::_aborting: recover policy none >Thread-4756::DEBUG::2012-12-02 23:10:18,275::task::568::TaskManager.Task::(_updateState) Task=`d1ce2373-90a7-4f2b-aee3-15de53ab63ec`::moving from state aborting -> state failed >Thread-4756::DEBUG::2012-12-02 23:10:18,275::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4756::DEBUG::2012-12-02 23:10:18,275::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4756::ERROR::2012-12-02 23:10:18,275::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-4819::DEBUG::2012-12-02 23:10:22,250::task::568::TaskManager.Task::(_updateState) Task=`17f41de9-4211-4fe6-886f-a7f179e85ef2`::moving from state init -> state preparing >Thread-4819::INFO::2012-12-02 23:10:22,250::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4819::INFO::2012-12-02 23:10:22,250::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4819::DEBUG::2012-12-02 23:10:22,250::task::1151::TaskManager.Task::(prepare) Task=`17f41de9-4211-4fe6-886f-a7f179e85ef2`::finished: {'2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'delay': '0', 'lastCheck': 1354482173.1084299, 'code': 200, 'valid': False}} >Thread-4819::DEBUG::2012-12-02 23:10:22,250::task::568::TaskManager.Task::(_updateState) Task=`17f41de9-4211-4fe6-886f-a7f179e85ef2`::moving from state preparing -> state finished >Thread-4819::DEBUG::2012-12-02 23:10:22,251::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4819::DEBUG::2012-12-02 23:10:22,251::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4819::DEBUG::2012-12-02 23:10:22,251::task::957::TaskManager.Task::(_decref) Task=`17f41de9-4211-4fe6-886f-a7f179e85ef2`::ref 0 aborting False >Thread-4823::DEBUG::2012-12-02 23:10:27,485::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4823::DEBUG::2012-12-02 23:10:27,486::task::568::TaskManager.Task::(_updateState) Task=`ce9f994b-caf2-487d-9031-fe39d9eac4ab`::moving from state init -> state preparing >Thread-4823::INFO::2012-12-02 23:10:27,486::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-4823::INFO::2012-12-02 23:10:27,486::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-4823::DEBUG::2012-12-02 23:10:27,486::task::1151::TaskManager.Task::(prepare) Task=`ce9f994b-caf2-487d-9031-fe39d9eac4ab`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-4823::DEBUG::2012-12-02 23:10:27,486::task::568::TaskManager.Task::(_updateState) Task=`ce9f994b-caf2-487d-9031-fe39d9eac4ab`::moving from state preparing -> state finished >Thread-4823::DEBUG::2012-12-02 23:10:27,486::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4823::DEBUG::2012-12-02 23:10:27,487::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4823::DEBUG::2012-12-02 23:10:27,487::task::957::TaskManager.Task::(_decref) Task=`ce9f994b-caf2-487d-9031-fe39d9eac4ab`::ref 0 aborting False >Thread-4824::DEBUG::2012-12-02 23:10:27,640::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4824::DEBUG::2012-12-02 23:10:27,641::task::568::TaskManager.Task::(_updateState) Task=`d5d982ed-33ea-4ba2-9f96-5d758f3bc9de`::moving from state init -> state preparing >Thread-4824::INFO::2012-12-02 23:10:27,641::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-4824::DEBUG::2012-12-02 23:10:27,646::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252' (cwd None) >Thread-4824::DEBUG::2012-12-02 23:10:27,763::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4824::DEBUG::2012-12-02 23:10:27,763::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4824::DEBUG::2012-12-02 23:10:27,763::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4824::DEBUG::2012-12-02 23:10:27,763::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4824::DEBUG::2012-12-02 23:10:27,764::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4824::DEBUG::2012-12-02 23:10:27,764::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4824::INFO::2012-12-02 23:10:27,764::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-4824::DEBUG::2012-12-02 23:10:27,764::task::1151::TaskManager.Task::(prepare) Task=`d5d982ed-33ea-4ba2-9f96-5d758f3bc9de`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-4824::DEBUG::2012-12-02 23:10:27,764::task::568::TaskManager.Task::(_updateState) Task=`d5d982ed-33ea-4ba2-9f96-5d758f3bc9de`::moving from state preparing -> state finished >Thread-4824::DEBUG::2012-12-02 23:10:27,764::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4824::DEBUG::2012-12-02 23:10:27,765::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4824::DEBUG::2012-12-02 23:10:27,765::task::957::TaskManager.Task::(_decref) Task=`d5d982ed-33ea-4ba2-9f96-5d758f3bc9de`::ref 0 aborting False >Thread-4825::DEBUG::2012-12-02 23:10:27,993::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4825::DEBUG::2012-12-02 23:10:27,994::task::568::TaskManager.Task::(_updateState) Task=`344adca4-b0e3-41a1-8dc6-b3edc178edc2`::moving from state init -> state preparing >Thread-4825::INFO::2012-12-02 23:10:27,994::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', remove=False, options=None) >Thread-4825::DEBUG::2012-12-02 23:10:27,994::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6be5d28e-426b-4d5b-8733-8e2ea2c4df66`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4825::DEBUG::2012-12-02 23:10:27,994::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-4825::DEBUG::2012-12-02 23:10:27,995::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-4825::DEBUG::2012-12-02 23:10:27,995::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6be5d28e-426b-4d5b-8733-8e2ea2c4df66`::Granted request >Thread-4825::DEBUG::2012-12-02 23:10:27,995::task::794::TaskManager.Task::(resourceAcquired) Task=`344adca4-b0e3-41a1-8dc6-b3edc178edc2`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-4825::DEBUG::2012-12-02 23:10:27,995::task::957::TaskManager.Task::(_decref) Task=`344adca4-b0e3-41a1-8dc6-b3edc178edc2`::ref 1 aborting False >Thread-4825::INFO::2012-12-02 23:10:27,995::sp::658::Storage.StoragePool::(disconnect) Disconnect from the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 >Thread-4825::INFO::2012-12-02 23:10:27,996::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: True >Thread-4825::DEBUG::2012-12-02 23:10:27,996::task::1151::TaskManager.Task::(prepare) Task=`344adca4-b0e3-41a1-8dc6-b3edc178edc2`::finished: True >Thread-4825::DEBUG::2012-12-02 23:10:27,996::task::568::TaskManager.Task::(_updateState) Task=`344adca4-b0e3-41a1-8dc6-b3edc178edc2`::moving from state preparing -> state finished >Thread-4825::DEBUG::2012-12-02 23:10:27,996::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4825::DEBUG::2012-12-02 23:10:27,996::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4825::DEBUG::2012-12-02 23:10:27,996::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4825::DEBUG::2012-12-02 23:10:27,996::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4825::DEBUG::2012-12-02 23:10:27,997::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4825::DEBUG::2012-12-02 23:10:27,997::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4825::DEBUG::2012-12-02 23:10:27,997::task::957::TaskManager.Task::(_decref) Task=`344adca4-b0e3-41a1-8dc6-b3edc178edc2`::ref 0 aborting False >Thread-4826::DEBUG::2012-12-02 23:10:28,162::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4826::DEBUG::2012-12-02 23:10:28,162::task::568::TaskManager.Task::(_updateState) Task=`ec0b8e6f-689a-42da-a110-a2e3b2c3d6d5`::moving from state init -> state preparing >Thread-4826::INFO::2012-12-02 23:10:28,162::logUtils::37::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', poolName='dc', masterDom='225bca28-f0a0-4899-b137-32acd04f57d4', domDict={'225bca28-f0a0-4899-b137-32acd04f57d4': 'active', '2ee486e4-abfd-4f11-9b29-42bfc74d7064': 'active'}, masterVersion=2, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=2, options=None) >Thread-4826::INFO::2012-12-02 23:10:28,162::hsm::1483::Storage.HSM::(reconstructMaster) spUUID=479cf195-bb99-45c5-b400-433d810eaa48 master=225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4826::DEBUG::2012-12-02 23:10:28,163::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-4826::DEBUG::2012-12-02 23:10:28,163::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4826::DEBUG::2012-12-02 23:10:28,163::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-4826::DEBUG::2012-12-02 23:10:28,163::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4826::DEBUG::2012-12-02 23:10:28,163::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:28,190::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-4826::DEBUG::2012-12-02 23:10:28,191::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4827::DEBUG::2012-12-02 23:10:28,428::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4827::DEBUG::2012-12-02 23:10:28,429::task::568::TaskManager.Task::(_updateState) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::moving from state init -> state preparing >Thread-4827::INFO::2012-12-02 23:10:28,429::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4827::ERROR::2012-12-02 23:10:28,429::task::833::TaskManager.Task::(_setError) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-4827::DEBUG::2012-12-02 23:10:28,429::task::852::TaskManager.Task::(_run) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::Task._run: b88bfa1d-87f9-4de9-8f38-fa6d3a01c079 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4827::DEBUG::2012-12-02 23:10:28,429::task::1177::TaskManager.Task::(stop) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::stopping in state preparing (force False) >Thread-4827::DEBUG::2012-12-02 23:10:28,430::task::957::TaskManager.Task::(_decref) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::ref 1 aborting True >Thread-4827::INFO::2012-12-02 23:10:28,430::task::1134::TaskManager.Task::(prepare) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-4827::DEBUG::2012-12-02 23:10:28,430::task::1139::TaskManager.Task::(prepare) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::Prepare: aborted: Unknown pool id, pool not connected >Thread-4827::DEBUG::2012-12-02 23:10:28,430::task::957::TaskManager.Task::(_decref) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::ref 0 aborting True >Thread-4827::DEBUG::2012-12-02 23:10:28,430::task::892::TaskManager.Task::(_doAbort) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::Task._doAbort: force False >Thread-4827::DEBUG::2012-12-02 23:10:28,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4827::DEBUG::2012-12-02 23:10:28,430::task::568::TaskManager.Task::(_updateState) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::moving from state preparing -> state aborting >Thread-4827::DEBUG::2012-12-02 23:10:28,430::task::523::TaskManager.Task::(__state_aborting) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::_aborting: recover policy none >Thread-4827::DEBUG::2012-12-02 23:10:28,431::task::568::TaskManager.Task::(_updateState) Task=`b88bfa1d-87f9-4de9-8f38-fa6d3a01c079`::moving from state aborting -> state failed >Thread-4827::DEBUG::2012-12-02 23:10:28,431::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4827::DEBUG::2012-12-02 23:10:28,431::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4827::ERROR::2012-12-02 23:10:28,431::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-4828::DEBUG::2012-12-02 23:10:28,627::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4828::DEBUG::2012-12-02 23:10:28,627::task::568::TaskManager.Task::(_updateState) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::moving from state init -> state preparing >Thread-4828::INFO::2012-12-02 23:10:28,627::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-4828::DEBUG::2012-12-02 23:10:28,628::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`84404ffc-2d93-4d78-bc5f-2dbb68e9e35b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4828::DEBUG::2012-12-02 23:10:28,628::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-4828::DEBUG::2012-12-02 23:10:28,628::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-4828::DEBUG::2012-12-02 23:10:28,628::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`84404ffc-2d93-4d78-bc5f-2dbb68e9e35b`::Granted request >Thread-4828::INFO::2012-12-02 23:10:28,628::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-4828::DEBUG::2012-12-02 23:10:28,629::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4828::DEBUG::2012-12-02 23:10:28,629::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4828::DEBUG::2012-12-02 23:10:28,629::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4828::DEBUG::2012-12-02 23:10:28,629::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4828::DEBUG::2012-12-02 23:10:28,629::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4828::DEBUG::2012-12-02 23:10:28,630::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,825::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:28,930::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-4826::DEBUG::2012-12-02 23:10:28,931::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,934::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,934::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,934::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,934::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,934::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,934::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4826::DEBUG::2012-12-02 23:10:28,935::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:28,936::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:29,152::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4826::WARNING::2012-12-02 23:10:29,154::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4826::DEBUG::2012-12-02 23:10:29,154::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:29,191::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4826::DEBUG::2012-12-02 23:10:29,191::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-4826::DEBUG::2012-12-02 23:10:29,197::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Regular', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a7f8cb215a2f97d3e2c97789a0d5ceca778bfa1b'] >Thread-4826::DEBUG::2012-12-02 23:10:29,232::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-4826::DEBUG::2012-12-02 23:10:29,233::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '225bca28-f0a0-4899-b137-32acd04f57d4_imageNS' >Thread-4826::DEBUG::2012-12-02 23:10:29,233::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS' >Thread-4826::INFO::2012-12-02 23:10:29,233::fileSD::260::Storage.StorageDomain::(validate) sdUUID=225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4828::ERROR::2012-12-02 23:10:29,234::sp::1505::Storage.StoragePool::(getMasterDomain) Requested master domain 225bca28-f0a0-4899-b137-32acd04f57d4 is not a master domain at all >Thread-4828::DEBUG::2012-12-02 23:10:29,234::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4828::DEBUG::2012-12-02 23:10:29,235::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4828::DEBUG::2012-12-02 23:10:29,235::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4828::DEBUG::2012-12-02 23:10:29,235::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4828::ERROR::2012-12-02 23:10:29,235::task::833::TaskManager.Task::(_setError) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1506, in getMasterDomain > raise se.StoragePoolWrongMaster(self.spUUID, msdUUID) >StoragePoolWrongMaster: Wrong Master domain or its version: 'SD=225bca28-f0a0-4899-b137-32acd04f57d4, pool=479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::852::TaskManager.Task::(_run) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::Task._run: 7ca8f8aa-82da-4708-92e2-1a1c944b0691 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::1177::TaskManager.Task::(stop) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::stopping in state preparing (force False) >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::957::TaskManager.Task::(_decref) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::ref 1 aborting True >Thread-4828::INFO::2012-12-02 23:10:29,236::task::1134::TaskManager.Task::(prepare) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::aborting: Task is aborted: 'Wrong Master domain or its version' - code 324 >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::1139::TaskManager.Task::(prepare) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::Prepare: aborted: Wrong Master domain or its version >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::957::TaskManager.Task::(_decref) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::ref 0 aborting True >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::892::TaskManager.Task::(_doAbort) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::Task._doAbort: force False >Thread-4828::DEBUG::2012-12-02 23:10:29,236::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4828::DEBUG::2012-12-02 23:10:29,236::task::568::TaskManager.Task::(_updateState) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::moving from state preparing -> state aborting >Thread-4828::DEBUG::2012-12-02 23:10:29,237::task::523::TaskManager.Task::(__state_aborting) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::_aborting: recover policy none >Thread-4828::DEBUG::2012-12-02 23:10:29,237::task::568::TaskManager.Task::(_updateState) Task=`7ca8f8aa-82da-4708-92e2-1a1c944b0691`::moving from state aborting -> state failed >Thread-4828::DEBUG::2012-12-02 23:10:29,237::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4828::DEBUG::2012-12-02 23:10:29,237::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4828::ERROR::2012-12-02 23:10:29,237::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Wrong Master domain or its version: 'SD=225bca28-f0a0-4899-b137-32acd04f57d4, pool=479cf195-bb99-45c5-b400-433d810eaa48'", 'code': 324}} >Thread-4826::DEBUG::2012-12-02 23:10:29,241::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Regular', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a7f8cb215a2f97d3e2c97789a0d5ceca778bfa1b'] >Thread-4826::DEBUG::2012-12-02 23:10:29,242::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`108c5398-5ac5-4e40-96bc-ba734df98903`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4826::DEBUG::2012-12-02 23:10:29,242::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-4826::DEBUG::2012-12-02 23:10:29,242::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-4826::DEBUG::2012-12-02 23:10:29,242::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`108c5398-5ac5-4e40-96bc-ba734df98903`::Granted request >Thread-4826::DEBUG::2012-12-02 23:10:29,242::task::794::TaskManager.Task::(resourceAcquired) Task=`ec0b8e6f-689a-42da-a110-a2e3b2c3d6d5`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-4826::DEBUG::2012-12-02 23:10:29,242::task::957::TaskManager.Task::(_decref) Task=`ec0b8e6f-689a-42da-a110-a2e3b2c3d6d5`::ref 1 aborting False >Thread-4826::INFO::2012-12-02 23:10:29,242::sp::727::Storage.StoragePool::(reconstructMaster) spUUID=479cf195-bb99-45c5-b400-433d810eaa48 hostId=2 poolName=dc msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4 domDict={'225bca28-f0a0-4899-b137-32acd04f57d4': 'active', '2ee486e4-abfd-4f11-9b29-42bfc74d7064': 'active'} masterVersion=2 leaseparams=({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5}) >Thread-4826::INFO::2012-12-02 23:10:29,242::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 225bca28-f0a0-4899-b137-32acd04f57d4 (id: 2) >Thread-4893::DEBUG::2012-12-02 23:10:33,669::task::568::TaskManager.Task::(_updateState) Task=`655c0a11-4b32-4af3-a39f-8ddb90100d1e`::moving from state init -> state preparing >Thread-4893::INFO::2012-12-02 23:10:33,669::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4893::INFO::2012-12-02 23:10:33,669::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-4893::DEBUG::2012-12-02 23:10:33,669::task::1151::TaskManager.Task::(prepare) Task=`655c0a11-4b32-4af3-a39f-8ddb90100d1e`::finished: {} >Thread-4893::DEBUG::2012-12-02 23:10:33,669::task::568::TaskManager.Task::(_updateState) Task=`655c0a11-4b32-4af3-a39f-8ddb90100d1e`::moving from state preparing -> state finished >Thread-4893::DEBUG::2012-12-02 23:10:33,669::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4893::DEBUG::2012-12-02 23:10:33,669::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4893::DEBUG::2012-12-02 23:10:33,670::task::957::TaskManager.Task::(_decref) Task=`655c0a11-4b32-4af3-a39f-8ddb90100d1e`::ref 0 aborting False >Thread-4897::DEBUG::2012-12-02 23:10:39,391::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4897::DEBUG::2012-12-02 23:10:39,392::task::568::TaskManager.Task::(_updateState) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::moving from state init -> state preparing >Thread-4897::INFO::2012-12-02 23:10:39,392::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4897::ERROR::2012-12-02 23:10:39,392::task::833::TaskManager.Task::(_setError) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-4897::DEBUG::2012-12-02 23:10:39,392::task::852::TaskManager.Task::(_run) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::Task._run: 1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4897::DEBUG::2012-12-02 23:10:39,392::task::1177::TaskManager.Task::(stop) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::stopping in state preparing (force False) >Thread-4897::DEBUG::2012-12-02 23:10:39,392::task::957::TaskManager.Task::(_decref) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::ref 1 aborting True >Thread-4897::INFO::2012-12-02 23:10:39,393::task::1134::TaskManager.Task::(prepare) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-4897::DEBUG::2012-12-02 23:10:39,393::task::1139::TaskManager.Task::(prepare) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::Prepare: aborted: Unknown pool id, pool not connected >Thread-4897::DEBUG::2012-12-02 23:10:39,393::task::957::TaskManager.Task::(_decref) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::ref 0 aborting True >Thread-4897::DEBUG::2012-12-02 23:10:39,393::task::892::TaskManager.Task::(_doAbort) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::Task._doAbort: force False >Thread-4897::DEBUG::2012-12-02 23:10:39,393::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4897::DEBUG::2012-12-02 23:10:39,393::task::568::TaskManager.Task::(_updateState) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::moving from state preparing -> state aborting >Thread-4897::DEBUG::2012-12-02 23:10:39,393::task::523::TaskManager.Task::(__state_aborting) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::_aborting: recover policy none >Thread-4897::DEBUG::2012-12-02 23:10:39,393::task::568::TaskManager.Task::(_updateState) Task=`1e5bd9d3-49d7-4ac0-a2dc-761f403ecee0`::moving from state aborting -> state failed >Thread-4897::DEBUG::2012-12-02 23:10:39,393::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4897::DEBUG::2012-12-02 23:10:39,394::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4897::ERROR::2012-12-02 23:10:39,394::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-4898::DEBUG::2012-12-02 23:10:39,545::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4898::DEBUG::2012-12-02 23:10:39,545::task::568::TaskManager.Task::(_updateState) Task=`990d1d03-a813-4bed-ab91-d15a928973d1`::moving from state init -> state preparing >Thread-4898::INFO::2012-12-02 23:10:39,546::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-4898::DEBUG::2012-12-02 23:10:39,546::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-4898::DEBUG::2012-12-02 23:10:39,546::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-4898::INFO::2012-12-02 23:10:39,546::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-4898::DEBUG::2012-12-02 23:10:39,546::task::1151::TaskManager.Task::(prepare) Task=`990d1d03-a813-4bed-ab91-d15a928973d1`::finished: {'allTasksStatus': {}} >Thread-4898::DEBUG::2012-12-02 23:10:39,546::task::568::TaskManager.Task::(_updateState) Task=`990d1d03-a813-4bed-ab91-d15a928973d1`::moving from state preparing -> state finished >Thread-4898::DEBUG::2012-12-02 23:10:39,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4898::DEBUG::2012-12-02 23:10:39,546::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4898::DEBUG::2012-12-02 23:10:39,547::task::957::TaskManager.Task::(_decref) Task=`990d1d03-a813-4bed-ab91-d15a928973d1`::ref 0 aborting False >Thread-4899::DEBUG::2012-12-02 23:10:39,692::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4899::DEBUG::2012-12-02 23:10:39,692::task::568::TaskManager.Task::(_updateState) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::moving from state init -> state preparing >Thread-4899::INFO::2012-12-02 23:10:39,692::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4899::ERROR::2012-12-02 23:10:39,692::task::833::TaskManager.Task::(_setError) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-4899::DEBUG::2012-12-02 23:10:39,693::task::852::TaskManager.Task::(_run) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::Task._run: f0e493c2-1987-4830-8f66-7a8bb86884f8 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-4899::DEBUG::2012-12-02 23:10:39,693::task::1177::TaskManager.Task::(stop) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::stopping in state preparing (force False) >Thread-4899::DEBUG::2012-12-02 23:10:39,693::task::957::TaskManager.Task::(_decref) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::ref 1 aborting True >Thread-4899::INFO::2012-12-02 23:10:39,693::task::1134::TaskManager.Task::(prepare) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-4899::DEBUG::2012-12-02 23:10:39,693::task::1139::TaskManager.Task::(prepare) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::Prepare: aborted: Unknown pool id, pool not connected >Thread-4899::DEBUG::2012-12-02 23:10:39,693::task::957::TaskManager.Task::(_decref) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::ref 0 aborting True >Thread-4899::DEBUG::2012-12-02 23:10:39,693::task::892::TaskManager.Task::(_doAbort) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::Task._doAbort: force False >Thread-4899::DEBUG::2012-12-02 23:10:39,693::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4899::DEBUG::2012-12-02 23:10:39,694::task::568::TaskManager.Task::(_updateState) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::moving from state preparing -> state aborting >Thread-4899::DEBUG::2012-12-02 23:10:39,694::task::523::TaskManager.Task::(__state_aborting) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::_aborting: recover policy none >Thread-4899::DEBUG::2012-12-02 23:10:39,694::task::568::TaskManager.Task::(_updateState) Task=`f0e493c2-1987-4830-8f66-7a8bb86884f8`::moving from state aborting -> state failed >Thread-4899::DEBUG::2012-12-02 23:10:39,694::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4899::DEBUG::2012-12-02 23:10:39,694::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4899::ERROR::2012-12-02 23:10:39,694::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-4900::DEBUG::2012-12-02 23:10:39,863::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4900::DEBUG::2012-12-02 23:10:39,863::task::568::TaskManager.Task::(_updateState) Task=`3611eb9d-3294-48bc-b496-944b6aa66296`::moving from state init -> state preparing >Thread-4900::INFO::2012-12-02 23:10:39,863::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-4900::DEBUG::2012-12-02 23:10:39,864::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`bf096e61-8a53-4f4e-aecf-5d2625301d9c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4900::DEBUG::2012-12-02 23:10:39,864::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-4900::DEBUG::2012-12-02 23:10:39,864::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is currently locked, Entering queue (1 in queue) >Thread-4826::DEBUG::2012-12-02 23:10:50,244::safelease::178::SANLock::(acquireHostId) Host id for domain 225bca28-f0a0-4899-b137-32acd04f57d4 successfully acquired (id: 2) >Thread-4826::INFO::2012-12-02 23:10:50,245::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 225bca28-f0a0-4899-b137-32acd04f57d4 (id: 2) >Thread-4826::DEBUG::2012-12-02 23:10:50,609::safelease::235::SANLock::(acquire) Cluster lock for domain 225bca28-f0a0-4899-b137-32acd04f57d4 successfully acquired (id: 2) >Thread-4826::INFO::2012-12-02 23:10:50,609::sp::702::Storage.StoragePool::(createMaster) setting master domain for spUUID 479cf195-bb99-45c5-b400-433d810eaa48 on sdUUID=225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4826::DEBUG::2012-12-02 23:10:50,610::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,610::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,614::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Regular', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=a7f8cb215a2f97d3e2c97789a0d5ceca778bfa1b'] >Thread-4826::DEBUG::2012-12-02 23:10:50,614::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,615::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,615::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,615::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-4826::DEBUG::2012-12-02 23:10:50,615::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=07fb669722ef0133e261acbecf42d9c63d5eabe7'] >Thread-4826::DEBUG::2012-12-02 23:10:50,644::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-4826::INFO::2012-12-02 23:10:50,644::sp::757::Storage.StoragePool::(reconstructMaster) Set storage pool domains: {'225bca28-f0a0-4899-b137-32acd04f57d4': 'Active', '2ee486e4-abfd-4f11-9b29-42bfc74d7064': 'Active'} >Thread-4826::DEBUG::2012-12-02 23:10:50,644::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,644::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-4826::DEBUG::2012-12-02 23:10:50,645::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-4826::DEBUG::2012-12-02 23:10:50,683::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-4826::DEBUG::2012-12-02 23:10:50,683::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:50,683::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:50,683::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:50,683::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:50,683::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:50,684::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:50,684::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-4826::DEBUG::2012-12-02 23:10:50,684::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4826::DEBUG::2012-12-02 23:10:50,684::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-4826::DEBUG::2012-12-02 23:10:50,684::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4826::DEBUG::2012-12-02 23:10:50,684::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:50,710::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-4826::DEBUG::2012-12-02 23:10:50,710::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4826::DEBUG::2012-12-02 23:10:51,343::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:51,411::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-4826::DEBUG::2012-12-02 23:10:51,412::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,412::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,412::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,412::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,413::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,413::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,413::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4826::DEBUG::2012-12-02 23:10:51,413::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,415::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:51,624::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4826::WARNING::2012-12-02 23:10:51,625::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4826::DEBUG::2012-12-02 23:10:51,625::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,659::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4826::DEBUG::2012-12-02 23:10:51,659::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-4826::DEBUG::2012-12-02 23:10:51,668::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-4826::DEBUG::2012-12-02 23:10:51,671::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-4826::WARNING::2012-12-02 23:10:51,671::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-4826::WARNING::2012-12-02 23:10:51,671::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-4826::DEBUG::2012-12-02 23:10:51,672::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 225bca28-f0a0-4899-b137-32acd04f57d4 verified, version 2 >Thread-4826::DEBUG::2012-12-02 23:10:51,672::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-4826::DEBUG::2012-12-02 23:10:51,672::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4826::DEBUG::2012-12-02 23:10:51,678::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-4826::DEBUG::2012-12-02 23:10:51,679::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `479cf195-bb99-45c5-b400-433d810eaa48` started monitoring domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Thread-4826::DEBUG::2012-12-02 23:10:51,679::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,680::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:51,888::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4826::WARNING::2012-12-02 23:10:51,889::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4826::DEBUG::2012-12-02 23:10:51,889::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4826::ERROR::2012-12-02 23:10:51,918::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4826::DEBUG::2012-12-02 23:10:51,919::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4826::DEBUG::2012-12-02 23:10:51,919::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4826::DEBUG::2012-12-02 23:10:51,920::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4826::DEBUG::2012-12-02 23:10:52,126::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4826::WARNING::2012-12-02 23:10:52,127::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4826::DEBUG::2012-12-02 23:10:52,128::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4826::INFO::2012-12-02 23:10:52,155::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4826::INFO::2012-12-02 23:10:52,155::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-4826::INFO::2012-12-02 23:10:52,156::safelease::239::SANLock::(release) Releasing cluster lock for domain 225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4826::DEBUG::2012-12-02 23:10:52,172::safelease::250::SANLock::(release) Cluster lock for domain 225bca28-f0a0-4899-b137-32acd04f57d4 successfully released >Thread-4826::INFO::2012-12-02 23:10:52,172::logUtils::39::dispatcher::(wrapper) Run and protect: reconstructMaster, Return response: None >Thread-4826::DEBUG::2012-12-02 23:10:52,172::task::1151::TaskManager.Task::(prepare) Task=`ec0b8e6f-689a-42da-a110-a2e3b2c3d6d5`::finished: None >Thread-4963::DEBUG::2012-12-02 23:10:52,172::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 225bca28-f0a0-4899-b137-32acd04f57d4 is stopping >Thread-4826::DEBUG::2012-12-02 23:10:52,173::task::568::TaskManager.Task::(_updateState) Task=`ec0b8e6f-689a-42da-a110-a2e3b2c3d6d5`::moving from state preparing -> state finished >Thread-4963::INFO::2012-12-02 23:10:52,173::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 225bca28-f0a0-4899-b137-32acd04f57d4 (id: 2) >Thread-4826::DEBUG::2012-12-02 23:10:52,173::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4826::DEBUG::2012-12-02 23:10:52,173::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4826::DEBUG::2012-12-02 23:10:52,174::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4826::DEBUG::2012-12-02 23:10:52,174::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4826::DEBUG::2012-12-02 23:10:52,174::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4826::DEBUG::2012-12-02 23:10:52,174::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' has 1 requests in queue. Handling top request. >Thread-4826::DEBUG::2012-12-02 23:10:52,174::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`bf096e61-8a53-4f4e-aecf-5d2625301d9c`::Granted request >Thread-4826::DEBUG::2012-12-02 23:10:52,174::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.479cf195-bb99-45c5-b400-433d810eaa48 - exclusive: granted' was granted >Thread-4900::INFO::2012-12-02 23:10:52,175::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-4826::DEBUG::2012-12-02 23:10:52,175::task::957::TaskManager.Task::(_decref) Task=`ec0b8e6f-689a-42da-a110-a2e3b2c3d6d5`::ref 0 aborting False >Thread-4900::DEBUG::2012-12-02 23:10:52,175::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,176::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,176::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,176::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,177::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,177::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,177::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-4900::DEBUG::2012-12-02 23:10:52,177::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4900::DEBUG::2012-12-02 23:10:52,177::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-4900::DEBUG::2012-12-02 23:10:52,177::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4900::DEBUG::2012-12-02 23:10:52,177::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-4900::DEBUG::2012-12-02 23:10:52,203::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-4900::DEBUG::2012-12-02 23:10:52,203::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-5089::DEBUG::2012-12-02 23:10:52,471::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5089::DEBUG::2012-12-02 23:10:52,471::task::568::TaskManager.Task::(_updateState) Task=`b84b2658-116a-45ec-bc6c-4fb5eaac5c1b`::moving from state init -> state preparing >Thread-5089::INFO::2012-12-02 23:10:52,471::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-5089::DEBUG::2012-12-02 23:10:52,472::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3ff477ca-6a51-4b0f-bede-66799c821495`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5089::DEBUG::2012-12-02 23:10:52,472::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5089::DEBUG::2012-12-02 23:10:52,472::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is currently locked, Entering queue (1 in queue) >Thread-4900::DEBUG::2012-12-02 23:10:52,836::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-4900::DEBUG::2012-12-02 23:10:52,934::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-4900::DEBUG::2012-12-02 23:10:52,934::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,934::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,935::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,935::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,935::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,935::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,935::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4900::DEBUG::2012-12-02 23:10:52,935::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:52,936::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4900::DEBUG::2012-12-02 23:10:53,143::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4900::WARNING::2012-12-02 23:10:53,144::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4900::DEBUG::2012-12-02 23:10:53,144::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:53,179::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4900::DEBUG::2012-12-02 23:10:53,179::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-4900::DEBUG::2012-12-02 23:10:53,187::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-4900::DEBUG::2012-12-02 23:10:53,189::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-4900::WARNING::2012-12-02 23:10:53,189::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-4900::WARNING::2012-12-02 23:10:53,189::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-4900::DEBUG::2012-12-02 23:10:53,190::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 225bca28-f0a0-4899-b137-32acd04f57d4 verified, version 2 >Thread-4900::DEBUG::2012-12-02 23:10:53,190::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-4900::DEBUG::2012-12-02 23:10:53,190::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-4900::DEBUG::2012-12-02 23:10:53,197::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-4900::DEBUG::2012-12-02 23:10:53,197::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `479cf195-bb99-45c5-b400-433d810eaa48` started monitoring domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Thread-4900::DEBUG::2012-12-02 23:10:53,198::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:53,198::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4900::DEBUG::2012-12-02 23:10:53,405::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4900::WARNING::2012-12-02 23:10:53,407::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4900::DEBUG::2012-12-02 23:10:53,407::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4900::ERROR::2012-12-02 23:10:53,437::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4900::DEBUG::2012-12-02 23:10:53,437::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-4900::DEBUG::2012-12-02 23:10:53,437::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4900::DEBUG::2012-12-02 23:10:53,438::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4900::DEBUG::2012-12-02 23:10:53,644::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4900::WARNING::2012-12-02 23:10:53,646::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4900::DEBUG::2012-12-02 23:10:53,646::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4900::INFO::2012-12-02 23:10:53,675::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4900::INFO::2012-12-02 23:10:53,675::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-4900::DEBUG::2012-12-02 23:10:53,676::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4900::DEBUG::2012-12-02 23:10:53,676::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4900::DEBUG::2012-12-02 23:10:53,676::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4900::DEBUG::2012-12-02 23:10:53,676::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' has 1 requests in queue. Handling top request. >Thread-4900::DEBUG::2012-12-02 23:10:53,676::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3ff477ca-6a51-4b0f-bede-66799c821495`::Granted request >Thread-4900::DEBUG::2012-12-02 23:10:53,676::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.479cf195-bb99-45c5-b400-433d810eaa48 - shared: granted' was granted >Thread-4900::DEBUG::2012-12-02 23:10:53,677::resourceManager::597::ResourceManager::(releaseResource) This is a shared lock. Granting all shared requests >Thread-4900::INFO::2012-12-02 23:10:53,677::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-5089::DEBUG::2012-12-02 23:10:53,677::task::794::TaskManager.Task::(resourceAcquired) Task=`b84b2658-116a-45ec-bc6c-4fb5eaac5c1b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4900::DEBUG::2012-12-02 23:10:53,677::task::1151::TaskManager.Task::(prepare) Task=`3611eb9d-3294-48bc-b496-944b6aa66296`::finished: True >Thread-5089::DEBUG::2012-12-02 23:10:53,677::task::957::TaskManager.Task::(_decref) Task=`b84b2658-116a-45ec-bc6c-4fb5eaac5c1b`::ref 1 aborting False >Thread-4900::DEBUG::2012-12-02 23:10:53,678::task::568::TaskManager.Task::(_updateState) Task=`3611eb9d-3294-48bc-b496-944b6aa66296`::moving from state preparing -> state finished >Thread-5089::INFO::2012-12-02 23:10:53,678::fileSD::260::Storage.StorageDomain::(validate) sdUUID=225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-4900::DEBUG::2012-12-02 23:10:53,678::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4900::DEBUG::2012-12-02 23:10:53,678::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4900::DEBUG::2012-12-02 23:10:53,679::task::957::TaskManager.Task::(_decref) Task=`3611eb9d-3294-48bc-b496-944b6aa66296`::ref 0 aborting False >Thread-5089::DEBUG::2012-12-02 23:10:53,684::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-5089::DEBUG::2012-12-02 23:10:53,685::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:53,685::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:53,685::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:53,686::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:53,686::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:53,686::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:53,686::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-5089::DEBUG::2012-12-02 23:10:53,686::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-5089::DEBUG::2012-12-02 23:10:53,686::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-5089::DEBUG::2012-12-02 23:10:53,686::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-5089::DEBUG::2012-12-02 23:10:53,687::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-5089::DEBUG::2012-12-02 23:10:53,711::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-5089::DEBUG::2012-12-02 23:10:53,711::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-5276::DEBUG::2012-12-02 23:10:53,818::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5276::DEBUG::2012-12-02 23:10:53,818::task::568::TaskManager.Task::(_updateState) Task=`20ec86fa-3514-4c19-bd00-c598b455f5c0`::moving from state init -> state preparing >Thread-5276::INFO::2012-12-02 23:10:53,818::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4963::DEBUG::2012-12-02 23:10:54,173::safelease::193::SANLock::(releaseHostId) Host id for domain 225bca28-f0a0-4899-b137-32acd04f57d4 released successfully (id: 2) >Thread-5089::DEBUG::2012-12-02 23:10:54,344::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-5089::DEBUG::2012-12-02 23:10:54,416::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-5089::DEBUG::2012-12-02 23:10:54,416::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,416::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,416::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,417::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,417::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,417::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,417::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-5089::DEBUG::2012-12-02 23:10:54,417::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,419::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5089::DEBUG::2012-12-02 23:10:54,625::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5089::WARNING::2012-12-02 23:10:54,627::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5089::DEBUG::2012-12-02 23:10:54,627::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,662::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5089::DEBUG::2012-12-02 23:10:54,662::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-5089::DEBUG::2012-12-02 23:10:54,670::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-5089::DEBUG::2012-12-02 23:10:54,671::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-5089::WARNING::2012-12-02 23:10:54,672::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-5089::WARNING::2012-12-02 23:10:54,672::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-5276::INFO::2012-12-02 23:10:54,672::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-5089::DEBUG::2012-12-02 23:10:54,673::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 225bca28-f0a0-4899-b137-32acd04f57d4 verified, version 2 >Thread-5276::DEBUG::2012-12-02 23:10:54,673::task::1151::TaskManager.Task::(prepare) Task=`20ec86fa-3514-4c19-bd00-c598b455f5c0`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-5089::DEBUG::2012-12-02 23:10:54,673::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-5276::DEBUG::2012-12-02 23:10:54,673::task::568::TaskManager.Task::(_updateState) Task=`20ec86fa-3514-4c19-bd00-c598b455f5c0`::moving from state preparing -> state finished >Thread-5089::DEBUG::2012-12-02 23:10:54,673::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-5276::DEBUG::2012-12-02 23:10:54,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5276::DEBUG::2012-12-02 23:10:54,674::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5276::DEBUG::2012-12-02 23:10:54,674::task::957::TaskManager.Task::(_decref) Task=`20ec86fa-3514-4c19-bd00-c598b455f5c0`::ref 0 aborting False >Thread-5089::DEBUG::2012-12-02 23:10:54,680::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >Thread-5089::DEBUG::2012-12-02 23:10:54,680::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,681::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5089::DEBUG::2012-12-02 23:10:54,888::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5089::WARNING::2012-12-02 23:10:54,889::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5089::DEBUG::2012-12-02 23:10:54,890::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5089::ERROR::2012-12-02 23:10:54,919::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5089::DEBUG::2012-12-02 23:10:54,919::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-5089::DEBUG::2012-12-02 23:10:54,919::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5089::DEBUG::2012-12-02 23:10:54,920::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5402::DEBUG::2012-12-02 23:10:54,961::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5402::DEBUG::2012-12-02 23:10:54,961::task::568::TaskManager.Task::(_updateState) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::moving from state init -> state preparing >Thread-5402::INFO::2012-12-02 23:10:54,961::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', prevID=-1, prevLVER='-1', recoveryMode=None, scsiFencing='false', maxHostID=250, domVersion='3', options=None) >Thread-5402::DEBUG::2012-12-02 23:10:54,961::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`54b2497c-7e39-49e6-9dca-02ac52a45fcf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5402::DEBUG::2012-12-02 23:10:54,962::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-5402::DEBUG::2012-12-02 23:10:54,962::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is currently locked, Entering queue (1 in queue) >Thread-5089::DEBUG::2012-12-02 23:10:55,128::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5089::WARNING::2012-12-02 23:10:55,130::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5089::DEBUG::2012-12-02 23:10:55,130::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5089::INFO::2012-12-02 23:10:55,157::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5089::INFO::2012-12-02 23:10:55,158::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-5089::INFO::2012-12-02 23:10:55,158::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-5089::DEBUG::2012-12-02 23:10:55,158::task::1151::TaskManager.Task::(prepare) Task=`b84b2658-116a-45ec-bc6c-4fb5eaac5c1b`::finished: None >Thread-5089::DEBUG::2012-12-02 23:10:55,158::task::568::TaskManager.Task::(_updateState) Task=`b84b2658-116a-45ec-bc6c-4fb5eaac5c1b`::moving from state preparing -> state finished >Thread-5089::DEBUG::2012-12-02 23:10:55,158::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5089::DEBUG::2012-12-02 23:10:55,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5089::DEBUG::2012-12-02 23:10:55,159::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5089::DEBUG::2012-12-02 23:10:55,159::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5089::DEBUG::2012-12-02 23:10:55,159::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5089::DEBUG::2012-12-02 23:10:55,159::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' has 1 requests in queue. Handling top request. >Thread-5089::DEBUG::2012-12-02 23:10:55,159::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`54b2497c-7e39-49e6-9dca-02ac52a45fcf`::Granted request >Thread-5089::DEBUG::2012-12-02 23:10:55,160::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.479cf195-bb99-45c5-b400-433d810eaa48 - exclusive: granted' was granted >Thread-5089::DEBUG::2012-12-02 23:10:55,160::task::957::TaskManager.Task::(_decref) Task=`b84b2658-116a-45ec-bc6c-4fb5eaac5c1b`::ref 0 aborting False >Thread-5402::DEBUG::2012-12-02 23:10:55,160::task::794::TaskManager.Task::(resourceAcquired) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-5402::DEBUG::2012-12-02 23:10:55,160::task::957::TaskManager.Task::(_decref) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::ref 1 aborting False >Thread-5402::INFO::2012-12-02 23:10:55,161::logUtils::39::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-5402::DEBUG::2012-12-02 23:10:55,161::task::1146::TaskManager.Task::(prepare) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::Prepare: 1 jobs exist, move to acquiring >Thread-5402::DEBUG::2012-12-02 23:10:55,161::task::568::TaskManager.Task::(_updateState) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::moving from state preparing -> state acquiring >Thread-5402::DEBUG::2012-12-02 23:10:55,161::task::568::TaskManager.Task::(_updateState) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::moving from state acquiring -> state queued >Thread-5402::DEBUG::2012-12-02 23:10:55,161::taskManager::50::TaskManager::(_queueTask) queuing task: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5402::DEBUG::2012-12-02 23:10:55,162::taskManager::56::TaskManager::(_queueTask) task queued: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-7::DEBUG::2012-12-02 23:10:55,162::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-5402::DEBUG::2012-12-02 23:10:55,162::task::1148::TaskManager.Task::(prepare) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::returning >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,162::threadPool::212::Misc.ThreadPool.WorkerThread::(run) Task: 3d8ad152-e945-43ea-bdef-fe11d53183fd running: <bound method Task.commit of <storage.task.Task instance at 0x7f191845ca70>> with: None >Thread-5402::DEBUG::2012-12-02 23:10:55,162::task::957::TaskManager.Task::(_decref) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::ref 0 aborting False >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,162::task::1159::TaskManager.Task::(commit) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::committing task: 3d8ad152-e945-43ea-bdef-fe11d53183fd >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,163::task::568::TaskManager.Task::(_updateState) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::moving from state queued -> state running >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,163::task::872::TaskManager.Task::(_runJobs) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f19184ddc90>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,163::task::304::TaskManager.Task::(run) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7f19184ddc90>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) callback None >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,163::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,164::misc::1063::SamplingMethod::(__call__) Got in to sampling method >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,171::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,172::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,172::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5464::DEBUG::2012-12-02 23:10:55,332::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5464::DEBUG::2012-12-02 23:10:55,332::task::568::TaskManager.Task::(_updateState) Task=`ca5f0d31-7745-47f8-83d2-a3f7fc94cd80`::moving from state init -> state preparing >Thread-5464::INFO::2012-12-02 23:10:55,332::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2251', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e', 'port': ''}], options=None) >Thread-5464::DEBUG::2012-12-02 23:10:55,333::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2251' (cwd None) >Thread-5464::DEBUG::2012-12-02 23:10:55,361::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-5464::DEBUG::2012-12-02 23:10:55,361::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-5464::DEBUG::2012-12-02 23:10:55,361::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-5464::DEBUG::2012-12-02 23:10:55,362::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-5464::DEBUG::2012-12-02 23:10:55,362::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,384::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:10:55,385::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,386::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:55,394::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-5464::DEBUG::2012-12-02 23:10:55,395::misc::1071::SamplingMethod::(__call__) Returning last result >3d8ad152-e945-43ea-bdef-fe11d53183fd::ERROR::2012-12-02 23:10:55,417::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,417::misc::1071::SamplingMethod::(__call__) Returning last result >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:10:55,425::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::INFO::2012-12-02 23:10:55,425::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 225bca28-f0a0-4899-b137-32acd04f57d4 (id: 2) >Thread-5464::DEBUG::2012-12-02 23:10:56,031::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-5464::DEBUG::2012-12-02 23:10:56,100::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-5464::DEBUG::2012-12-02 23:10:56,100::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:56,101::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:56,101::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:56,101::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:56,101::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:56,101::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-5464::DEBUG::2012-12-02 23:10:56,101::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-5464::INFO::2012-12-02 23:10:56,102::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e'}]} >Thread-5464::DEBUG::2012-12-02 23:10:56,102::task::1151::TaskManager.Task::(prepare) Task=`ca5f0d31-7745-47f8-83d2-a3f7fc94cd80`::finished: {'statuslist': [{'status': 0, 'id': 'd643bdcd-d433-45df-af04-f1c9231dce2e'}]} >Thread-5464::DEBUG::2012-12-02 23:10:56,102::task::568::TaskManager.Task::(_updateState) Task=`ca5f0d31-7745-47f8-83d2-a3f7fc94cd80`::moving from state preparing -> state finished >Thread-5464::DEBUG::2012-12-02 23:10:56,102::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5464::DEBUG::2012-12-02 23:10:56,102::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5464::DEBUG::2012-12-02 23:10:56,102::task::957::TaskManager.Task::(_decref) Task=`ca5f0d31-7745-47f8-83d2-a3f7fc94cd80`::ref 0 aborting False >Thread-5526::DEBUG::2012-12-02 23:10:56,317::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5526::DEBUG::2012-12-02 23:10:56,317::task::568::TaskManager.Task::(_updateState) Task=`17ac57ed-f301-442a-9a03-f273bd3c8674`::moving from state init -> state preparing >Thread-5526::INFO::2012-12-02 23:10:56,318::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5526::DEBUG::2012-12-02 23:10:56,318::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5526::DEBUG::2012-12-02 23:10:56,318::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5526::INFO::2012-12-02 23:10:56,318::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5526::DEBUG::2012-12-02 23:10:56,318::task::1151::TaskManager.Task::(prepare) Task=`17ac57ed-f301-442a-9a03-f273bd3c8674`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5526::DEBUG::2012-12-02 23:10:56,318::task::568::TaskManager.Task::(_updateState) Task=`17ac57ed-f301-442a-9a03-f273bd3c8674`::moving from state preparing -> state finished >Thread-5526::DEBUG::2012-12-02 23:10:56,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5526::DEBUG::2012-12-02 23:10:56,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5526::DEBUG::2012-12-02 23:10:56,319::task::957::TaskManager.Task::(_decref) Task=`17ac57ed-f301-442a-9a03-f273bd3c8674`::ref 0 aborting False >Thread-5527::DEBUG::2012-12-02 23:10:56,951::task::568::TaskManager.Task::(_updateState) Task=`ae07bb82-048d-4db9-97c9-494e9d528bb9`::moving from state init -> state preparing >Thread-5527::INFO::2012-12-02 23:10:56,951::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5527::INFO::2012-12-02 23:10:56,951::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00640106201172', 'lastCheck': 1354482653.4453681, 'code': 0, 'valid': True}} >Thread-5527::DEBUG::2012-12-02 23:10:56,952::task::1151::TaskManager.Task::(prepare) Task=`ae07bb82-048d-4db9-97c9-494e9d528bb9`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00640106201172', 'lastCheck': 1354482653.4453681, 'code': 0, 'valid': True}} >Thread-5527::DEBUG::2012-12-02 23:10:56,952::task::568::TaskManager.Task::(_updateState) Task=`ae07bb82-048d-4db9-97c9-494e9d528bb9`::moving from state preparing -> state finished >Thread-5527::DEBUG::2012-12-02 23:10:56,952::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5527::DEBUG::2012-12-02 23:10:56,952::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5527::DEBUG::2012-12-02 23:10:56,952::task::957::TaskManager.Task::(_decref) Task=`ae07bb82-048d-4db9-97c9-494e9d528bb9`::ref 0 aborting False >Thread-5529::DEBUG::2012-12-02 23:10:57,468::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5529::DEBUG::2012-12-02 23:10:57,468::task::568::TaskManager.Task::(_updateState) Task=`765ec310-5ca9-4c8e-8b6f-6bc481627da9`::moving from state init -> state preparing >Thread-5529::INFO::2012-12-02 23:10:57,468::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5529::DEBUG::2012-12-02 23:10:57,468::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5529::DEBUG::2012-12-02 23:10:57,469::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5529::INFO::2012-12-02 23:10:57,469::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5529::DEBUG::2012-12-02 23:10:57,469::task::1151::TaskManager.Task::(prepare) Task=`765ec310-5ca9-4c8e-8b6f-6bc481627da9`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5529::DEBUG::2012-12-02 23:10:57,469::task::568::TaskManager.Task::(_updateState) Task=`765ec310-5ca9-4c8e-8b6f-6bc481627da9`::moving from state preparing -> state finished >Thread-5529::DEBUG::2012-12-02 23:10:57,469::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5529::DEBUG::2012-12-02 23:10:57,469::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5529::DEBUG::2012-12-02 23:10:57,469::task::957::TaskManager.Task::(_decref) Task=`765ec310-5ca9-4c8e-8b6f-6bc481627da9`::ref 0 aborting False >Thread-5530::DEBUG::2012-12-02 23:10:58,605::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5530::DEBUG::2012-12-02 23:10:58,605::task::568::TaskManager.Task::(_updateState) Task=`924bce77-c5a8-4164-99bd-77e318b5f146`::moving from state init -> state preparing >Thread-5530::INFO::2012-12-02 23:10:58,606::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5530::DEBUG::2012-12-02 23:10:58,606::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5530::DEBUG::2012-12-02 23:10:58,606::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5530::INFO::2012-12-02 23:10:58,606::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5530::DEBUG::2012-12-02 23:10:58,606::task::1151::TaskManager.Task::(prepare) Task=`924bce77-c5a8-4164-99bd-77e318b5f146`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5530::DEBUG::2012-12-02 23:10:58,606::task::568::TaskManager.Task::(_updateState) Task=`924bce77-c5a8-4164-99bd-77e318b5f146`::moving from state preparing -> state finished >Thread-5530::DEBUG::2012-12-02 23:10:58,606::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5530::DEBUG::2012-12-02 23:10:58,607::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5530::DEBUG::2012-12-02 23:10:58,607::task::957::TaskManager.Task::(_decref) Task=`924bce77-c5a8-4164-99bd-77e318b5f146`::ref 0 aborting False >Thread-5532::DEBUG::2012-12-02 23:10:59,748::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5532::DEBUG::2012-12-02 23:10:59,748::task::568::TaskManager.Task::(_updateState) Task=`4dff81a7-400b-422f-a4c0-74aa2ab312cf`::moving from state init -> state preparing >Thread-5532::INFO::2012-12-02 23:10:59,749::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5532::DEBUG::2012-12-02 23:10:59,749::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5532::DEBUG::2012-12-02 23:10:59,749::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5532::INFO::2012-12-02 23:10:59,749::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5532::DEBUG::2012-12-02 23:10:59,749::task::1151::TaskManager.Task::(prepare) Task=`4dff81a7-400b-422f-a4c0-74aa2ab312cf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5532::DEBUG::2012-12-02 23:10:59,749::task::568::TaskManager.Task::(_updateState) Task=`4dff81a7-400b-422f-a4c0-74aa2ab312cf`::moving from state preparing -> state finished >Thread-5532::DEBUG::2012-12-02 23:10:59,749::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5532::DEBUG::2012-12-02 23:10:59,750::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5532::DEBUG::2012-12-02 23:10:59,750::task::957::TaskManager.Task::(_decref) Task=`4dff81a7-400b-422f-a4c0-74aa2ab312cf`::ref 0 aborting False >Thread-5533::DEBUG::2012-12-02 23:11:00,886::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5533::DEBUG::2012-12-02 23:11:00,887::task::568::TaskManager.Task::(_updateState) Task=`4a712d34-5ed3-4fa4-a563-f582bc869574`::moving from state init -> state preparing >Thread-5533::INFO::2012-12-02 23:11:00,887::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5533::DEBUG::2012-12-02 23:11:00,887::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5533::DEBUG::2012-12-02 23:11:00,887::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5533::INFO::2012-12-02 23:11:00,887::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5533::DEBUG::2012-12-02 23:11:00,887::task::1151::TaskManager.Task::(prepare) Task=`4a712d34-5ed3-4fa4-a563-f582bc869574`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5533::DEBUG::2012-12-02 23:11:00,887::task::568::TaskManager.Task::(_updateState) Task=`4a712d34-5ed3-4fa4-a563-f582bc869574`::moving from state preparing -> state finished >Thread-5533::DEBUG::2012-12-02 23:11:00,888::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5533::DEBUG::2012-12-02 23:11:00,888::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5533::DEBUG::2012-12-02 23:11:00,888::task::957::TaskManager.Task::(_decref) Task=`4a712d34-5ed3-4fa4-a563-f582bc869574`::ref 0 aborting False >Thread-5535::DEBUG::2012-12-02 23:11:02,041::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5535::DEBUG::2012-12-02 23:11:02,042::task::568::TaskManager.Task::(_updateState) Task=`1b788702-0927-46d3-98e9-3a9fa643afb6`::moving from state init -> state preparing >Thread-5535::INFO::2012-12-02 23:11:02,042::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5535::DEBUG::2012-12-02 23:11:02,042::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5535::DEBUG::2012-12-02 23:11:02,042::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5535::INFO::2012-12-02 23:11:02,042::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5535::DEBUG::2012-12-02 23:11:02,042::task::1151::TaskManager.Task::(prepare) Task=`1b788702-0927-46d3-98e9-3a9fa643afb6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5535::DEBUG::2012-12-02 23:11:02,043::task::568::TaskManager.Task::(_updateState) Task=`1b788702-0927-46d3-98e9-3a9fa643afb6`::moving from state preparing -> state finished >Thread-5535::DEBUG::2012-12-02 23:11:02,043::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5535::DEBUG::2012-12-02 23:11:02,043::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5535::DEBUG::2012-12-02 23:11:02,043::task::957::TaskManager.Task::(_decref) Task=`1b788702-0927-46d3-98e9-3a9fa643afb6`::ref 0 aborting False >Thread-5536::DEBUG::2012-12-02 23:11:03,190::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5536::DEBUG::2012-12-02 23:11:03,190::task::568::TaskManager.Task::(_updateState) Task=`568dbbd2-c204-4ea9-ac24-015f308d3f8e`::moving from state init -> state preparing >Thread-5536::INFO::2012-12-02 23:11:03,190::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5536::DEBUG::2012-12-02 23:11:03,190::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5536::DEBUG::2012-12-02 23:11:03,190::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5536::INFO::2012-12-02 23:11:03,190::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5536::DEBUG::2012-12-02 23:11:03,191::task::1151::TaskManager.Task::(prepare) Task=`568dbbd2-c204-4ea9-ac24-015f308d3f8e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5536::DEBUG::2012-12-02 23:11:03,191::task::568::TaskManager.Task::(_updateState) Task=`568dbbd2-c204-4ea9-ac24-015f308d3f8e`::moving from state preparing -> state finished >Thread-5536::DEBUG::2012-12-02 23:11:03,191::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5536::DEBUG::2012-12-02 23:11:03,191::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5536::DEBUG::2012-12-02 23:11:03,191::task::957::TaskManager.Task::(_decref) Task=`568dbbd2-c204-4ea9-ac24-015f308d3f8e`::ref 0 aborting False >Thread-5538::DEBUG::2012-12-02 23:11:04,342::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5538::DEBUG::2012-12-02 23:11:04,342::task::568::TaskManager.Task::(_updateState) Task=`2e6dec69-a1f2-493c-ae9f-1ff2a162f945`::moving from state init -> state preparing >Thread-5538::INFO::2012-12-02 23:11:04,342::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5538::DEBUG::2012-12-02 23:11:04,343::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5538::DEBUG::2012-12-02 23:11:04,343::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5538::INFO::2012-12-02 23:11:04,343::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5538::DEBUG::2012-12-02 23:11:04,343::task::1151::TaskManager.Task::(prepare) Task=`2e6dec69-a1f2-493c-ae9f-1ff2a162f945`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5538::DEBUG::2012-12-02 23:11:04,343::task::568::TaskManager.Task::(_updateState) Task=`2e6dec69-a1f2-493c-ae9f-1ff2a162f945`::moving from state preparing -> state finished >Thread-5538::DEBUG::2012-12-02 23:11:04,343::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5538::DEBUG::2012-12-02 23:11:04,343::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5538::DEBUG::2012-12-02 23:11:04,344::task::957::TaskManager.Task::(_decref) Task=`2e6dec69-a1f2-493c-ae9f-1ff2a162f945`::ref 0 aborting False >Thread-5539::DEBUG::2012-12-02 23:11:05,484::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5539::DEBUG::2012-12-02 23:11:05,484::task::568::TaskManager.Task::(_updateState) Task=`a17ed8be-56d5-432b-b89e-482df1e4680c`::moving from state init -> state preparing >Thread-5539::INFO::2012-12-02 23:11:05,484::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5539::DEBUG::2012-12-02 23:11:05,484::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5539::DEBUG::2012-12-02 23:11:05,484::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5539::INFO::2012-12-02 23:11:05,485::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5539::DEBUG::2012-12-02 23:11:05,485::task::1151::TaskManager.Task::(prepare) Task=`a17ed8be-56d5-432b-b89e-482df1e4680c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5539::DEBUG::2012-12-02 23:11:05,485::task::568::TaskManager.Task::(_updateState) Task=`a17ed8be-56d5-432b-b89e-482df1e4680c`::moving from state preparing -> state finished >Thread-5539::DEBUG::2012-12-02 23:11:05,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5539::DEBUG::2012-12-02 23:11:05,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5539::DEBUG::2012-12-02 23:11:05,485::task::957::TaskManager.Task::(_decref) Task=`a17ed8be-56d5-432b-b89e-482df1e4680c`::ref 0 aborting False >Thread-5541::DEBUG::2012-12-02 23:11:06,620::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5541::DEBUG::2012-12-02 23:11:06,621::task::568::TaskManager.Task::(_updateState) Task=`74bd65b3-023a-483c-9272-c8220e0eb043`::moving from state init -> state preparing >Thread-5541::INFO::2012-12-02 23:11:06,621::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5541::DEBUG::2012-12-02 23:11:06,621::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5541::DEBUG::2012-12-02 23:11:06,621::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5541::INFO::2012-12-02 23:11:06,621::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5541::DEBUG::2012-12-02 23:11:06,621::task::1151::TaskManager.Task::(prepare) Task=`74bd65b3-023a-483c-9272-c8220e0eb043`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5541::DEBUG::2012-12-02 23:11:06,622::task::568::TaskManager.Task::(_updateState) Task=`74bd65b3-023a-483c-9272-c8220e0eb043`::moving from state preparing -> state finished >Thread-5541::DEBUG::2012-12-02 23:11:06,622::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5541::DEBUG::2012-12-02 23:11:06,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5541::DEBUG::2012-12-02 23:11:06,622::task::957::TaskManager.Task::(_decref) Task=`74bd65b3-023a-483c-9272-c8220e0eb043`::ref 0 aborting False >Thread-5542::DEBUG::2012-12-02 23:11:07,756::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5542::DEBUG::2012-12-02 23:11:07,756::task::568::TaskManager.Task::(_updateState) Task=`3856381a-cfcd-4941-afa9-7e7a000462ec`::moving from state init -> state preparing >Thread-5542::INFO::2012-12-02 23:11:07,757::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5542::DEBUG::2012-12-02 23:11:07,757::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5542::DEBUG::2012-12-02 23:11:07,757::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5542::INFO::2012-12-02 23:11:07,757::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5542::DEBUG::2012-12-02 23:11:07,757::task::1151::TaskManager.Task::(prepare) Task=`3856381a-cfcd-4941-afa9-7e7a000462ec`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5542::DEBUG::2012-12-02 23:11:07,757::task::568::TaskManager.Task::(_updateState) Task=`3856381a-cfcd-4941-afa9-7e7a000462ec`::moving from state preparing -> state finished >Thread-5542::DEBUG::2012-12-02 23:11:07,757::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5542::DEBUG::2012-12-02 23:11:07,757::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5542::DEBUG::2012-12-02 23:11:07,758::task::957::TaskManager.Task::(_decref) Task=`3856381a-cfcd-4941-afa9-7e7a000462ec`::ref 0 aborting False >Thread-5543::DEBUG::2012-12-02 23:11:07,855::task::568::TaskManager.Task::(_updateState) Task=`dcf5390f-ac62-4d01-8a55-b4158250adbc`::moving from state init -> state preparing >Thread-5543::INFO::2012-12-02 23:11:07,855::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5543::INFO::2012-12-02 23:11:07,855::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00640106201172', 'lastCheck': 1354482653.4453681, 'code': 0, 'valid': True}} >Thread-5543::DEBUG::2012-12-02 23:11:07,855::task::1151::TaskManager.Task::(prepare) Task=`dcf5390f-ac62-4d01-8a55-b4158250adbc`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00640106201172', 'lastCheck': 1354482653.4453681, 'code': 0, 'valid': True}} >Thread-5543::DEBUG::2012-12-02 23:11:07,855::task::568::TaskManager.Task::(_updateState) Task=`dcf5390f-ac62-4d01-8a55-b4158250adbc`::moving from state preparing -> state finished >Thread-5543::DEBUG::2012-12-02 23:11:07,856::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5543::DEBUG::2012-12-02 23:11:07,856::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5543::DEBUG::2012-12-02 23:11:07,856::task::957::TaskManager.Task::(_decref) Task=`dcf5390f-ac62-4d01-8a55-b4158250adbc`::ref 0 aborting False >Thread-5545::DEBUG::2012-12-02 23:11:08,903::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5545::DEBUG::2012-12-02 23:11:08,903::task::568::TaskManager.Task::(_updateState) Task=`47d8f5af-950b-4930-aff7-dcfc29c9804a`::moving from state init -> state preparing >Thread-5545::INFO::2012-12-02 23:11:08,903::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5545::DEBUG::2012-12-02 23:11:08,903::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5545::DEBUG::2012-12-02 23:11:08,903::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5545::INFO::2012-12-02 23:11:08,904::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5545::DEBUG::2012-12-02 23:11:08,904::task::1151::TaskManager.Task::(prepare) Task=`47d8f5af-950b-4930-aff7-dcfc29c9804a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5545::DEBUG::2012-12-02 23:11:08,904::task::568::TaskManager.Task::(_updateState) Task=`47d8f5af-950b-4930-aff7-dcfc29c9804a`::moving from state preparing -> state finished >Thread-5545::DEBUG::2012-12-02 23:11:08,904::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5545::DEBUG::2012-12-02 23:11:08,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5545::DEBUG::2012-12-02 23:11:08,904::task::957::TaskManager.Task::(_decref) Task=`47d8f5af-950b-4930-aff7-dcfc29c9804a`::ref 0 aborting False >Thread-5546::DEBUG::2012-12-02 23:11:10,058::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5546::DEBUG::2012-12-02 23:11:10,059::task::568::TaskManager.Task::(_updateState) Task=`febac438-823d-4d1c-98f1-9e7058bbf4d2`::moving from state init -> state preparing >Thread-5546::INFO::2012-12-02 23:11:10,059::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5546::DEBUG::2012-12-02 23:11:10,059::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5546::DEBUG::2012-12-02 23:11:10,059::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5546::INFO::2012-12-02 23:11:10,059::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5546::DEBUG::2012-12-02 23:11:10,059::task::1151::TaskManager.Task::(prepare) Task=`febac438-823d-4d1c-98f1-9e7058bbf4d2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5546::DEBUG::2012-12-02 23:11:10,059::task::568::TaskManager.Task::(_updateState) Task=`febac438-823d-4d1c-98f1-9e7058bbf4d2`::moving from state preparing -> state finished >Thread-5546::DEBUG::2012-12-02 23:11:10,059::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5546::DEBUG::2012-12-02 23:11:10,060::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5546::DEBUG::2012-12-02 23:11:10,060::task::957::TaskManager.Task::(_decref) Task=`febac438-823d-4d1c-98f1-9e7058bbf4d2`::ref 0 aborting False >Thread-5548::DEBUG::2012-12-02 23:11:11,198::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5548::DEBUG::2012-12-02 23:11:11,198::task::568::TaskManager.Task::(_updateState) Task=`57e91514-72cb-48e7-b549-8d7d5b7e0d87`::moving from state init -> state preparing >Thread-5548::INFO::2012-12-02 23:11:11,198::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5548::DEBUG::2012-12-02 23:11:11,198::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5548::DEBUG::2012-12-02 23:11:11,198::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5548::INFO::2012-12-02 23:11:11,198::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5548::DEBUG::2012-12-02 23:11:11,199::task::1151::TaskManager.Task::(prepare) Task=`57e91514-72cb-48e7-b549-8d7d5b7e0d87`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5548::DEBUG::2012-12-02 23:11:11,199::task::568::TaskManager.Task::(_updateState) Task=`57e91514-72cb-48e7-b549-8d7d5b7e0d87`::moving from state preparing -> state finished >Thread-5548::DEBUG::2012-12-02 23:11:11,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5548::DEBUG::2012-12-02 23:11:11,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5548::DEBUG::2012-12-02 23:11:11,199::task::957::TaskManager.Task::(_decref) Task=`57e91514-72cb-48e7-b549-8d7d5b7e0d87`::ref 0 aborting False >Thread-5549::DEBUG::2012-12-02 23:11:12,341::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5549::DEBUG::2012-12-02 23:11:12,342::task::568::TaskManager.Task::(_updateState) Task=`f54a5e9c-c9d5-495f-8650-08f75467ca2f`::moving from state init -> state preparing >Thread-5549::INFO::2012-12-02 23:11:12,342::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5549::DEBUG::2012-12-02 23:11:12,342::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5549::DEBUG::2012-12-02 23:11:12,342::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5549::INFO::2012-12-02 23:11:12,342::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5549::DEBUG::2012-12-02 23:11:12,342::task::1151::TaskManager.Task::(prepare) Task=`f54a5e9c-c9d5-495f-8650-08f75467ca2f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5549::DEBUG::2012-12-02 23:11:12,342::task::568::TaskManager.Task::(_updateState) Task=`f54a5e9c-c9d5-495f-8650-08f75467ca2f`::moving from state preparing -> state finished >Thread-5549::DEBUG::2012-12-02 23:11:12,343::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5549::DEBUG::2012-12-02 23:11:12,343::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5549::DEBUG::2012-12-02 23:11:12,343::task::957::TaskManager.Task::(_decref) Task=`f54a5e9c-c9d5-495f-8650-08f75467ca2f`::ref 0 aborting False >Thread-5551::DEBUG::2012-12-02 23:11:13,493::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5551::DEBUG::2012-12-02 23:11:13,493::task::568::TaskManager.Task::(_updateState) Task=`7850a9ba-db16-4c8e-91c2-c3af86edbe61`::moving from state init -> state preparing >Thread-5551::INFO::2012-12-02 23:11:13,494::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5551::DEBUG::2012-12-02 23:11:13,494::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5551::DEBUG::2012-12-02 23:11:13,494::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5551::INFO::2012-12-02 23:11:13,494::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5551::DEBUG::2012-12-02 23:11:13,494::task::1151::TaskManager.Task::(prepare) Task=`7850a9ba-db16-4c8e-91c2-c3af86edbe61`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5551::DEBUG::2012-12-02 23:11:13,494::task::568::TaskManager.Task::(_updateState) Task=`7850a9ba-db16-4c8e-91c2-c3af86edbe61`::moving from state preparing -> state finished >Thread-5551::DEBUG::2012-12-02 23:11:13,494::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5551::DEBUG::2012-12-02 23:11:13,494::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5551::DEBUG::2012-12-02 23:11:13,495::task::957::TaskManager.Task::(_decref) Task=`7850a9ba-db16-4c8e-91c2-c3af86edbe61`::ref 0 aborting False >Thread-5553::DEBUG::2012-12-02 23:11:14,633::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5553::DEBUG::2012-12-02 23:11:14,634::task::568::TaskManager.Task::(_updateState) Task=`0869ab66-a5c2-4a12-88c4-0ed1c394a695`::moving from state init -> state preparing >Thread-5553::INFO::2012-12-02 23:11:14,634::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5553::DEBUG::2012-12-02 23:11:14,634::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5553::DEBUG::2012-12-02 23:11:14,634::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5553::INFO::2012-12-02 23:11:14,634::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5553::DEBUG::2012-12-02 23:11:14,634::task::1151::TaskManager.Task::(prepare) Task=`0869ab66-a5c2-4a12-88c4-0ed1c394a695`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5553::DEBUG::2012-12-02 23:11:14,635::task::568::TaskManager.Task::(_updateState) Task=`0869ab66-a5c2-4a12-88c4-0ed1c394a695`::moving from state preparing -> state finished >Thread-5553::DEBUG::2012-12-02 23:11:14,635::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5553::DEBUG::2012-12-02 23:11:14,635::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5553::DEBUG::2012-12-02 23:11:14,635::task::957::TaskManager.Task::(_decref) Task=`0869ab66-a5c2-4a12-88c4-0ed1c394a695`::ref 0 aborting False >Thread-5554::DEBUG::2012-12-02 23:11:15,785::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5554::DEBUG::2012-12-02 23:11:15,785::task::568::TaskManager.Task::(_updateState) Task=`c57b7f9d-19af-4190-90b8-b8b944fc7846`::moving from state init -> state preparing >Thread-5554::INFO::2012-12-02 23:11:15,785::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5554::DEBUG::2012-12-02 23:11:15,785::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5554::DEBUG::2012-12-02 23:11:15,786::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5554::INFO::2012-12-02 23:11:15,786::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5554::DEBUG::2012-12-02 23:11:15,786::task::1151::TaskManager.Task::(prepare) Task=`c57b7f9d-19af-4190-90b8-b8b944fc7846`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5554::DEBUG::2012-12-02 23:11:15,786::task::568::TaskManager.Task::(_updateState) Task=`c57b7f9d-19af-4190-90b8-b8b944fc7846`::moving from state preparing -> state finished >Thread-5554::DEBUG::2012-12-02 23:11:15,786::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5554::DEBUG::2012-12-02 23:11:15,786::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5554::DEBUG::2012-12-02 23:11:15,786::task::957::TaskManager.Task::(_decref) Task=`c57b7f9d-19af-4190-90b8-b8b944fc7846`::ref 0 aborting False >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,426::safelease::178::SANLock::(acquireHostId) Host id for domain 225bca28-f0a0-4899-b137-32acd04f57d4 successfully acquired (id: 2) >3d8ad152-e945-43ea-bdef-fe11d53183fd::INFO::2012-12-02 23:11:16,427::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 225bca28-f0a0-4899-b137-32acd04f57d4 (id: 2) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,746::safelease::235::SANLock::(acquire) Cluster lock for domain 225bca28-f0a0-4899-b137-32acd04f57d4 successfully acquired (id: 2) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,746::sp::256::Storage.StoragePool::(startSpm) spm lock acquired successfully >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,751::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=91b8149373fa1c059fcdbd0209138ef3171d8521'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,751::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,751::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,751::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,778::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,778::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`93cd61e3-9468-4f62-9ce1-ce1d70bd7038`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,779::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,779::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,779::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`93cd61e3-9468-4f62-9ce1-ce1d70bd7038`::Granted request >3d8ad152-e945-43ea-bdef-fe11d53183fd::INFO::2012-12-02 23:11:16,779::sp::403::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `225bca28-f0a0-4899-b137-32acd04f57d4` >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,779::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.225bca28-f0a0-4899-b137-32acd04f57d4`ReqID=`7d605e56-1c5e-4d2c-bf77-b19c061aabfc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,780::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.225bca28-f0a0-4899-b137-32acd04f57d4' for lock type 'exclusive' >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,780::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.225bca28-f0a0-4899-b137-32acd04f57d4' is free. Now locking as 'exclusive' (1 active user) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,780::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.225bca28-f0a0-4899-b137-32acd04f57d4`ReqID=`7d605e56-1c5e-4d2c-bf77-b19c061aabfc`::Granted request >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,780::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.225bca28-f0a0-4899-b137-32acd04f57d4' >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,780::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.225bca28-f0a0-4899-b137-32acd04f57d4' (0 active users) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,781::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.225bca28-f0a0-4899-b137-32acd04f57d4' is free, finding out if anyone is waiting for it. >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,781::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.225bca28-f0a0-4899-b137-32acd04f57d4', Clearing records. >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,781::sp::407::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,781::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,782::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5556::DEBUG::2012-12-02 23:11:16,925::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5556::DEBUG::2012-12-02 23:11:16,926::task::568::TaskManager.Task::(_updateState) Task=`edce2485-ca9a-448d-a4ea-afdfeeec5871`::moving from state init -> state preparing >Thread-5556::INFO::2012-12-02 23:11:16,926::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5556::DEBUG::2012-12-02 23:11:16,926::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5556::DEBUG::2012-12-02 23:11:16,927::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5556::INFO::2012-12-02 23:11:16,927::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5556::DEBUG::2012-12-02 23:11:16,927::task::1151::TaskManager.Task::(prepare) Task=`edce2485-ca9a-448d-a4ea-afdfeeec5871`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5556::DEBUG::2012-12-02 23:11:16,927::task::568::TaskManager.Task::(_updateState) Task=`edce2485-ca9a-448d-a4ea-afdfeeec5871`::moving from state preparing -> state finished >Thread-5556::DEBUG::2012-12-02 23:11:16,927::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5556::DEBUG::2012-12-02 23:11:16,927::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5556::DEBUG::2012-12-02 23:11:16,927::task::957::TaskManager.Task::(_decref) Task=`edce2485-ca9a-448d-a4ea-afdfeeec5871`::ref 0 aborting False >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,997::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:11:16,998::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:16,999::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,033::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,033::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,047::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:11:17,047::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:11:17,047::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,047::sp::414::Storage.StoragePool::(_upgradePool) Registering with state change event >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,047::sp::416::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,048::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5618::DEBUG::2012-12-02 23:11:17,048::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,048::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5618::DEBUG::2012-12-02 23:11:17,049::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,049::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,053::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5618::DEBUG::2012-12-02 23:11:17,258::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5618::WARNING::2012-12-02 23:11:17,259::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5618::DEBUG::2012-12-02 23:11:17,260::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:11:17,307::sd::583::Storage.StorageDomain::(createMasterTree) vms dir not found, creating (/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4/master/vms) >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:11:17,318::sd::588::Storage.StorageDomain::(createMasterTree) tasks dir not found, creating (/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4/master/tasks) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,330::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,331::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,536::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >3d8ad152-e945-43ea-bdef-fe11d53183fd::WARNING::2012-12-02 23:11:17,538::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,538::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3d8ad152-e945-43ea-bdef-fe11d53183fd::ERROR::2012-12-02 23:11:17,567::sp::277::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 275, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper > return f(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1568, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,571::sp::305::Storage.StoragePool::(startSpm) ended. >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,571::task::568::TaskManager.Task::(_updateState) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::moving from state running -> state finished >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,571::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,571::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,572::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,572::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,572::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,572::task::880::TaskManager.Task::(_runJobs) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::Task.run: exit - success: result >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,572::task::957::TaskManager.Task::(_decref) Task=`3d8ad152-e945-43ea-bdef-fe11d53183fd`::ref 0 aborting False >3d8ad152-e945-43ea-bdef-fe11d53183fd::DEBUG::2012-12-02 23:11:17,572::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-5741::DEBUG::2012-12-02 23:11:18,077::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5741::DEBUG::2012-12-02 23:11:18,078::task::568::TaskManager.Task::(_updateState) Task=`34321dbc-8984-4ff3-b5a4-08e06ee2d90e`::moving from state init -> state preparing >Thread-5741::INFO::2012-12-02 23:11:18,078::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5741::DEBUG::2012-12-02 23:11:18,078::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5741::DEBUG::2012-12-02 23:11:18,078::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'} >Thread-5741::INFO::2012-12-02 23:11:18,078::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5741::DEBUG::2012-12-02 23:11:18,079::task::1151::TaskManager.Task::(prepare) Task=`34321dbc-8984-4ff3-b5a4-08e06ee2d90e`::finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '3d8ad152-e945-43ea-bdef-fe11d53183fd'}} >Thread-5741::DEBUG::2012-12-02 23:11:18,079::task::568::TaskManager.Task::(_updateState) Task=`34321dbc-8984-4ff3-b5a4-08e06ee2d90e`::moving from state preparing -> state finished >Thread-5741::DEBUG::2012-12-02 23:11:18,079::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5741::DEBUG::2012-12-02 23:11:18,079::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5741::DEBUG::2012-12-02 23:11:18,079::task::957::TaskManager.Task::(_decref) Task=`34321dbc-8984-4ff3-b5a4-08e06ee2d90e`::ref 0 aborting False >Thread-5742::DEBUG::2012-12-02 23:11:18,216::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5742::DEBUG::2012-12-02 23:11:18,216::task::568::TaskManager.Task::(_updateState) Task=`83f76db1-d31e-46a9-91e9-a4813385c5bc`::moving from state init -> state preparing >Thread-5742::INFO::2012-12-02 23:11:18,217::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5742::INFO::2012-12-02 23:11:18,217::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5742::DEBUG::2012-12-02 23:11:18,217::task::1151::TaskManager.Task::(prepare) Task=`83f76db1-d31e-46a9-91e9-a4813385c5bc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5742::DEBUG::2012-12-02 23:11:18,217::task::568::TaskManager.Task::(_updateState) Task=`83f76db1-d31e-46a9-91e9-a4813385c5bc`::moving from state preparing -> state finished >Thread-5742::DEBUG::2012-12-02 23:11:18,217::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5742::DEBUG::2012-12-02 23:11:18,217::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5742::DEBUG::2012-12-02 23:11:18,217::task::957::TaskManager.Task::(_decref) Task=`83f76db1-d31e-46a9-91e9-a4813385c5bc`::ref 0 aborting False >Thread-5743::DEBUG::2012-12-02 23:11:18,360::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5743::DEBUG::2012-12-02 23:11:18,361::task::568::TaskManager.Task::(_updateState) Task=`7880a47d-f2e6-4309-8666-4f52ae43863d`::moving from state init -> state preparing >Thread-5743::INFO::2012-12-02 23:11:18,361::logUtils::37::dispatcher::(wrapper) Run and protect: clearTask(taskID='3d8ad152-e945-43ea-bdef-fe11d53183fd', spUUID=None, options=None) >Thread-5743::DEBUG::2012-12-02 23:11:18,361::taskManager::157::TaskManager::(clearTask) Entry. taskID: 3d8ad152-e945-43ea-bdef-fe11d53183fd >Thread-5743::DEBUG::2012-12-02 23:11:18,361::taskManager::162::TaskManager::(clearTask) Return. >Thread-5743::INFO::2012-12-02 23:11:18,361::logUtils::39::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-5743::DEBUG::2012-12-02 23:11:18,361::task::1151::TaskManager.Task::(prepare) Task=`7880a47d-f2e6-4309-8666-4f52ae43863d`::finished: None >Thread-5743::DEBUG::2012-12-02 23:11:18,361::task::568::TaskManager.Task::(_updateState) Task=`7880a47d-f2e6-4309-8666-4f52ae43863d`::moving from state preparing -> state finished >Thread-5743::DEBUG::2012-12-02 23:11:18,362::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5743::DEBUG::2012-12-02 23:11:18,362::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5743::DEBUG::2012-12-02 23:11:18,362::task::957::TaskManager.Task::(_decref) Task=`7880a47d-f2e6-4309-8666-4f52ae43863d`::ref 0 aborting False >Thread-5744::DEBUG::2012-12-02 23:11:18,563::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5744::DEBUG::2012-12-02 23:11:18,563::task::568::TaskManager.Task::(_updateState) Task=`9dd34004-e0a5-4939-9a8c-af490763645b`::moving from state init -> state preparing >Thread-5744::INFO::2012-12-02 23:11:18,563::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5744::DEBUG::2012-12-02 23:11:18,564::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`06b5ef7f-fbe2-4ebd-91a0-52dc0ed9b651`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5744::DEBUG::2012-12-02 23:11:18,564::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5744::DEBUG::2012-12-02 23:11:18,564::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5744::DEBUG::2012-12-02 23:11:18,564::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`06b5ef7f-fbe2-4ebd-91a0-52dc0ed9b651`::Granted request >Thread-5744::DEBUG::2012-12-02 23:11:18,564::task::794::TaskManager.Task::(resourceAcquired) Task=`9dd34004-e0a5-4939-9a8c-af490763645b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5744::DEBUG::2012-12-02 23:11:18,565::task::957::TaskManager.Task::(_decref) Task=`9dd34004-e0a5-4939-9a8c-af490763645b`::ref 1 aborting False >Thread-5744::DEBUG::2012-12-02 23:11:18,571::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5744::DEBUG::2012-12-02 23:11:18,571::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5744::DEBUG::2012-12-02 23:11:18,784::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5744::WARNING::2012-12-02 23:11:18,785::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5744::DEBUG::2012-12-02 23:11:18,786::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5793::DEBUG::2012-12-02 23:11:18,813::task::568::TaskManager.Task::(_updateState) Task=`3578da0b-6ece-40db-9a91-e01f45921113`::moving from state init -> state preparing >Thread-5793::INFO::2012-12-02 23:11:18,814::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5793::INFO::2012-12-02 23:11:18,815::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00666189193726', 'lastCheck': 1354482676.4271059, 'code': 0, 'valid': True}} >Thread-5793::DEBUG::2012-12-02 23:11:18,815::task::1151::TaskManager.Task::(prepare) Task=`3578da0b-6ece-40db-9a91-e01f45921113`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00666189193726', 'lastCheck': 1354482676.4271059, 'code': 0, 'valid': True}} >Thread-5793::DEBUG::2012-12-02 23:11:18,816::task::568::TaskManager.Task::(_updateState) Task=`3578da0b-6ece-40db-9a91-e01f45921113`::moving from state preparing -> state finished >Thread-5793::DEBUG::2012-12-02 23:11:18,816::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5793::DEBUG::2012-12-02 23:11:18,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5793::DEBUG::2012-12-02 23:11:18,817::task::957::TaskManager.Task::(_decref) Task=`3578da0b-6ece-40db-9a91-e01f45921113`::ref 0 aborting False >Thread-5744::WARNING::2012-12-02 23:11:18,821::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5744::INFO::2012-12-02 23:11:18,821::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5744::DEBUG::2012-12-02 23:11:18,821::task::1151::TaskManager.Task::(prepare) Task=`9dd34004-e0a5-4939-9a8c-af490763645b`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5744::DEBUG::2012-12-02 23:11:18,821::task::568::TaskManager.Task::(_updateState) Task=`9dd34004-e0a5-4939-9a8c-af490763645b`::moving from state preparing -> state finished >Thread-5744::DEBUG::2012-12-02 23:11:18,821::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5744::DEBUG::2012-12-02 23:11:18,821::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5744::DEBUG::2012-12-02 23:11:18,822::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5744::DEBUG::2012-12-02 23:11:18,822::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5744::DEBUG::2012-12-02 23:11:18,822::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5744::DEBUG::2012-12-02 23:11:18,822::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5744::DEBUG::2012-12-02 23:11:18,822::task::957::TaskManager.Task::(_decref) Task=`9dd34004-e0a5-4939-9a8c-af490763645b`::ref 0 aborting False >Thread-5808::DEBUG::2012-12-02 23:11:19,027::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5808::DEBUG::2012-12-02 23:11:19,027::task::568::TaskManager.Task::(_updateState) Task=`4b336eb7-473d-41e3-985f-96eaeabac244`::moving from state init -> state preparing >Thread-5808::INFO::2012-12-02 23:11:19,027::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-5808::DEBUG::2012-12-02 23:11:19,028::taskManager::177::TaskManager::(getAllTasksInfo) Entry. >Thread-5808::DEBUG::2012-12-02 23:11:19,028::taskManager::188::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-5808::INFO::2012-12-02 23:11:19,028::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-5808::DEBUG::2012-12-02 23:11:19,028::task::1151::TaskManager.Task::(prepare) Task=`4b336eb7-473d-41e3-985f-96eaeabac244`::finished: {'allTasksInfo': {}} >Thread-5808::DEBUG::2012-12-02 23:11:19,028::task::568::TaskManager.Task::(_updateState) Task=`4b336eb7-473d-41e3-985f-96eaeabac244`::moving from state preparing -> state finished >Thread-5808::DEBUG::2012-12-02 23:11:19,028::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5808::DEBUG::2012-12-02 23:11:19,028::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5808::DEBUG::2012-12-02 23:11:19,028::task::957::TaskManager.Task::(_decref) Task=`4b336eb7-473d-41e3-985f-96eaeabac244`::ref 0 aborting False >Thread-5809::DEBUG::2012-12-02 23:11:19,174::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5809::DEBUG::2012-12-02 23:11:19,174::task::568::TaskManager.Task::(_updateState) Task=`4d1a2849-f0e4-47ee-9c50-c7904a957ac6`::moving from state init -> state preparing >Thread-5809::INFO::2012-12-02 23:11:19,174::logUtils::37::dispatcher::(wrapper) Run and protect: updateVM(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', vmList=[], sdUUID=None, options=None) >Thread-5809::DEBUG::2012-12-02 23:11:19,175::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`25b39230-790c-4058-a1a6-6b1c74c3bce0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5809::DEBUG::2012-12-02 23:11:19,175::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5809::DEBUG::2012-12-02 23:11:19,175::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5809::DEBUG::2012-12-02 23:11:19,175::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`25b39230-790c-4058-a1a6-6b1c74c3bce0`::Granted request >Thread-5809::DEBUG::2012-12-02 23:11:19,175::task::794::TaskManager.Task::(resourceAcquired) Task=`4d1a2849-f0e4-47ee-9c50-c7904a957ac6`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5809::DEBUG::2012-12-02 23:11:19,175::task::957::TaskManager.Task::(_decref) Task=`4d1a2849-f0e4-47ee-9c50-c7904a957ac6`::ref 1 aborting False >Thread-5809::INFO::2012-12-02 23:11:19,176::sp::1247::Storage.StoragePool::(updateVM) spUUID=479cf195-bb99-45c5-b400-433d810eaa48 sdUUID=225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5809::INFO::2012-12-02 23:11:19,177::logUtils::39::dispatcher::(wrapper) Run and protect: updateVM, Return response: None >Thread-5809::DEBUG::2012-12-02 23:11:19,177::task::1151::TaskManager.Task::(prepare) Task=`4d1a2849-f0e4-47ee-9c50-c7904a957ac6`::finished: None >Thread-5809::DEBUG::2012-12-02 23:11:19,177::task::568::TaskManager.Task::(_updateState) Task=`4d1a2849-f0e4-47ee-9c50-c7904a957ac6`::moving from state preparing -> state finished >Thread-5809::DEBUG::2012-12-02 23:11:19,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5809::DEBUG::2012-12-02 23:11:19,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5809::DEBUG::2012-12-02 23:11:19,178::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5809::DEBUG::2012-12-02 23:11:19,178::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5809::DEBUG::2012-12-02 23:11:19,178::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5809::DEBUG::2012-12-02 23:11:19,178::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5809::DEBUG::2012-12-02 23:11:19,178::task::957::TaskManager.Task::(_decref) Task=`4d1a2849-f0e4-47ee-9c50-c7904a957ac6`::ref 0 aborting False >Thread-5814::DEBUG::2012-12-02 23:11:29,023::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5814::DEBUG::2012-12-02 23:11:29,023::task::568::TaskManager.Task::(_updateState) Task=`a9a44516-52e0-417f-b964-808b4867e2ab`::moving from state init -> state preparing >Thread-5814::INFO::2012-12-02 23:11:29,023::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5814::INFO::2012-12-02 23:11:29,023::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5814::DEBUG::2012-12-02 23:11:29,023::task::1151::TaskManager.Task::(prepare) Task=`a9a44516-52e0-417f-b964-808b4867e2ab`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5814::DEBUG::2012-12-02 23:11:29,024::task::568::TaskManager.Task::(_updateState) Task=`a9a44516-52e0-417f-b964-808b4867e2ab`::moving from state preparing -> state finished >Thread-5814::DEBUG::2012-12-02 23:11:29,024::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5814::DEBUG::2012-12-02 23:11:29,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5814::DEBUG::2012-12-02 23:11:29,024::task::957::TaskManager.Task::(_decref) Task=`a9a44516-52e0-417f-b964-808b4867e2ab`::ref 0 aborting False >Thread-5815::DEBUG::2012-12-02 23:11:29,171::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5815::DEBUG::2012-12-02 23:11:29,172::task::568::TaskManager.Task::(_updateState) Task=`efb1b273-e79e-4aa4-b999-28e79f13304c`::moving from state init -> state preparing >Thread-5815::INFO::2012-12-02 23:11:29,172::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5815::DEBUG::2012-12-02 23:11:29,172::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`bcf01baf-8c58-4ca0-807a-6dd109e13889`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5815::DEBUG::2012-12-02 23:11:29,172::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5815::DEBUG::2012-12-02 23:11:29,173::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5815::DEBUG::2012-12-02 23:11:29,173::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`bcf01baf-8c58-4ca0-807a-6dd109e13889`::Granted request >Thread-5815::DEBUG::2012-12-02 23:11:29,173::task::794::TaskManager.Task::(resourceAcquired) Task=`efb1b273-e79e-4aa4-b999-28e79f13304c`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5815::DEBUG::2012-12-02 23:11:29,173::task::957::TaskManager.Task::(_decref) Task=`efb1b273-e79e-4aa4-b999-28e79f13304c`::ref 1 aborting False >Thread-5815::DEBUG::2012-12-02 23:11:29,178::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5815::DEBUG::2012-12-02 23:11:29,178::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5815::DEBUG::2012-12-02 23:11:29,387::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5815::WARNING::2012-12-02 23:11:29,388::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5815::DEBUG::2012-12-02 23:11:29,388::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5815::WARNING::2012-12-02 23:11:29,417::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5815::INFO::2012-12-02 23:11:29,417::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5815::DEBUG::2012-12-02 23:11:29,417::task::1151::TaskManager.Task::(prepare) Task=`efb1b273-e79e-4aa4-b999-28e79f13304c`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5815::DEBUG::2012-12-02 23:11:29,417::task::568::TaskManager.Task::(_updateState) Task=`efb1b273-e79e-4aa4-b999-28e79f13304c`::moving from state preparing -> state finished >Thread-5815::DEBUG::2012-12-02 23:11:29,418::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5815::DEBUG::2012-12-02 23:11:29,418::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5815::DEBUG::2012-12-02 23:11:29,418::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5815::DEBUG::2012-12-02 23:11:29,418::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5815::DEBUG::2012-12-02 23:11:29,418::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5815::DEBUG::2012-12-02 23:11:29,418::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5815::DEBUG::2012-12-02 23:11:29,419::task::957::TaskManager.Task::(_decref) Task=`efb1b273-e79e-4aa4-b999-28e79f13304c`::ref 0 aborting False >Thread-5877::DEBUG::2012-12-02 23:11:29,710::task::568::TaskManager.Task::(_updateState) Task=`2315781c-6a0a-4991-b3a8-98cb28a1c39d`::moving from state init -> state preparing >Thread-5877::INFO::2012-12-02 23:11:29,710::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5877::INFO::2012-12-02 23:11:29,710::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00739598274231', 'lastCheck': 1354482686.4454889, 'code': 0, 'valid': True}} >Thread-5877::DEBUG::2012-12-02 23:11:29,710::task::1151::TaskManager.Task::(prepare) Task=`2315781c-6a0a-4991-b3a8-98cb28a1c39d`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00739598274231', 'lastCheck': 1354482686.4454889, 'code': 0, 'valid': True}} >Thread-5877::DEBUG::2012-12-02 23:11:29,711::task::568::TaskManager.Task::(_updateState) Task=`2315781c-6a0a-4991-b3a8-98cb28a1c39d`::moving from state preparing -> state finished >Thread-5877::DEBUG::2012-12-02 23:11:29,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5877::DEBUG::2012-12-02 23:11:29,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5877::DEBUG::2012-12-02 23:11:29,711::task::957::TaskManager.Task::(_decref) Task=`2315781c-6a0a-4991-b3a8-98cb28a1c39d`::ref 0 aborting False >Thread-5883::DEBUG::2012-12-02 23:11:39,658::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5883::DEBUG::2012-12-02 23:11:39,658::task::568::TaskManager.Task::(_updateState) Task=`ff549fed-8994-4d66-ad9c-7e8b9aed8270`::moving from state init -> state preparing >Thread-5883::INFO::2012-12-02 23:11:39,658::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5883::INFO::2012-12-02 23:11:39,658::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5883::DEBUG::2012-12-02 23:11:39,658::task::1151::TaskManager.Task::(prepare) Task=`ff549fed-8994-4d66-ad9c-7e8b9aed8270`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5883::DEBUG::2012-12-02 23:11:39,659::task::568::TaskManager.Task::(_updateState) Task=`ff549fed-8994-4d66-ad9c-7e8b9aed8270`::moving from state preparing -> state finished >Thread-5883::DEBUG::2012-12-02 23:11:39,659::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5883::DEBUG::2012-12-02 23:11:39,659::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5883::DEBUG::2012-12-02 23:11:39,659::task::957::TaskManager.Task::(_decref) Task=`ff549fed-8994-4d66-ad9c-7e8b9aed8270`::ref 0 aborting False >Thread-5884::DEBUG::2012-12-02 23:11:39,798::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5884::DEBUG::2012-12-02 23:11:39,799::task::568::TaskManager.Task::(_updateState) Task=`4c0a8fd9-a8ae-4b18-8bff-ab96d52c7504`::moving from state init -> state preparing >Thread-5884::INFO::2012-12-02 23:11:39,799::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5884::DEBUG::2012-12-02 23:11:39,799::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`935e5780-ce8c-4e27-ba1a-92d163871448`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5884::DEBUG::2012-12-02 23:11:39,799::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5884::DEBUG::2012-12-02 23:11:39,800::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5884::DEBUG::2012-12-02 23:11:39,800::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`935e5780-ce8c-4e27-ba1a-92d163871448`::Granted request >Thread-5884::DEBUG::2012-12-02 23:11:39,800::task::794::TaskManager.Task::(resourceAcquired) Task=`4c0a8fd9-a8ae-4b18-8bff-ab96d52c7504`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5884::DEBUG::2012-12-02 23:11:39,800::task::957::TaskManager.Task::(_decref) Task=`4c0a8fd9-a8ae-4b18-8bff-ab96d52c7504`::ref 1 aborting False >Thread-5884::DEBUG::2012-12-02 23:11:39,806::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5884::DEBUG::2012-12-02 23:11:39,806::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5884::DEBUG::2012-12-02 23:11:40,016::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5884::WARNING::2012-12-02 23:11:40,018::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5884::DEBUG::2012-12-02 23:11:40,018::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5884::WARNING::2012-12-02 23:11:40,048::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5884::INFO::2012-12-02 23:11:40,048::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5884::DEBUG::2012-12-02 23:11:40,048::task::1151::TaskManager.Task::(prepare) Task=`4c0a8fd9-a8ae-4b18-8bff-ab96d52c7504`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5884::DEBUG::2012-12-02 23:11:40,049::task::568::TaskManager.Task::(_updateState) Task=`4c0a8fd9-a8ae-4b18-8bff-ab96d52c7504`::moving from state preparing -> state finished >Thread-5884::DEBUG::2012-12-02 23:11:40,049::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5884::DEBUG::2012-12-02 23:11:40,049::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5884::DEBUG::2012-12-02 23:11:40,049::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5884::DEBUG::2012-12-02 23:11:40,049::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5884::DEBUG::2012-12-02 23:11:40,049::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5884::DEBUG::2012-12-02 23:11:40,049::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5884::DEBUG::2012-12-02 23:11:40,050::task::957::TaskManager.Task::(_decref) Task=`4c0a8fd9-a8ae-4b18-8bff-ab96d52c7504`::ref 0 aborting False >Thread-5946::DEBUG::2012-12-02 23:11:40,636::task::568::TaskManager.Task::(_updateState) Task=`e8e60dca-acb6-4775-baaa-f54733fc89ed`::moving from state init -> state preparing >Thread-5946::INFO::2012-12-02 23:11:40,637::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5946::INFO::2012-12-02 23:11:40,637::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00747108459473', 'lastCheck': 1354482696.463124, 'code': 0, 'valid': True}} >Thread-5946::DEBUG::2012-12-02 23:11:40,637::task::1151::TaskManager.Task::(prepare) Task=`e8e60dca-acb6-4775-baaa-f54733fc89ed`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00747108459473', 'lastCheck': 1354482696.463124, 'code': 0, 'valid': True}} >Thread-5946::DEBUG::2012-12-02 23:11:40,637::task::568::TaskManager.Task::(_updateState) Task=`e8e60dca-acb6-4775-baaa-f54733fc89ed`::moving from state preparing -> state finished >Thread-5946::DEBUG::2012-12-02 23:11:40,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5946::DEBUG::2012-12-02 23:11:40,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5946::DEBUG::2012-12-02 23:11:40,637::task::957::TaskManager.Task::(_decref) Task=`e8e60dca-acb6-4775-baaa-f54733fc89ed`::ref 0 aborting False >Thread-5952::DEBUG::2012-12-02 23:11:50,291::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5952::DEBUG::2012-12-02 23:11:50,291::task::568::TaskManager.Task::(_updateState) Task=`8ef78752-2cfe-4af2-b508-4ae7722686d2`::moving from state init -> state preparing >Thread-5952::INFO::2012-12-02 23:11:50,291::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5952::INFO::2012-12-02 23:11:50,292::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5952::DEBUG::2012-12-02 23:11:50,292::task::1151::TaskManager.Task::(prepare) Task=`8ef78752-2cfe-4af2-b508-4ae7722686d2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5952::DEBUG::2012-12-02 23:11:50,292::task::568::TaskManager.Task::(_updateState) Task=`8ef78752-2cfe-4af2-b508-4ae7722686d2`::moving from state preparing -> state finished >Thread-5952::DEBUG::2012-12-02 23:11:50,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5952::DEBUG::2012-12-02 23:11:50,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5952::DEBUG::2012-12-02 23:11:50,292::task::957::TaskManager.Task::(_decref) Task=`8ef78752-2cfe-4af2-b508-4ae7722686d2`::ref 0 aborting False >Thread-5953::DEBUG::2012-12-02 23:11:50,439::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5953::DEBUG::2012-12-02 23:11:50,440::task::568::TaskManager.Task::(_updateState) Task=`ee754b2b-3a8f-45e9-a5b9-bb0dc90e266f`::moving from state init -> state preparing >Thread-5953::INFO::2012-12-02 23:11:50,440::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5953::DEBUG::2012-12-02 23:11:50,440::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`85e8bb3c-98c2-43c0-9a99-bab42cab5c3c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5953::DEBUG::2012-12-02 23:11:50,440::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5953::DEBUG::2012-12-02 23:11:50,441::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5953::DEBUG::2012-12-02 23:11:50,441::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`85e8bb3c-98c2-43c0-9a99-bab42cab5c3c`::Granted request >Thread-5953::DEBUG::2012-12-02 23:11:50,441::task::794::TaskManager.Task::(resourceAcquired) Task=`ee754b2b-3a8f-45e9-a5b9-bb0dc90e266f`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5953::DEBUG::2012-12-02 23:11:50,441::task::957::TaskManager.Task::(_decref) Task=`ee754b2b-3a8f-45e9-a5b9-bb0dc90e266f`::ref 1 aborting False >Thread-5953::DEBUG::2012-12-02 23:11:50,447::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5953::DEBUG::2012-12-02 23:11:50,447::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5953::DEBUG::2012-12-02 23:11:50,653::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5953::WARNING::2012-12-02 23:11:50,655::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5953::DEBUG::2012-12-02 23:11:50,655::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5953::WARNING::2012-12-02 23:11:50,685::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5953::INFO::2012-12-02 23:11:50,685::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5953::DEBUG::2012-12-02 23:11:50,685::task::1151::TaskManager.Task::(prepare) Task=`ee754b2b-3a8f-45e9-a5b9-bb0dc90e266f`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-5953::DEBUG::2012-12-02 23:11:50,685::task::568::TaskManager.Task::(_updateState) Task=`ee754b2b-3a8f-45e9-a5b9-bb0dc90e266f`::moving from state preparing -> state finished >Thread-5953::DEBUG::2012-12-02 23:11:50,685::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5953::DEBUG::2012-12-02 23:11:50,686::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5953::DEBUG::2012-12-02 23:11:50,686::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5953::DEBUG::2012-12-02 23:11:50,686::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5953::DEBUG::2012-12-02 23:11:50,686::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5953::DEBUG::2012-12-02 23:11:50,686::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5953::DEBUG::2012-12-02 23:11:50,686::task::957::TaskManager.Task::(_decref) Task=`ee754b2b-3a8f-45e9-a5b9-bb0dc90e266f`::ref 0 aborting False >Thread-6015::DEBUG::2012-12-02 23:11:51,558::task::568::TaskManager.Task::(_updateState) Task=`41a6290c-c4cd-414d-a8e3-10cf3bbb4abf`::moving from state init -> state preparing >Thread-6015::INFO::2012-12-02 23:11:51,559::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6015::INFO::2012-12-02 23:11:51,559::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00526094436646', 'lastCheck': 1354482706.478523, 'code': 0, 'valid': True}} >Thread-6015::DEBUG::2012-12-02 23:11:51,559::task::1151::TaskManager.Task::(prepare) Task=`41a6290c-c4cd-414d-a8e3-10cf3bbb4abf`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00526094436646', 'lastCheck': 1354482706.478523, 'code': 0, 'valid': True}} >Thread-6015::DEBUG::2012-12-02 23:11:51,559::task::568::TaskManager.Task::(_updateState) Task=`41a6290c-c4cd-414d-a8e3-10cf3bbb4abf`::moving from state preparing -> state finished >Thread-6015::DEBUG::2012-12-02 23:11:51,559::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6015::DEBUG::2012-12-02 23:11:51,559::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6015::DEBUG::2012-12-02 23:11:51,559::task::957::TaskManager.Task::(_decref) Task=`41a6290c-c4cd-414d-a8e3-10cf3bbb4abf`::ref 0 aborting False >Thread-6021::DEBUG::2012-12-02 23:12:00,926::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6021::DEBUG::2012-12-02 23:12:00,926::task::568::TaskManager.Task::(_updateState) Task=`16fdc0fe-3ce2-48c7-809a-c7f727a4fa36`::moving from state init -> state preparing >Thread-6021::INFO::2012-12-02 23:12:00,927::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6021::INFO::2012-12-02 23:12:00,927::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6021::DEBUG::2012-12-02 23:12:00,927::task::1151::TaskManager.Task::(prepare) Task=`16fdc0fe-3ce2-48c7-809a-c7f727a4fa36`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6021::DEBUG::2012-12-02 23:12:00,927::task::568::TaskManager.Task::(_updateState) Task=`16fdc0fe-3ce2-48c7-809a-c7f727a4fa36`::moving from state preparing -> state finished >Thread-6021::DEBUG::2012-12-02 23:12:00,927::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6021::DEBUG::2012-12-02 23:12:00,927::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6021::DEBUG::2012-12-02 23:12:00,927::task::957::TaskManager.Task::(_decref) Task=`16fdc0fe-3ce2-48c7-809a-c7f727a4fa36`::ref 0 aborting False >Thread-6022::DEBUG::2012-12-02 23:12:01,067::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6022::DEBUG::2012-12-02 23:12:01,068::task::568::TaskManager.Task::(_updateState) Task=`878970d9-8334-467a-873c-620ead47b087`::moving from state init -> state preparing >Thread-6022::INFO::2012-12-02 23:12:01,068::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6022::DEBUG::2012-12-02 23:12:01,068::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`58e73bdd-126f-4f2b-859c-a6703a4373f4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6022::DEBUG::2012-12-02 23:12:01,068::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6022::DEBUG::2012-12-02 23:12:01,068::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6022::DEBUG::2012-12-02 23:12:01,069::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`58e73bdd-126f-4f2b-859c-a6703a4373f4`::Granted request >Thread-6022::DEBUG::2012-12-02 23:12:01,069::task::794::TaskManager.Task::(resourceAcquired) Task=`878970d9-8334-467a-873c-620ead47b087`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6022::DEBUG::2012-12-02 23:12:01,069::task::957::TaskManager.Task::(_decref) Task=`878970d9-8334-467a-873c-620ead47b087`::ref 1 aborting False >Thread-6022::DEBUG::2012-12-02 23:12:01,075::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6022::DEBUG::2012-12-02 23:12:01,076::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6022::DEBUG::2012-12-02 23:12:01,280::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6022::WARNING::2012-12-02 23:12:01,281::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6022::DEBUG::2012-12-02 23:12:01,282::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6022::WARNING::2012-12-02 23:12:01,311::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6022::INFO::2012-12-02 23:12:01,312::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6022::DEBUG::2012-12-02 23:12:01,312::task::1151::TaskManager.Task::(prepare) Task=`878970d9-8334-467a-873c-620ead47b087`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6022::DEBUG::2012-12-02 23:12:01,312::task::568::TaskManager.Task::(_updateState) Task=`878970d9-8334-467a-873c-620ead47b087`::moving from state preparing -> state finished >Thread-6022::DEBUG::2012-12-02 23:12:01,312::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6022::DEBUG::2012-12-02 23:12:01,312::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6022::DEBUG::2012-12-02 23:12:01,313::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6022::DEBUG::2012-12-02 23:12:01,313::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6022::DEBUG::2012-12-02 23:12:01,313::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6022::DEBUG::2012-12-02 23:12:01,313::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6022::DEBUG::2012-12-02 23:12:01,313::task::957::TaskManager.Task::(_decref) Task=`878970d9-8334-467a-873c-620ead47b087`::ref 0 aborting False >Thread-6084::DEBUG::2012-12-02 23:12:02,472::task::568::TaskManager.Task::(_updateState) Task=`1862c20d-5d51-487f-a110-80f36a8a6040`::moving from state init -> state preparing >Thread-6084::INFO::2012-12-02 23:12:02,472::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6084::INFO::2012-12-02 23:12:02,472::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00871992111206', 'lastCheck': 1354482716.5002029, 'code': 0, 'valid': True}} >Thread-6084::DEBUG::2012-12-02 23:12:02,472::task::1151::TaskManager.Task::(prepare) Task=`1862c20d-5d51-487f-a110-80f36a8a6040`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00871992111206', 'lastCheck': 1354482716.5002029, 'code': 0, 'valid': True}} >Thread-6084::DEBUG::2012-12-02 23:12:02,473::task::568::TaskManager.Task::(_updateState) Task=`1862c20d-5d51-487f-a110-80f36a8a6040`::moving from state preparing -> state finished >Thread-6084::DEBUG::2012-12-02 23:12:02,473::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6084::DEBUG::2012-12-02 23:12:02,473::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6084::DEBUG::2012-12-02 23:12:02,473::task::957::TaskManager.Task::(_decref) Task=`1862c20d-5d51-487f-a110-80f36a8a6040`::ref 0 aborting False >Thread-6090::DEBUG::2012-12-02 23:12:11,519::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6090::DEBUG::2012-12-02 23:12:11,519::task::568::TaskManager.Task::(_updateState) Task=`d41670e1-94b0-442d-bc5d-462487e1fd01`::moving from state init -> state preparing >Thread-6090::INFO::2012-12-02 23:12:11,519::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6090::INFO::2012-12-02 23:12:11,520::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6090::DEBUG::2012-12-02 23:12:11,520::task::1151::TaskManager.Task::(prepare) Task=`d41670e1-94b0-442d-bc5d-462487e1fd01`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6090::DEBUG::2012-12-02 23:12:11,520::task::568::TaskManager.Task::(_updateState) Task=`d41670e1-94b0-442d-bc5d-462487e1fd01`::moving from state preparing -> state finished >Thread-6090::DEBUG::2012-12-02 23:12:11,520::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6090::DEBUG::2012-12-02 23:12:11,520::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6090::DEBUG::2012-12-02 23:12:11,520::task::957::TaskManager.Task::(_decref) Task=`d41670e1-94b0-442d-bc5d-462487e1fd01`::ref 0 aborting False >Thread-6091::DEBUG::2012-12-02 23:12:11,664::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6091::DEBUG::2012-12-02 23:12:11,665::task::568::TaskManager.Task::(_updateState) Task=`e7947dbb-c327-4d4b-afc6-eef13262c42b`::moving from state init -> state preparing >Thread-6091::INFO::2012-12-02 23:12:11,665::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6091::DEBUG::2012-12-02 23:12:11,665::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`db4d1e52-5680-41a6-9f9a-63b873f46c96`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6091::DEBUG::2012-12-02 23:12:11,665::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6091::DEBUG::2012-12-02 23:12:11,665::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6091::DEBUG::2012-12-02 23:12:11,666::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`db4d1e52-5680-41a6-9f9a-63b873f46c96`::Granted request >Thread-6091::DEBUG::2012-12-02 23:12:11,666::task::794::TaskManager.Task::(resourceAcquired) Task=`e7947dbb-c327-4d4b-afc6-eef13262c42b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6091::DEBUG::2012-12-02 23:12:11,666::task::957::TaskManager.Task::(_decref) Task=`e7947dbb-c327-4d4b-afc6-eef13262c42b`::ref 1 aborting False >Thread-6091::DEBUG::2012-12-02 23:12:11,672::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6091::DEBUG::2012-12-02 23:12:11,673::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6091::DEBUG::2012-12-02 23:12:11,879::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6091::WARNING::2012-12-02 23:12:11,881::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6091::DEBUG::2012-12-02 23:12:11,881::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6091::WARNING::2012-12-02 23:12:11,910::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6091::INFO::2012-12-02 23:12:11,911::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6091::DEBUG::2012-12-02 23:12:11,911::task::1151::TaskManager.Task::(prepare) Task=`e7947dbb-c327-4d4b-afc6-eef13262c42b`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6091::DEBUG::2012-12-02 23:12:11,911::task::568::TaskManager.Task::(_updateState) Task=`e7947dbb-c327-4d4b-afc6-eef13262c42b`::moving from state preparing -> state finished >Thread-6091::DEBUG::2012-12-02 23:12:11,911::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6091::DEBUG::2012-12-02 23:12:11,911::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6091::DEBUG::2012-12-02 23:12:11,912::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6091::DEBUG::2012-12-02 23:12:11,912::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6091::DEBUG::2012-12-02 23:12:11,912::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6091::DEBUG::2012-12-02 23:12:11,912::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6091::DEBUG::2012-12-02 23:12:11,912::task::957::TaskManager.Task::(_decref) Task=`e7947dbb-c327-4d4b-afc6-eef13262c42b`::ref 0 aborting False >Thread-6153::DEBUG::2012-12-02 23:12:13,385::task::568::TaskManager.Task::(_updateState) Task=`bfd66c95-0965-4545-bda3-b5676041fcb0`::moving from state init -> state preparing >Thread-6153::INFO::2012-12-02 23:12:13,385::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6153::INFO::2012-12-02 23:12:13,385::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0073390007019', 'lastCheck': 1354482726.519155, 'code': 0, 'valid': True}} >Thread-6153::DEBUG::2012-12-02 23:12:13,386::task::1151::TaskManager.Task::(prepare) Task=`bfd66c95-0965-4545-bda3-b5676041fcb0`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0073390007019', 'lastCheck': 1354482726.519155, 'code': 0, 'valid': True}} >Thread-6153::DEBUG::2012-12-02 23:12:13,386::task::568::TaskManager.Task::(_updateState) Task=`bfd66c95-0965-4545-bda3-b5676041fcb0`::moving from state preparing -> state finished >Thread-6153::DEBUG::2012-12-02 23:12:13,386::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6153::DEBUG::2012-12-02 23:12:13,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6153::DEBUG::2012-12-02 23:12:13,386::task::957::TaskManager.Task::(_decref) Task=`bfd66c95-0965-4545-bda3-b5676041fcb0`::ref 0 aborting False >Thread-6158::DEBUG::2012-12-02 23:12:22,122::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6158::DEBUG::2012-12-02 23:12:22,122::task::568::TaskManager.Task::(_updateState) Task=`1f462e93-8dbc-43d1-bfc8-3043d5262a23`::moving from state init -> state preparing >Thread-6158::INFO::2012-12-02 23:12:22,123::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6158::INFO::2012-12-02 23:12:22,123::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6158::DEBUG::2012-12-02 23:12:22,123::task::1151::TaskManager.Task::(prepare) Task=`1f462e93-8dbc-43d1-bfc8-3043d5262a23`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6158::DEBUG::2012-12-02 23:12:22,123::task::568::TaskManager.Task::(_updateState) Task=`1f462e93-8dbc-43d1-bfc8-3043d5262a23`::moving from state preparing -> state finished >Thread-6158::DEBUG::2012-12-02 23:12:22,123::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6158::DEBUG::2012-12-02 23:12:22,123::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6158::DEBUG::2012-12-02 23:12:22,124::task::957::TaskManager.Task::(_decref) Task=`1f462e93-8dbc-43d1-bfc8-3043d5262a23`::ref 0 aborting False >Thread-6160::DEBUG::2012-12-02 23:12:22,263::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6160::DEBUG::2012-12-02 23:12:22,264::task::568::TaskManager.Task::(_updateState) Task=`a94ccfcc-8954-4e27-b00a-c536633f2527`::moving from state init -> state preparing >Thread-6160::INFO::2012-12-02 23:12:22,264::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6160::DEBUG::2012-12-02 23:12:22,264::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`344e1c2c-80c7-45be-94d6-6270739388f9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6160::DEBUG::2012-12-02 23:12:22,264::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6160::DEBUG::2012-12-02 23:12:22,265::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6160::DEBUG::2012-12-02 23:12:22,265::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`344e1c2c-80c7-45be-94d6-6270739388f9`::Granted request >Thread-6160::DEBUG::2012-12-02 23:12:22,265::task::794::TaskManager.Task::(resourceAcquired) Task=`a94ccfcc-8954-4e27-b00a-c536633f2527`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6160::DEBUG::2012-12-02 23:12:22,265::task::957::TaskManager.Task::(_decref) Task=`a94ccfcc-8954-4e27-b00a-c536633f2527`::ref 1 aborting False >Thread-6160::DEBUG::2012-12-02 23:12:22,270::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6160::DEBUG::2012-12-02 23:12:22,270::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6160::DEBUG::2012-12-02 23:12:22,474::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6160::WARNING::2012-12-02 23:12:22,476::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6160::DEBUG::2012-12-02 23:12:22,476::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6160::WARNING::2012-12-02 23:12:22,510::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6160::INFO::2012-12-02 23:12:22,511::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6160::DEBUG::2012-12-02 23:12:22,511::task::1151::TaskManager.Task::(prepare) Task=`a94ccfcc-8954-4e27-b00a-c536633f2527`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6160::DEBUG::2012-12-02 23:12:22,511::task::568::TaskManager.Task::(_updateState) Task=`a94ccfcc-8954-4e27-b00a-c536633f2527`::moving from state preparing -> state finished >Thread-6160::DEBUG::2012-12-02 23:12:22,511::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6160::DEBUG::2012-12-02 23:12:22,512::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6160::DEBUG::2012-12-02 23:12:22,512::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6160::DEBUG::2012-12-02 23:12:22,512::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6160::DEBUG::2012-12-02 23:12:22,512::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6160::DEBUG::2012-12-02 23:12:22,512::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6160::DEBUG::2012-12-02 23:12:22,512::task::957::TaskManager.Task::(_decref) Task=`a94ccfcc-8954-4e27-b00a-c536633f2527`::ref 0 aborting False >Thread-6222::DEBUG::2012-12-02 23:12:24,316::task::568::TaskManager.Task::(_updateState) Task=`260e319f-4145-41ea-8f5b-6d41765f18be`::moving from state init -> state preparing >Thread-6222::INFO::2012-12-02 23:12:24,316::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6222::INFO::2012-12-02 23:12:24,316::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00735998153687', 'lastCheck': 1354482736.5366421, 'code': 0, 'valid': True}} >Thread-6222::DEBUG::2012-12-02 23:12:24,316::task::1151::TaskManager.Task::(prepare) Task=`260e319f-4145-41ea-8f5b-6d41765f18be`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00735998153687', 'lastCheck': 1354482736.5366421, 'code': 0, 'valid': True}} >Thread-6222::DEBUG::2012-12-02 23:12:24,316::task::568::TaskManager.Task::(_updateState) Task=`260e319f-4145-41ea-8f5b-6d41765f18be`::moving from state preparing -> state finished >Thread-6222::DEBUG::2012-12-02 23:12:24,317::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6222::DEBUG::2012-12-02 23:12:24,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6222::DEBUG::2012-12-02 23:12:24,317::task::957::TaskManager.Task::(_decref) Task=`260e319f-4145-41ea-8f5b-6d41765f18be`::ref 0 aborting False >Thread-6227::DEBUG::2012-12-02 23:12:32,705::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6227::DEBUG::2012-12-02 23:12:32,705::task::568::TaskManager.Task::(_updateState) Task=`cc3c2015-1cd0-4dc2-bbc2-f9f45adcd0fe`::moving from state init -> state preparing >Thread-6227::INFO::2012-12-02 23:12:32,706::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6227::INFO::2012-12-02 23:12:32,706::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6227::DEBUG::2012-12-02 23:12:32,706::task::1151::TaskManager.Task::(prepare) Task=`cc3c2015-1cd0-4dc2-bbc2-f9f45adcd0fe`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6227::DEBUG::2012-12-02 23:12:32,706::task::568::TaskManager.Task::(_updateState) Task=`cc3c2015-1cd0-4dc2-bbc2-f9f45adcd0fe`::moving from state preparing -> state finished >Thread-6227::DEBUG::2012-12-02 23:12:32,706::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6227::DEBUG::2012-12-02 23:12:32,706::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6227::DEBUG::2012-12-02 23:12:32,706::task::957::TaskManager.Task::(_decref) Task=`cc3c2015-1cd0-4dc2-bbc2-f9f45adcd0fe`::ref 0 aborting False >Thread-6228::DEBUG::2012-12-02 23:12:32,845::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6228::DEBUG::2012-12-02 23:12:32,845::task::568::TaskManager.Task::(_updateState) Task=`46f820f9-655f-45aa-b4db-57521ed064d2`::moving from state init -> state preparing >Thread-6228::INFO::2012-12-02 23:12:32,845::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6228::DEBUG::2012-12-02 23:12:32,846::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`34612d4a-bb46-41e6-9c47-4c70f908186c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6228::DEBUG::2012-12-02 23:12:32,846::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6228::DEBUG::2012-12-02 23:12:32,846::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6228::DEBUG::2012-12-02 23:12:32,846::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`34612d4a-bb46-41e6-9c47-4c70f908186c`::Granted request >Thread-6228::DEBUG::2012-12-02 23:12:32,846::task::794::TaskManager.Task::(resourceAcquired) Task=`46f820f9-655f-45aa-b4db-57521ed064d2`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6228::DEBUG::2012-12-02 23:12:32,847::task::957::TaskManager.Task::(_decref) Task=`46f820f9-655f-45aa-b4db-57521ed064d2`::ref 1 aborting False >Thread-6228::DEBUG::2012-12-02 23:12:32,853::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6228::DEBUG::2012-12-02 23:12:32,853::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6228::DEBUG::2012-12-02 23:12:33,062::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6228::WARNING::2012-12-02 23:12:33,063::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6228::DEBUG::2012-12-02 23:12:33,063::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6228::WARNING::2012-12-02 23:12:33,095::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6228::INFO::2012-12-02 23:12:33,096::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6228::DEBUG::2012-12-02 23:12:33,096::task::1151::TaskManager.Task::(prepare) Task=`46f820f9-655f-45aa-b4db-57521ed064d2`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6228::DEBUG::2012-12-02 23:12:33,096::task::568::TaskManager.Task::(_updateState) Task=`46f820f9-655f-45aa-b4db-57521ed064d2`::moving from state preparing -> state finished >Thread-6228::DEBUG::2012-12-02 23:12:33,096::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6228::DEBUG::2012-12-02 23:12:33,096::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6228::DEBUG::2012-12-02 23:12:33,096::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6228::DEBUG::2012-12-02 23:12:33,097::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6228::DEBUG::2012-12-02 23:12:33,097::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6228::DEBUG::2012-12-02 23:12:33,097::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6228::DEBUG::2012-12-02 23:12:33,097::task::957::TaskManager.Task::(_decref) Task=`46f820f9-655f-45aa-b4db-57521ed064d2`::ref 0 aborting False >Thread-6291::DEBUG::2012-12-02 23:12:35,212::task::568::TaskManager.Task::(_updateState) Task=`77ec933e-700e-4098-ae85-3b5165c1af4c`::moving from state init -> state preparing >Thread-6291::INFO::2012-12-02 23:12:35,212::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6291::INFO::2012-12-02 23:12:35,213::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00776600837708', 'lastCheck': 1354482746.5543981, 'code': 0, 'valid': True}} >Thread-6291::DEBUG::2012-12-02 23:12:35,213::task::1151::TaskManager.Task::(prepare) Task=`77ec933e-700e-4098-ae85-3b5165c1af4c`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00776600837708', 'lastCheck': 1354482746.5543981, 'code': 0, 'valid': True}} >Thread-6291::DEBUG::2012-12-02 23:12:35,213::task::568::TaskManager.Task::(_updateState) Task=`77ec933e-700e-4098-ae85-3b5165c1af4c`::moving from state preparing -> state finished >Thread-6291::DEBUG::2012-12-02 23:12:35,213::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6291::DEBUG::2012-12-02 23:12:35,213::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6291::DEBUG::2012-12-02 23:12:35,213::task::957::TaskManager.Task::(_decref) Task=`77ec933e-700e-4098-ae85-3b5165c1af4c`::ref 0 aborting False >Thread-6296::DEBUG::2012-12-02 23:12:43,311::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6296::DEBUG::2012-12-02 23:12:43,311::task::568::TaskManager.Task::(_updateState) Task=`5adcdc4b-1b5c-4215-8453-60a3091d7dbc`::moving from state init -> state preparing >Thread-6296::INFO::2012-12-02 23:12:43,311::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6296::INFO::2012-12-02 23:12:43,311::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6296::DEBUG::2012-12-02 23:12:43,312::task::1151::TaskManager.Task::(prepare) Task=`5adcdc4b-1b5c-4215-8453-60a3091d7dbc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6296::DEBUG::2012-12-02 23:12:43,312::task::568::TaskManager.Task::(_updateState) Task=`5adcdc4b-1b5c-4215-8453-60a3091d7dbc`::moving from state preparing -> state finished >Thread-6296::DEBUG::2012-12-02 23:12:43,312::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6296::DEBUG::2012-12-02 23:12:43,312::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6296::DEBUG::2012-12-02 23:12:43,312::task::957::TaskManager.Task::(_decref) Task=`5adcdc4b-1b5c-4215-8453-60a3091d7dbc`::ref 0 aborting False >Thread-6297::DEBUG::2012-12-02 23:12:43,449::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6297::DEBUG::2012-12-02 23:12:43,449::task::568::TaskManager.Task::(_updateState) Task=`7fd31180-f1d2-494b-b218-ed1940f96925`::moving from state init -> state preparing >Thread-6297::INFO::2012-12-02 23:12:43,449::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6297::DEBUG::2012-12-02 23:12:43,449::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b3fc86af-f70c-484d-9268-b46001c44d46`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6297::DEBUG::2012-12-02 23:12:43,450::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6297::DEBUG::2012-12-02 23:12:43,450::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6297::DEBUG::2012-12-02 23:12:43,450::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b3fc86af-f70c-484d-9268-b46001c44d46`::Granted request >Thread-6297::DEBUG::2012-12-02 23:12:43,450::task::794::TaskManager.Task::(resourceAcquired) Task=`7fd31180-f1d2-494b-b218-ed1940f96925`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6297::DEBUG::2012-12-02 23:12:43,450::task::957::TaskManager.Task::(_decref) Task=`7fd31180-f1d2-494b-b218-ed1940f96925`::ref 1 aborting False >Thread-6297::DEBUG::2012-12-02 23:12:43,456::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6297::DEBUG::2012-12-02 23:12:43,457::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6297::DEBUG::2012-12-02 23:12:43,662::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6297::WARNING::2012-12-02 23:12:43,664::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6297::DEBUG::2012-12-02 23:12:43,664::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6297::WARNING::2012-12-02 23:12:43,693::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6297::INFO::2012-12-02 23:12:43,693::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6297::DEBUG::2012-12-02 23:12:43,693::task::1151::TaskManager.Task::(prepare) Task=`7fd31180-f1d2-494b-b218-ed1940f96925`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6297::DEBUG::2012-12-02 23:12:43,693::task::568::TaskManager.Task::(_updateState) Task=`7fd31180-f1d2-494b-b218-ed1940f96925`::moving from state preparing -> state finished >Thread-6297::DEBUG::2012-12-02 23:12:43,694::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6297::DEBUG::2012-12-02 23:12:43,694::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6297::DEBUG::2012-12-02 23:12:43,694::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6297::DEBUG::2012-12-02 23:12:43,694::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6297::DEBUG::2012-12-02 23:12:43,694::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6297::DEBUG::2012-12-02 23:12:43,694::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6297::DEBUG::2012-12-02 23:12:43,695::task::957::TaskManager.Task::(_decref) Task=`7fd31180-f1d2-494b-b218-ed1940f96925`::ref 0 aborting False >Thread-6360::DEBUG::2012-12-02 23:12:46,123::task::568::TaskManager.Task::(_updateState) Task=`52441bdc-7a7f-49e7-90dd-acde39730491`::moving from state init -> state preparing >Thread-6360::INFO::2012-12-02 23:12:46,123::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6360::INFO::2012-12-02 23:12:46,123::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00780391693115', 'lastCheck': 1354482756.572479, 'code': 0, 'valid': True}} >Thread-6360::DEBUG::2012-12-02 23:12:46,124::task::1151::TaskManager.Task::(prepare) Task=`52441bdc-7a7f-49e7-90dd-acde39730491`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00780391693115', 'lastCheck': 1354482756.572479, 'code': 0, 'valid': True}} >Thread-6360::DEBUG::2012-12-02 23:12:46,124::task::568::TaskManager.Task::(_updateState) Task=`52441bdc-7a7f-49e7-90dd-acde39730491`::moving from state preparing -> state finished >Thread-6360::DEBUG::2012-12-02 23:12:46,124::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6360::DEBUG::2012-12-02 23:12:46,124::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6360::DEBUG::2012-12-02 23:12:46,124::task::957::TaskManager.Task::(_decref) Task=`52441bdc-7a7f-49e7-90dd-acde39730491`::ref 0 aborting False >Thread-6365::DEBUG::2012-12-02 23:12:53,912::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6365::DEBUG::2012-12-02 23:12:53,912::task::568::TaskManager.Task::(_updateState) Task=`5c205873-3d06-49cb-aaa8-4f4e1fd8c095`::moving from state init -> state preparing >Thread-6365::INFO::2012-12-02 23:12:53,912::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6365::INFO::2012-12-02 23:12:53,912::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6365::DEBUG::2012-12-02 23:12:53,913::task::1151::TaskManager.Task::(prepare) Task=`5c205873-3d06-49cb-aaa8-4f4e1fd8c095`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6365::DEBUG::2012-12-02 23:12:53,913::task::568::TaskManager.Task::(_updateState) Task=`5c205873-3d06-49cb-aaa8-4f4e1fd8c095`::moving from state preparing -> state finished >Thread-6365::DEBUG::2012-12-02 23:12:53,913::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6365::DEBUG::2012-12-02 23:12:53,913::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6365::DEBUG::2012-12-02 23:12:53,913::task::957::TaskManager.Task::(_decref) Task=`5c205873-3d06-49cb-aaa8-4f4e1fd8c095`::ref 0 aborting False >Thread-6366::DEBUG::2012-12-02 23:12:54,058::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6366::DEBUG::2012-12-02 23:12:54,058::task::568::TaskManager.Task::(_updateState) Task=`d8fb3c3c-066b-415e-9a7b-fd705b12bd9a`::moving from state init -> state preparing >Thread-6366::INFO::2012-12-02 23:12:54,058::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6366::DEBUG::2012-12-02 23:12:54,058::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ad20f828-c53f-4b67-a6c2-ac029b76cd50`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6366::DEBUG::2012-12-02 23:12:54,059::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6366::DEBUG::2012-12-02 23:12:54,059::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6366::DEBUG::2012-12-02 23:12:54,059::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ad20f828-c53f-4b67-a6c2-ac029b76cd50`::Granted request >Thread-6366::DEBUG::2012-12-02 23:12:54,059::task::794::TaskManager.Task::(resourceAcquired) Task=`d8fb3c3c-066b-415e-9a7b-fd705b12bd9a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6366::DEBUG::2012-12-02 23:12:54,059::task::957::TaskManager.Task::(_decref) Task=`d8fb3c3c-066b-415e-9a7b-fd705b12bd9a`::ref 1 aborting False >Thread-6366::DEBUG::2012-12-02 23:12:54,066::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6366::DEBUG::2012-12-02 23:12:54,066::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6366::DEBUG::2012-12-02 23:12:54,273::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6366::WARNING::2012-12-02 23:12:54,274::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6366::DEBUG::2012-12-02 23:12:54,275::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6366::WARNING::2012-12-02 23:12:54,303::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6366::INFO::2012-12-02 23:12:54,304::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6366::DEBUG::2012-12-02 23:12:54,304::task::1151::TaskManager.Task::(prepare) Task=`d8fb3c3c-066b-415e-9a7b-fd705b12bd9a`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6366::DEBUG::2012-12-02 23:12:54,304::task::568::TaskManager.Task::(_updateState) Task=`d8fb3c3c-066b-415e-9a7b-fd705b12bd9a`::moving from state preparing -> state finished >Thread-6366::DEBUG::2012-12-02 23:12:54,304::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6366::DEBUG::2012-12-02 23:12:54,304::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6366::DEBUG::2012-12-02 23:12:54,304::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6366::DEBUG::2012-12-02 23:12:54,305::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6366::DEBUG::2012-12-02 23:12:54,305::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6366::DEBUG::2012-12-02 23:12:54,305::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6366::DEBUG::2012-12-02 23:12:54,305::task::957::TaskManager.Task::(_decref) Task=`d8fb3c3c-066b-415e-9a7b-fd705b12bd9a`::ref 0 aborting False >Thread-6429::DEBUG::2012-12-02 23:12:57,091::task::568::TaskManager.Task::(_updateState) Task=`ddc46747-9130-4803-aefe-949ab9b3512d`::moving from state init -> state preparing >Thread-6429::INFO::2012-12-02 23:12:57,091::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6429::INFO::2012-12-02 23:12:57,091::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00847792625427', 'lastCheck': 1354482776.6122, 'code': 0, 'valid': True}} >Thread-6429::DEBUG::2012-12-02 23:12:57,091::task::1151::TaskManager.Task::(prepare) Task=`ddc46747-9130-4803-aefe-949ab9b3512d`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00847792625427', 'lastCheck': 1354482776.6122, 'code': 0, 'valid': True}} >Thread-6429::DEBUG::2012-12-02 23:12:57,092::task::568::TaskManager.Task::(_updateState) Task=`ddc46747-9130-4803-aefe-949ab9b3512d`::moving from state preparing -> state finished >Thread-6429::DEBUG::2012-12-02 23:12:57,092::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6429::DEBUG::2012-12-02 23:12:57,092::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6429::DEBUG::2012-12-02 23:12:57,092::task::957::TaskManager.Task::(_decref) Task=`ddc46747-9130-4803-aefe-949ab9b3512d`::ref 0 aborting False >Thread-6434::DEBUG::2012-12-02 23:13:04,511::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6434::DEBUG::2012-12-02 23:13:04,511::task::568::TaskManager.Task::(_updateState) Task=`5f6e5572-4a51-4a43-b42e-59bc7f39d2b3`::moving from state init -> state preparing >Thread-6434::INFO::2012-12-02 23:13:04,512::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6434::INFO::2012-12-02 23:13:04,512::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6434::DEBUG::2012-12-02 23:13:04,512::task::1151::TaskManager.Task::(prepare) Task=`5f6e5572-4a51-4a43-b42e-59bc7f39d2b3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6434::DEBUG::2012-12-02 23:13:04,512::task::568::TaskManager.Task::(_updateState) Task=`5f6e5572-4a51-4a43-b42e-59bc7f39d2b3`::moving from state preparing -> state finished >Thread-6434::DEBUG::2012-12-02 23:13:04,512::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6434::DEBUG::2012-12-02 23:13:04,512::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6434::DEBUG::2012-12-02 23:13:04,513::task::957::TaskManager.Task::(_decref) Task=`5f6e5572-4a51-4a43-b42e-59bc7f39d2b3`::ref 0 aborting False >Thread-6435::DEBUG::2012-12-02 23:13:04,674::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6435::DEBUG::2012-12-02 23:13:04,674::task::568::TaskManager.Task::(_updateState) Task=`6b25db0f-7142-4ace-8dbf-c828dc5604a9`::moving from state init -> state preparing >Thread-6435::INFO::2012-12-02 23:13:04,674::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6435::DEBUG::2012-12-02 23:13:04,675::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c1009caa-c348-4537-b67b-40e74bda94a5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6435::DEBUG::2012-12-02 23:13:04,675::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6435::DEBUG::2012-12-02 23:13:04,675::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6435::DEBUG::2012-12-02 23:13:04,675::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c1009caa-c348-4537-b67b-40e74bda94a5`::Granted request >Thread-6435::DEBUG::2012-12-02 23:13:04,675::task::794::TaskManager.Task::(resourceAcquired) Task=`6b25db0f-7142-4ace-8dbf-c828dc5604a9`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6435::DEBUG::2012-12-02 23:13:04,676::task::957::TaskManager.Task::(_decref) Task=`6b25db0f-7142-4ace-8dbf-c828dc5604a9`::ref 1 aborting False >Thread-6435::DEBUG::2012-12-02 23:13:04,680::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6435::DEBUG::2012-12-02 23:13:04,680::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6435::DEBUG::2012-12-02 23:13:04,888::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6435::WARNING::2012-12-02 23:13:04,889::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6435::DEBUG::2012-12-02 23:13:04,890::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6435::WARNING::2012-12-02 23:13:04,918::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6435::INFO::2012-12-02 23:13:04,918::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6435::DEBUG::2012-12-02 23:13:04,918::task::1151::TaskManager.Task::(prepare) Task=`6b25db0f-7142-4ace-8dbf-c828dc5604a9`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6435::DEBUG::2012-12-02 23:13:04,918::task::568::TaskManager.Task::(_updateState) Task=`6b25db0f-7142-4ace-8dbf-c828dc5604a9`::moving from state preparing -> state finished >Thread-6435::DEBUG::2012-12-02 23:13:04,918::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6435::DEBUG::2012-12-02 23:13:04,919::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6435::DEBUG::2012-12-02 23:13:04,919::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6435::DEBUG::2012-12-02 23:13:04,919::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6435::DEBUG::2012-12-02 23:13:04,919::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6435::DEBUG::2012-12-02 23:13:04,919::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6435::DEBUG::2012-12-02 23:13:04,919::task::957::TaskManager.Task::(_decref) Task=`6b25db0f-7142-4ace-8dbf-c828dc5604a9`::ref 0 aborting False >Thread-6498::DEBUG::2012-12-02 23:13:08,075::task::568::TaskManager.Task::(_updateState) Task=`4d08786b-c708-4917-9295-f386a6c96208`::moving from state init -> state preparing >Thread-6498::INFO::2012-12-02 23:13:08,075::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6498::INFO::2012-12-02 23:13:08,076::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00731015205383', 'lastCheck': 1354482786.6306779, 'code': 0, 'valid': True}} >Thread-6498::DEBUG::2012-12-02 23:13:08,076::task::1151::TaskManager.Task::(prepare) Task=`4d08786b-c708-4917-9295-f386a6c96208`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00731015205383', 'lastCheck': 1354482786.6306779, 'code': 0, 'valid': True}} >Thread-6498::DEBUG::2012-12-02 23:13:08,076::task::568::TaskManager.Task::(_updateState) Task=`4d08786b-c708-4917-9295-f386a6c96208`::moving from state preparing -> state finished >Thread-6498::DEBUG::2012-12-02 23:13:08,076::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6498::DEBUG::2012-12-02 23:13:08,076::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6498::DEBUG::2012-12-02 23:13:08,076::task::957::TaskManager.Task::(_decref) Task=`4d08786b-c708-4917-9295-f386a6c96208`::ref 0 aborting False >Thread-5152::DEBUG::2012-12-02 23:15:56,915::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:15:56,916::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5152::DEBUG::2012-12-02 23:15:56,916::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5152::DEBUG::2012-12-02 23:15:57,125::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5152::WARNING::2012-12-02 23:15:57,126::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5152::DEBUG::2012-12-02 23:15:57,127::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5152::DEBUG::2012-12-02 23:15:57,162::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:15:57,162::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-5152::DEBUG::2012-12-02 23:15:57,175::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-5152::DEBUG::2012-12-02 23:15:57,178::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-5152::WARNING::2012-12-02 23:15:57,178::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-5152::WARNING::2012-12-02 23:15:57,178::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-5152::DEBUG::2012-12-02 23:20:57,727::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:20:57,727::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5152::DEBUG::2012-12-02 23:20:57,727::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5152::DEBUG::2012-12-02 23:20:57,937::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5152::WARNING::2012-12-02 23:20:57,939::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5152::DEBUG::2012-12-02 23:20:57,939::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6600::WARNING::2012-12-02 23:20:57,956::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_iso2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6602::WARNING::2012-12-02 23:20:57,958::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6603::WARNING::2012-12-02 23:20:57,958::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6604::WARNING::2012-12-02 23:20:57,959::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6605::WARNING::2012-12-02 23:20:57,959::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6606::WARNING::2012-12-02 23:20:57,961::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6607::WARNING::2012-12-02 23:20:57,961::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6608::WARNING::2012-12-02 23:20:57,962::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6609::WARNING::2012-12-02 23:20:57,963::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6611::WARNING::2012-12-02 23:20:57,964::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6619::WARNING::2012-12-02 23:20:57,967::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6621::WARNING::2012-12-02 23:20:57,969::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-5152::DEBUG::2012-12-02 23:20:57,976::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:20:57,976::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-5152::DEBUG::2012-12-02 23:20:58,002::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-5152::DEBUG::2012-12-02 23:20:58,004::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-5152::WARNING::2012-12-02 23:20:58,004::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-5152::WARNING::2012-12-02 23:20:58,005::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-5152::DEBUG::2012-12-02 23:25:58,616::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:25:58,616::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5152::DEBUG::2012-12-02 23:25:58,617::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5152::DEBUG::2012-12-02 23:25:58,829::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5152::WARNING::2012-12-02 23:25:58,831::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5152::DEBUG::2012-12-02 23:25:58,831::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6683::WARNING::2012-12-02 23:25:58,859::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-5152::DEBUG::2012-12-02 23:25:58,867::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:25:58,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-5152::DEBUG::2012-12-02 23:25:58,877::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-5152::DEBUG::2012-12-02 23:25:58,880::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-5152::WARNING::2012-12-02 23:25:58,880::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-5152::WARNING::2012-12-02 23:25:58,880::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-6685::DEBUG::2012-12-02 23:28:27,088::task::568::TaskManager.Task::(_updateState) Task=`e7727fb0-dcf0-4ed7-9b44-ca120de59654`::moving from state init -> state preparing >Thread-6685::INFO::2012-12-02 23:28:27,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6685::INFO::2012-12-02 23:28:27,088::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00799012184143', 'lastCheck': 1354483699.168988, 'code': 0, 'valid': True}} >Thread-6685::DEBUG::2012-12-02 23:28:27,088::task::1151::TaskManager.Task::(prepare) Task=`e7727fb0-dcf0-4ed7-9b44-ca120de59654`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00799012184143', 'lastCheck': 1354483699.168988, 'code': 0, 'valid': True}} >Thread-6685::DEBUG::2012-12-02 23:28:27,089::task::568::TaskManager.Task::(_updateState) Task=`e7727fb0-dcf0-4ed7-9b44-ca120de59654`::moving from state preparing -> state finished >Thread-6685::DEBUG::2012-12-02 23:28:27,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6685::DEBUG::2012-12-02 23:28:27,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6685::DEBUG::2012-12-02 23:28:27,089::task::957::TaskManager.Task::(_decref) Task=`e7727fb0-dcf0-4ed7-9b44-ca120de59654`::ref 0 aborting False >Thread-6686::DEBUG::2012-12-02 23:28:27,294::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-6686::DEBUG::2012-12-02 23:28:27,305::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-6686::DEBUG::2012-12-02 23:28:27,317::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-6686::DEBUG::2012-12-02 23:28:27,317::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-6686::DEBUG::2012-12-02 23:28:27,327::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-6686::DEBUG::2012-12-02 23:28:27,327::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-6686::DEBUG::2012-12-02 23:28:27,339::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-6686::DEBUG::2012-12-02 23:28:27,340::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-6686::DEBUG::2012-12-02 23:28:27,352::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-6686::DEBUG::2012-12-02 23:28:27,353::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-6686::DEBUG::2012-12-02 23:28:27,366::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-6686::DEBUG::2012-12-02 23:28:27,367::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.202.48'}} >Thread-6691::DEBUG::2012-12-02 23:28:35,155::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6691::DEBUG::2012-12-02 23:28:35,155::task::568::TaskManager.Task::(_updateState) Task=`a254f513-dc96-4cec-a0e8-00e888326830`::moving from state init -> state preparing >Thread-6691::INFO::2012-12-02 23:28:35,156::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6691::INFO::2012-12-02 23:28:35,156::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6691::DEBUG::2012-12-02 23:28:35,156::task::1151::TaskManager.Task::(prepare) Task=`a254f513-dc96-4cec-a0e8-00e888326830`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6691::DEBUG::2012-12-02 23:28:35,156::task::568::TaskManager.Task::(_updateState) Task=`a254f513-dc96-4cec-a0e8-00e888326830`::moving from state preparing -> state finished >Thread-6691::DEBUG::2012-12-02 23:28:35,156::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6691::DEBUG::2012-12-02 23:28:35,156::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6691::DEBUG::2012-12-02 23:28:35,157::task::957::TaskManager.Task::(_decref) Task=`a254f513-dc96-4cec-a0e8-00e888326830`::ref 0 aborting False >Thread-6692::DEBUG::2012-12-02 23:28:35,340::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6692::DEBUG::2012-12-02 23:28:35,340::task::568::TaskManager.Task::(_updateState) Task=`b8e8fd30-7e58-4262-ae3d-9a1ab4087b7c`::moving from state init -> state preparing >Thread-6692::INFO::2012-12-02 23:28:35,340::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6692::DEBUG::2012-12-02 23:28:35,340::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f908b906-3867-4cc6-a1a8-61d2fe508782`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6692::DEBUG::2012-12-02 23:28:35,341::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6692::DEBUG::2012-12-02 23:28:35,341::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6692::DEBUG::2012-12-02 23:28:35,341::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f908b906-3867-4cc6-a1a8-61d2fe508782`::Granted request >Thread-6692::DEBUG::2012-12-02 23:28:35,341::task::794::TaskManager.Task::(resourceAcquired) Task=`b8e8fd30-7e58-4262-ae3d-9a1ab4087b7c`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6692::DEBUG::2012-12-02 23:28:35,341::task::957::TaskManager.Task::(_decref) Task=`b8e8fd30-7e58-4262-ae3d-9a1ab4087b7c`::ref 1 aborting False >Thread-6692::DEBUG::2012-12-02 23:28:35,347::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6692::DEBUG::2012-12-02 23:28:35,347::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6692::DEBUG::2012-12-02 23:28:35,554::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6692::WARNING::2012-12-02 23:28:35,555::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6692::DEBUG::2012-12-02 23:28:35,556::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6692::WARNING::2012-12-02 23:28:35,590::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6692::INFO::2012-12-02 23:28:35,590::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6692::DEBUG::2012-12-02 23:28:35,591::task::1151::TaskManager.Task::(prepare) Task=`b8e8fd30-7e58-4262-ae3d-9a1ab4087b7c`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6692::DEBUG::2012-12-02 23:28:35,591::task::568::TaskManager.Task::(_updateState) Task=`b8e8fd30-7e58-4262-ae3d-9a1ab4087b7c`::moving from state preparing -> state finished >Thread-6692::DEBUG::2012-12-02 23:28:35,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6692::DEBUG::2012-12-02 23:28:35,591::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6692::DEBUG::2012-12-02 23:28:35,591::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6692::DEBUG::2012-12-02 23:28:35,591::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6692::DEBUG::2012-12-02 23:28:35,592::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6692::DEBUG::2012-12-02 23:28:35,592::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6692::DEBUG::2012-12-02 23:28:35,592::task::957::TaskManager.Task::(_decref) Task=`b8e8fd30-7e58-4262-ae3d-9a1ab4087b7c`::ref 0 aborting False >Thread-6754::DEBUG::2012-12-02 23:28:35,845::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6754::DEBUG::2012-12-02 23:28:35,845::task::568::TaskManager.Task::(_updateState) Task=`86611e74-e84f-4f31-bc24-9b58657a60c3`::moving from state init -> state preparing >Thread-6754::INFO::2012-12-02 23:28:35,845::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-6754::DEBUG::2012-12-02 23:28:35,845::taskManager::177::TaskManager::(getAllTasksInfo) Entry. >Thread-6754::DEBUG::2012-12-02 23:28:35,846::taskManager::188::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-6754::INFO::2012-12-02 23:28:35,846::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-6754::DEBUG::2012-12-02 23:28:35,846::task::1151::TaskManager.Task::(prepare) Task=`86611e74-e84f-4f31-bc24-9b58657a60c3`::finished: {'allTasksInfo': {}} >Thread-6754::DEBUG::2012-12-02 23:28:35,846::task::568::TaskManager.Task::(_updateState) Task=`86611e74-e84f-4f31-bc24-9b58657a60c3`::moving from state preparing -> state finished >Thread-6754::DEBUG::2012-12-02 23:28:35,846::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6754::DEBUG::2012-12-02 23:28:35,846::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6754::DEBUG::2012-12-02 23:28:35,846::task::957::TaskManager.Task::(_decref) Task=`86611e74-e84f-4f31-bc24-9b58657a60c3`::ref 0 aborting False >Thread-6756::DEBUG::2012-12-02 23:28:39,131::task::568::TaskManager.Task::(_updateState) Task=`caf1b3b1-1b1a-4581-a394-5ec87f5e2d46`::moving from state init -> state preparing >Thread-6756::INFO::2012-12-02 23:28:39,132::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6756::INFO::2012-12-02 23:28:39,132::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0083429813385', 'lastCheck': 1354483709.189779, 'code': 0, 'valid': True}} >Thread-6756::DEBUG::2012-12-02 23:28:39,132::task::1151::TaskManager.Task::(prepare) Task=`caf1b3b1-1b1a-4581-a394-5ec87f5e2d46`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0083429813385', 'lastCheck': 1354483709.189779, 'code': 0, 'valid': True}} >Thread-6756::DEBUG::2012-12-02 23:28:39,132::task::568::TaskManager.Task::(_updateState) Task=`caf1b3b1-1b1a-4581-a394-5ec87f5e2d46`::moving from state preparing -> state finished >Thread-6756::DEBUG::2012-12-02 23:28:39,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6756::DEBUG::2012-12-02 23:28:39,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6756::DEBUG::2012-12-02 23:28:39,132::task::957::TaskManager.Task::(_decref) Task=`caf1b3b1-1b1a-4581-a394-5ec87f5e2d46`::ref 0 aborting False >Thread-6761::DEBUG::2012-12-02 23:28:45,838::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6761::DEBUG::2012-12-02 23:28:45,839::task::568::TaskManager.Task::(_updateState) Task=`e00e4cc7-49a5-47ef-90da-351b118a0d1d`::moving from state init -> state preparing >Thread-6761::INFO::2012-12-02 23:28:45,839::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6761::INFO::2012-12-02 23:28:45,839::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6761::DEBUG::2012-12-02 23:28:45,839::task::1151::TaskManager.Task::(prepare) Task=`e00e4cc7-49a5-47ef-90da-351b118a0d1d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6761::DEBUG::2012-12-02 23:28:45,839::task::568::TaskManager.Task::(_updateState) Task=`e00e4cc7-49a5-47ef-90da-351b118a0d1d`::moving from state preparing -> state finished >Thread-6761::DEBUG::2012-12-02 23:28:45,840::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6761::DEBUG::2012-12-02 23:28:45,840::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6761::DEBUG::2012-12-02 23:28:45,840::task::957::TaskManager.Task::(_decref) Task=`e00e4cc7-49a5-47ef-90da-351b118a0d1d`::ref 0 aborting False >Thread-6762::DEBUG::2012-12-02 23:28:46,000::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6762::DEBUG::2012-12-02 23:28:46,001::task::568::TaskManager.Task::(_updateState) Task=`32948bd7-b159-4556-80db-d36d27e93f6d`::moving from state init -> state preparing >Thread-6762::INFO::2012-12-02 23:28:46,001::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6762::DEBUG::2012-12-02 23:28:46,001::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a48cf0fa-3cb5-4d10-bdd8-74060e14acb6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6762::DEBUG::2012-12-02 23:28:46,001::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6762::DEBUG::2012-12-02 23:28:46,002::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6762::DEBUG::2012-12-02 23:28:46,002::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a48cf0fa-3cb5-4d10-bdd8-74060e14acb6`::Granted request >Thread-6762::DEBUG::2012-12-02 23:28:46,002::task::794::TaskManager.Task::(resourceAcquired) Task=`32948bd7-b159-4556-80db-d36d27e93f6d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6762::DEBUG::2012-12-02 23:28:46,002::task::957::TaskManager.Task::(_decref) Task=`32948bd7-b159-4556-80db-d36d27e93f6d`::ref 1 aborting False >Thread-6762::DEBUG::2012-12-02 23:28:46,007::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6762::DEBUG::2012-12-02 23:28:46,008::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6762::DEBUG::2012-12-02 23:28:46,216::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6762::WARNING::2012-12-02 23:28:46,218::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6762::DEBUG::2012-12-02 23:28:46,218::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6762::WARNING::2012-12-02 23:28:46,247::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6762::INFO::2012-12-02 23:28:46,247::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6762::DEBUG::2012-12-02 23:28:46,248::task::1151::TaskManager.Task::(prepare) Task=`32948bd7-b159-4556-80db-d36d27e93f6d`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6762::DEBUG::2012-12-02 23:28:46,248::task::568::TaskManager.Task::(_updateState) Task=`32948bd7-b159-4556-80db-d36d27e93f6d`::moving from state preparing -> state finished >Thread-6762::DEBUG::2012-12-02 23:28:46,248::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6762::DEBUG::2012-12-02 23:28:46,248::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6762::DEBUG::2012-12-02 23:28:46,248::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6762::DEBUG::2012-12-02 23:28:46,248::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6762::DEBUG::2012-12-02 23:28:46,249::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6762::DEBUG::2012-12-02 23:28:46,249::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6762::DEBUG::2012-12-02 23:28:46,249::task::957::TaskManager.Task::(_decref) Task=`32948bd7-b159-4556-80db-d36d27e93f6d`::ref 0 aborting False >Thread-6825::DEBUG::2012-12-02 23:28:50,123::task::568::TaskManager.Task::(_updateState) Task=`a9c0384f-9c2d-4282-962f-d54e9b082fdd`::moving from state init -> state preparing >Thread-6825::INFO::2012-12-02 23:28:50,123::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6825::INFO::2012-12-02 23:28:50,123::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00976085662842', 'lastCheck': 1354483729.2290249, 'code': 0, 'valid': True}} >Thread-6825::DEBUG::2012-12-02 23:28:50,123::task::1151::TaskManager.Task::(prepare) Task=`a9c0384f-9c2d-4282-962f-d54e9b082fdd`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00976085662842', 'lastCheck': 1354483729.2290249, 'code': 0, 'valid': True}} >Thread-6825::DEBUG::2012-12-02 23:28:50,124::task::568::TaskManager.Task::(_updateState) Task=`a9c0384f-9c2d-4282-962f-d54e9b082fdd`::moving from state preparing -> state finished >Thread-6825::DEBUG::2012-12-02 23:28:50,124::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6825::DEBUG::2012-12-02 23:28:50,124::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6825::DEBUG::2012-12-02 23:28:50,124::task::957::TaskManager.Task::(_decref) Task=`a9c0384f-9c2d-4282-962f-d54e9b082fdd`::ref 0 aborting False >Thread-6829::DEBUG::2012-12-02 23:28:56,481::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6829::DEBUG::2012-12-02 23:28:56,482::task::568::TaskManager.Task::(_updateState) Task=`53eed73b-5462-478d-a84b-f6333f4adffe`::moving from state init -> state preparing >Thread-6829::INFO::2012-12-02 23:28:56,482::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6829::INFO::2012-12-02 23:28:56,482::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6829::DEBUG::2012-12-02 23:28:56,482::task::1151::TaskManager.Task::(prepare) Task=`53eed73b-5462-478d-a84b-f6333f4adffe`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6829::DEBUG::2012-12-02 23:28:56,482::task::568::TaskManager.Task::(_updateState) Task=`53eed73b-5462-478d-a84b-f6333f4adffe`::moving from state preparing -> state finished >Thread-6829::DEBUG::2012-12-02 23:28:56,482::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6829::DEBUG::2012-12-02 23:28:56,483::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6829::DEBUG::2012-12-02 23:28:56,483::task::957::TaskManager.Task::(_decref) Task=`53eed73b-5462-478d-a84b-f6333f4adffe`::ref 0 aborting False >Thread-6830::DEBUG::2012-12-02 23:28:56,631::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6830::DEBUG::2012-12-02 23:28:56,632::task::568::TaskManager.Task::(_updateState) Task=`6d6f053a-9729-4459-a263-cf8f130273c4`::moving from state init -> state preparing >Thread-6830::INFO::2012-12-02 23:28:56,632::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6830::DEBUG::2012-12-02 23:28:56,632::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ee82a914-08e0-4a00-aa47-38daa82d7c95`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6830::DEBUG::2012-12-02 23:28:56,632::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6830::DEBUG::2012-12-02 23:28:56,632::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6830::DEBUG::2012-12-02 23:28:56,632::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ee82a914-08e0-4a00-aa47-38daa82d7c95`::Granted request >Thread-6830::DEBUG::2012-12-02 23:28:56,633::task::794::TaskManager.Task::(resourceAcquired) Task=`6d6f053a-9729-4459-a263-cf8f130273c4`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6830::DEBUG::2012-12-02 23:28:56,633::task::957::TaskManager.Task::(_decref) Task=`6d6f053a-9729-4459-a263-cf8f130273c4`::ref 1 aborting False >Thread-6830::DEBUG::2012-12-02 23:28:56,638::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6830::DEBUG::2012-12-02 23:28:56,638::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6830::DEBUG::2012-12-02 23:28:56,845::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6830::WARNING::2012-12-02 23:28:56,847::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6830::DEBUG::2012-12-02 23:28:56,847::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6830::WARNING::2012-12-02 23:28:56,877::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6830::INFO::2012-12-02 23:28:56,877::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6830::DEBUG::2012-12-02 23:28:56,877::task::1151::TaskManager.Task::(prepare) Task=`6d6f053a-9729-4459-a263-cf8f130273c4`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6830::DEBUG::2012-12-02 23:28:56,877::task::568::TaskManager.Task::(_updateState) Task=`6d6f053a-9729-4459-a263-cf8f130273c4`::moving from state preparing -> state finished >Thread-6830::DEBUG::2012-12-02 23:28:56,878::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6830::DEBUG::2012-12-02 23:28:56,878::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6830::DEBUG::2012-12-02 23:28:56,878::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6830::DEBUG::2012-12-02 23:28:56,878::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6830::DEBUG::2012-12-02 23:28:56,878::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6830::DEBUG::2012-12-02 23:28:56,878::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6830::DEBUG::2012-12-02 23:28:56,879::task::957::TaskManager.Task::(_decref) Task=`6d6f053a-9729-4459-a263-cf8f130273c4`::ref 0 aborting False >Thread-6894::DEBUG::2012-12-02 23:29:02,176::task::568::TaskManager.Task::(_updateState) Task=`6cf9e2da-0fb8-4938-8ac1-e0f473e15fd1`::moving from state init -> state preparing >Thread-6894::INFO::2012-12-02 23:29:02,177::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6894::INFO::2012-12-02 23:29:02,177::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.010890007019', 'lastCheck': 1354483739.2542329, 'code': 0, 'valid': True}} >Thread-6894::DEBUG::2012-12-02 23:29:02,177::task::1151::TaskManager.Task::(prepare) Task=`6cf9e2da-0fb8-4938-8ac1-e0f473e15fd1`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.010890007019', 'lastCheck': 1354483739.2542329, 'code': 0, 'valid': True}} >Thread-6894::DEBUG::2012-12-02 23:29:02,177::task::568::TaskManager.Task::(_updateState) Task=`6cf9e2da-0fb8-4938-8ac1-e0f473e15fd1`::moving from state preparing -> state finished >Thread-6894::DEBUG::2012-12-02 23:29:02,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6894::DEBUG::2012-12-02 23:29:02,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6894::DEBUG::2012-12-02 23:29:02,177::task::957::TaskManager.Task::(_decref) Task=`6cf9e2da-0fb8-4938-8ac1-e0f473e15fd1`::ref 0 aborting False >Thread-6898::DEBUG::2012-12-02 23:29:07,241::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6898::DEBUG::2012-12-02 23:29:07,242::task::568::TaskManager.Task::(_updateState) Task=`468c795b-5c79-48ee-9dce-0b17f927bb19`::moving from state init -> state preparing >Thread-6898::INFO::2012-12-02 23:29:07,242::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6898::INFO::2012-12-02 23:29:07,242::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6898::DEBUG::2012-12-02 23:29:07,242::task::1151::TaskManager.Task::(prepare) Task=`468c795b-5c79-48ee-9dce-0b17f927bb19`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6898::DEBUG::2012-12-02 23:29:07,242::task::568::TaskManager.Task::(_updateState) Task=`468c795b-5c79-48ee-9dce-0b17f927bb19`::moving from state preparing -> state finished >Thread-6898::DEBUG::2012-12-02 23:29:07,242::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6898::DEBUG::2012-12-02 23:29:07,243::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6898::DEBUG::2012-12-02 23:29:07,243::task::957::TaskManager.Task::(_decref) Task=`468c795b-5c79-48ee-9dce-0b17f927bb19`::ref 0 aborting False >Thread-6899::DEBUG::2012-12-02 23:29:07,388::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6899::DEBUG::2012-12-02 23:29:07,388::task::568::TaskManager.Task::(_updateState) Task=`948ae8ea-4290-4b93-84fa-f8cf62baeb91`::moving from state init -> state preparing >Thread-6899::INFO::2012-12-02 23:29:07,389::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6899::DEBUG::2012-12-02 23:29:07,389::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4daa06d9-9943-48e5-a461-2a49d873f959`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6899::DEBUG::2012-12-02 23:29:07,389::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6899::DEBUG::2012-12-02 23:29:07,389::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6899::DEBUG::2012-12-02 23:29:07,389::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4daa06d9-9943-48e5-a461-2a49d873f959`::Granted request >Thread-6899::DEBUG::2012-12-02 23:29:07,390::task::794::TaskManager.Task::(resourceAcquired) Task=`948ae8ea-4290-4b93-84fa-f8cf62baeb91`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6899::DEBUG::2012-12-02 23:29:07,390::task::957::TaskManager.Task::(_decref) Task=`948ae8ea-4290-4b93-84fa-f8cf62baeb91`::ref 1 aborting False >Thread-6899::DEBUG::2012-12-02 23:29:07,396::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6899::DEBUG::2012-12-02 23:29:07,396::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6899::DEBUG::2012-12-02 23:29:07,601::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6899::WARNING::2012-12-02 23:29:07,602::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6899::DEBUG::2012-12-02 23:29:07,603::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6899::WARNING::2012-12-02 23:29:07,631::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6899::INFO::2012-12-02 23:29:07,631::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6899::DEBUG::2012-12-02 23:29:07,631::task::1151::TaskManager.Task::(prepare) Task=`948ae8ea-4290-4b93-84fa-f8cf62baeb91`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6899::DEBUG::2012-12-02 23:29:07,632::task::568::TaskManager.Task::(_updateState) Task=`948ae8ea-4290-4b93-84fa-f8cf62baeb91`::moving from state preparing -> state finished >Thread-6899::DEBUG::2012-12-02 23:29:07,632::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6899::DEBUG::2012-12-02 23:29:07,632::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6899::DEBUG::2012-12-02 23:29:07,632::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6899::DEBUG::2012-12-02 23:29:07,632::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6899::DEBUG::2012-12-02 23:29:07,632::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6899::DEBUG::2012-12-02 23:29:07,633::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6899::DEBUG::2012-12-02 23:29:07,633::task::957::TaskManager.Task::(_decref) Task=`948ae8ea-4290-4b93-84fa-f8cf62baeb91`::ref 0 aborting False >Thread-6963::DEBUG::2012-12-02 23:29:13,106::task::568::TaskManager.Task::(_updateState) Task=`5352a606-2273-4f12-ad54-37dc7169e326`::moving from state init -> state preparing >Thread-6963::INFO::2012-12-02 23:29:13,106::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6963::INFO::2012-12-02 23:29:13,107::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00835299491882', 'lastCheck': 1354483749.2745309, 'code': 0, 'valid': True}} >Thread-6963::DEBUG::2012-12-02 23:29:13,107::task::1151::TaskManager.Task::(prepare) Task=`5352a606-2273-4f12-ad54-37dc7169e326`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00835299491882', 'lastCheck': 1354483749.2745309, 'code': 0, 'valid': True}} >Thread-6963::DEBUG::2012-12-02 23:29:13,107::task::568::TaskManager.Task::(_updateState) Task=`5352a606-2273-4f12-ad54-37dc7169e326`::moving from state preparing -> state finished >Thread-6963::DEBUG::2012-12-02 23:29:13,107::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6963::DEBUG::2012-12-02 23:29:13,107::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6963::DEBUG::2012-12-02 23:29:13,107::task::957::TaskManager.Task::(_decref) Task=`5352a606-2273-4f12-ad54-37dc7169e326`::ref 0 aborting False >Thread-6967::DEBUG::2012-12-02 23:29:18,009::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6967::DEBUG::2012-12-02 23:29:18,010::task::568::TaskManager.Task::(_updateState) Task=`d1d24303-9e44-415b-9059-4be57444c30b`::moving from state init -> state preparing >Thread-6967::INFO::2012-12-02 23:29:18,010::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6967::INFO::2012-12-02 23:29:18,010::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6967::DEBUG::2012-12-02 23:29:18,010::task::1151::TaskManager.Task::(prepare) Task=`d1d24303-9e44-415b-9059-4be57444c30b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6967::DEBUG::2012-12-02 23:29:18,010::task::568::TaskManager.Task::(_updateState) Task=`d1d24303-9e44-415b-9059-4be57444c30b`::moving from state preparing -> state finished >Thread-6967::DEBUG::2012-12-02 23:29:18,011::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6967::DEBUG::2012-12-02 23:29:18,011::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6967::DEBUG::2012-12-02 23:29:18,011::task::957::TaskManager.Task::(_decref) Task=`d1d24303-9e44-415b-9059-4be57444c30b`::ref 0 aborting False >Thread-6968::DEBUG::2012-12-02 23:29:18,155::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6968::DEBUG::2012-12-02 23:29:18,156::task::568::TaskManager.Task::(_updateState) Task=`3c891289-9563-43c3-895b-1f65f3517031`::moving from state init -> state preparing >Thread-6968::INFO::2012-12-02 23:29:18,156::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6968::DEBUG::2012-12-02 23:29:18,156::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c51aa6c7-e4c5-430b-ad57-f63068f39389`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6968::DEBUG::2012-12-02 23:29:18,156::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6968::DEBUG::2012-12-02 23:29:18,156::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6968::DEBUG::2012-12-02 23:29:18,156::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c51aa6c7-e4c5-430b-ad57-f63068f39389`::Granted request >Thread-6968::DEBUG::2012-12-02 23:29:18,157::task::794::TaskManager.Task::(resourceAcquired) Task=`3c891289-9563-43c3-895b-1f65f3517031`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6968::DEBUG::2012-12-02 23:29:18,157::task::957::TaskManager.Task::(_decref) Task=`3c891289-9563-43c3-895b-1f65f3517031`::ref 1 aborting False >Thread-6968::DEBUG::2012-12-02 23:29:18,163::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6968::DEBUG::2012-12-02 23:29:18,164::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6968::DEBUG::2012-12-02 23:29:18,370::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6968::WARNING::2012-12-02 23:29:18,371::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6968::DEBUG::2012-12-02 23:29:18,372::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6968::WARNING::2012-12-02 23:29:18,400::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6968::INFO::2012-12-02 23:29:18,400::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6968::DEBUG::2012-12-02 23:29:18,400::task::1151::TaskManager.Task::(prepare) Task=`3c891289-9563-43c3-895b-1f65f3517031`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-6968::DEBUG::2012-12-02 23:29:18,400::task::568::TaskManager.Task::(_updateState) Task=`3c891289-9563-43c3-895b-1f65f3517031`::moving from state preparing -> state finished >Thread-6968::DEBUG::2012-12-02 23:29:18,401::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6968::DEBUG::2012-12-02 23:29:18,401::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6968::DEBUG::2012-12-02 23:29:18,401::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6968::DEBUG::2012-12-02 23:29:18,401::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6968::DEBUG::2012-12-02 23:29:18,401::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6968::DEBUG::2012-12-02 23:29:18,401::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6968::DEBUG::2012-12-02 23:29:18,402::task::957::TaskManager.Task::(_decref) Task=`3c891289-9563-43c3-895b-1f65f3517031`::ref 0 aborting False >Thread-7032::DEBUG::2012-12-02 23:29:24,071::task::568::TaskManager.Task::(_updateState) Task=`f87aa140-4b70-46b8-a10e-c29e1aeb5e6f`::moving from state init -> state preparing >Thread-7032::INFO::2012-12-02 23:29:24,072::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7032::INFO::2012-12-02 23:29:24,072::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00854897499084', 'lastCheck': 1354483759.2937069, 'code': 0, 'valid': True}} >Thread-7032::DEBUG::2012-12-02 23:29:24,072::task::1151::TaskManager.Task::(prepare) Task=`f87aa140-4b70-46b8-a10e-c29e1aeb5e6f`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00854897499084', 'lastCheck': 1354483759.2937069, 'code': 0, 'valid': True}} >Thread-7032::DEBUG::2012-12-02 23:29:24,072::task::568::TaskManager.Task::(_updateState) Task=`f87aa140-4b70-46b8-a10e-c29e1aeb5e6f`::moving from state preparing -> state finished >Thread-7032::DEBUG::2012-12-02 23:29:24,072::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7032::DEBUG::2012-12-02 23:29:24,072::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7032::DEBUG::2012-12-02 23:29:24,072::task::957::TaskManager.Task::(_decref) Task=`f87aa140-4b70-46b8-a10e-c29e1aeb5e6f`::ref 0 aborting False >Thread-7036::DEBUG::2012-12-02 23:29:28,641::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7036::DEBUG::2012-12-02 23:29:28,641::task::568::TaskManager.Task::(_updateState) Task=`85f7bebb-f3f8-4cb2-84b3-39ed88daaa88`::moving from state init -> state preparing >Thread-7036::INFO::2012-12-02 23:29:28,641::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7036::INFO::2012-12-02 23:29:28,641::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7036::DEBUG::2012-12-02 23:29:28,642::task::1151::TaskManager.Task::(prepare) Task=`85f7bebb-f3f8-4cb2-84b3-39ed88daaa88`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7036::DEBUG::2012-12-02 23:29:28,642::task::568::TaskManager.Task::(_updateState) Task=`85f7bebb-f3f8-4cb2-84b3-39ed88daaa88`::moving from state preparing -> state finished >Thread-7036::DEBUG::2012-12-02 23:29:28,642::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7036::DEBUG::2012-12-02 23:29:28,642::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7036::DEBUG::2012-12-02 23:29:28,642::task::957::TaskManager.Task::(_decref) Task=`85f7bebb-f3f8-4cb2-84b3-39ed88daaa88`::ref 0 aborting False >Thread-7037::DEBUG::2012-12-02 23:29:28,785::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7037::DEBUG::2012-12-02 23:29:28,786::task::568::TaskManager.Task::(_updateState) Task=`b197d9f7-83d3-491d-8946-710f1239431d`::moving from state init -> state preparing >Thread-7037::INFO::2012-12-02 23:29:28,786::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7037::DEBUG::2012-12-02 23:29:28,786::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`aecc6ad7-d678-4c15-83ef-a606bd684c15`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7037::DEBUG::2012-12-02 23:29:28,786::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7037::DEBUG::2012-12-02 23:29:28,786::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7037::DEBUG::2012-12-02 23:29:28,787::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`aecc6ad7-d678-4c15-83ef-a606bd684c15`::Granted request >Thread-7037::DEBUG::2012-12-02 23:29:28,787::task::794::TaskManager.Task::(resourceAcquired) Task=`b197d9f7-83d3-491d-8946-710f1239431d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7037::DEBUG::2012-12-02 23:29:28,787::task::957::TaskManager.Task::(_decref) Task=`b197d9f7-83d3-491d-8946-710f1239431d`::ref 1 aborting False >Thread-7037::DEBUG::2012-12-02 23:29:28,793::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7037::DEBUG::2012-12-02 23:29:28,794::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7037::DEBUG::2012-12-02 23:29:28,998::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7037::WARNING::2012-12-02 23:29:28,999::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7037::DEBUG::2012-12-02 23:29:28,999::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7037::WARNING::2012-12-02 23:29:29,028::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7037::INFO::2012-12-02 23:29:29,029::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7037::DEBUG::2012-12-02 23:29:29,029::task::1151::TaskManager.Task::(prepare) Task=`b197d9f7-83d3-491d-8946-710f1239431d`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7037::DEBUG::2012-12-02 23:29:29,029::task::568::TaskManager.Task::(_updateState) Task=`b197d9f7-83d3-491d-8946-710f1239431d`::moving from state preparing -> state finished >Thread-7037::DEBUG::2012-12-02 23:29:29,029::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7037::DEBUG::2012-12-02 23:29:29,029::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7037::DEBUG::2012-12-02 23:29:29,030::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7037::DEBUG::2012-12-02 23:29:29,030::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7037::DEBUG::2012-12-02 23:29:29,030::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7037::DEBUG::2012-12-02 23:29:29,030::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7037::DEBUG::2012-12-02 23:29:29,030::task::957::TaskManager.Task::(_decref) Task=`b197d9f7-83d3-491d-8946-710f1239431d`::ref 0 aborting False >Thread-7101::DEBUG::2012-12-02 23:29:35,017::task::568::TaskManager.Task::(_updateState) Task=`31c06c0a-6b6b-444c-97d9-5851e7d4d87f`::moving from state init -> state preparing >Thread-7101::INFO::2012-12-02 23:29:35,017::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7101::INFO::2012-12-02 23:29:35,018::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00836110115051', 'lastCheck': 1354483769.3126459, 'code': 0, 'valid': True}} >Thread-7101::DEBUG::2012-12-02 23:29:35,018::task::1151::TaskManager.Task::(prepare) Task=`31c06c0a-6b6b-444c-97d9-5851e7d4d87f`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00836110115051', 'lastCheck': 1354483769.3126459, 'code': 0, 'valid': True}} >Thread-7101::DEBUG::2012-12-02 23:29:35,018::task::568::TaskManager.Task::(_updateState) Task=`31c06c0a-6b6b-444c-97d9-5851e7d4d87f`::moving from state preparing -> state finished >Thread-7101::DEBUG::2012-12-02 23:29:35,018::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7101::DEBUG::2012-12-02 23:29:35,018::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7101::DEBUG::2012-12-02 23:29:35,018::task::957::TaskManager.Task::(_decref) Task=`31c06c0a-6b6b-444c-97d9-5851e7d4d87f`::ref 0 aborting False >Thread-7104::DEBUG::2012-12-02 23:29:39,232::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7104::DEBUG::2012-12-02 23:29:39,232::task::568::TaskManager.Task::(_updateState) Task=`f897545d-bbec-4432-9ade-0fad2a1999d0`::moving from state init -> state preparing >Thread-7104::INFO::2012-12-02 23:29:39,232::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7104::INFO::2012-12-02 23:29:39,233::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7104::DEBUG::2012-12-02 23:29:39,233::task::1151::TaskManager.Task::(prepare) Task=`f897545d-bbec-4432-9ade-0fad2a1999d0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7104::DEBUG::2012-12-02 23:29:39,233::task::568::TaskManager.Task::(_updateState) Task=`f897545d-bbec-4432-9ade-0fad2a1999d0`::moving from state preparing -> state finished >Thread-7104::DEBUG::2012-12-02 23:29:39,233::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7104::DEBUG::2012-12-02 23:29:39,233::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7104::DEBUG::2012-12-02 23:29:39,233::task::957::TaskManager.Task::(_decref) Task=`f897545d-bbec-4432-9ade-0fad2a1999d0`::ref 0 aborting False >Thread-7105::DEBUG::2012-12-02 23:29:39,384::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7105::DEBUG::2012-12-02 23:29:39,384::task::568::TaskManager.Task::(_updateState) Task=`87ec570f-ef03-4ac9-8ddb-968343368ee2`::moving from state init -> state preparing >Thread-7105::INFO::2012-12-02 23:29:39,384::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7105::DEBUG::2012-12-02 23:29:39,385::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`26882aa9-aeaf-431c-87c7-36a6fbb302e7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7105::DEBUG::2012-12-02 23:29:39,385::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7105::DEBUG::2012-12-02 23:29:39,385::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7105::DEBUG::2012-12-02 23:29:39,385::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`26882aa9-aeaf-431c-87c7-36a6fbb302e7`::Granted request >Thread-7105::DEBUG::2012-12-02 23:29:39,385::task::794::TaskManager.Task::(resourceAcquired) Task=`87ec570f-ef03-4ac9-8ddb-968343368ee2`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7105::DEBUG::2012-12-02 23:29:39,385::task::957::TaskManager.Task::(_decref) Task=`87ec570f-ef03-4ac9-8ddb-968343368ee2`::ref 1 aborting False >Thread-7105::DEBUG::2012-12-02 23:29:39,390::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7105::DEBUG::2012-12-02 23:29:39,390::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7105::DEBUG::2012-12-02 23:29:39,596::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7105::WARNING::2012-12-02 23:29:39,597::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7105::DEBUG::2012-12-02 23:29:39,598::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7105::WARNING::2012-12-02 23:29:39,629::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7105::INFO::2012-12-02 23:29:39,630::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7105::DEBUG::2012-12-02 23:29:39,630::task::1151::TaskManager.Task::(prepare) Task=`87ec570f-ef03-4ac9-8ddb-968343368ee2`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7105::DEBUG::2012-12-02 23:29:39,630::task::568::TaskManager.Task::(_updateState) Task=`87ec570f-ef03-4ac9-8ddb-968343368ee2`::moving from state preparing -> state finished >Thread-7105::DEBUG::2012-12-02 23:29:39,630::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7105::DEBUG::2012-12-02 23:29:39,630::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7105::DEBUG::2012-12-02 23:29:39,631::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7105::DEBUG::2012-12-02 23:29:39,631::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7105::DEBUG::2012-12-02 23:29:39,631::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7105::DEBUG::2012-12-02 23:29:39,631::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7105::DEBUG::2012-12-02 23:29:39,631::task::957::TaskManager.Task::(_decref) Task=`87ec570f-ef03-4ac9-8ddb-968343368ee2`::ref 0 aborting False >Thread-7170::DEBUG::2012-12-02 23:29:46,062::task::568::TaskManager.Task::(_updateState) Task=`6ccd2a29-ed6e-4aaa-a2fb-8a4f08f851d3`::moving from state init -> state preparing >Thread-7170::INFO::2012-12-02 23:29:46,062::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7170::INFO::2012-12-02 23:29:46,062::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00788187980652', 'lastCheck': 1354483779.330982, 'code': 0, 'valid': True}} >Thread-7170::DEBUG::2012-12-02 23:29:46,062::task::1151::TaskManager.Task::(prepare) Task=`6ccd2a29-ed6e-4aaa-a2fb-8a4f08f851d3`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00788187980652', 'lastCheck': 1354483779.330982, 'code': 0, 'valid': True}} >Thread-7170::DEBUG::2012-12-02 23:29:46,063::task::568::TaskManager.Task::(_updateState) Task=`6ccd2a29-ed6e-4aaa-a2fb-8a4f08f851d3`::moving from state preparing -> state finished >Thread-7170::DEBUG::2012-12-02 23:29:46,063::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7170::DEBUG::2012-12-02 23:29:46,063::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7170::DEBUG::2012-12-02 23:29:46,063::task::957::TaskManager.Task::(_decref) Task=`6ccd2a29-ed6e-4aaa-a2fb-8a4f08f851d3`::ref 0 aborting False >Thread-7173::DEBUG::2012-12-02 23:29:49,878::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7173::DEBUG::2012-12-02 23:29:49,879::task::568::TaskManager.Task::(_updateState) Task=`f1e62d21-9bb1-4fc2-9a9c-d8e010efed17`::moving from state init -> state preparing >Thread-7173::INFO::2012-12-02 23:29:49,879::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7173::INFO::2012-12-02 23:29:49,879::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7173::DEBUG::2012-12-02 23:29:49,879::task::1151::TaskManager.Task::(prepare) Task=`f1e62d21-9bb1-4fc2-9a9c-d8e010efed17`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7173::DEBUG::2012-12-02 23:29:49,880::task::568::TaskManager.Task::(_updateState) Task=`f1e62d21-9bb1-4fc2-9a9c-d8e010efed17`::moving from state preparing -> state finished >Thread-7173::DEBUG::2012-12-02 23:29:49,880::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7173::DEBUG::2012-12-02 23:29:49,880::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7173::DEBUG::2012-12-02 23:29:49,880::task::957::TaskManager.Task::(_decref) Task=`f1e62d21-9bb1-4fc2-9a9c-d8e010efed17`::ref 0 aborting False >Thread-7174::DEBUG::2012-12-02 23:29:50,025::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7174::DEBUG::2012-12-02 23:29:50,026::task::568::TaskManager.Task::(_updateState) Task=`6290899d-8792-4221-9246-a567580065e3`::moving from state init -> state preparing >Thread-7174::INFO::2012-12-02 23:29:50,026::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7174::DEBUG::2012-12-02 23:29:50,026::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f90b8375-65b9-40ac-8cb1-160bb6a302ce`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7174::DEBUG::2012-12-02 23:29:50,026::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7174::DEBUG::2012-12-02 23:29:50,027::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7174::DEBUG::2012-12-02 23:29:50,027::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f90b8375-65b9-40ac-8cb1-160bb6a302ce`::Granted request >Thread-7174::DEBUG::2012-12-02 23:29:50,027::task::794::TaskManager.Task::(resourceAcquired) Task=`6290899d-8792-4221-9246-a567580065e3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7174::DEBUG::2012-12-02 23:29:50,027::task::957::TaskManager.Task::(_decref) Task=`6290899d-8792-4221-9246-a567580065e3`::ref 1 aborting False >Thread-7174::DEBUG::2012-12-02 23:29:50,034::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7174::DEBUG::2012-12-02 23:29:50,034::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7174::DEBUG::2012-12-02 23:29:50,242::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7174::WARNING::2012-12-02 23:29:50,243::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7174::DEBUG::2012-12-02 23:29:50,243::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7174::WARNING::2012-12-02 23:29:50,272::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7174::INFO::2012-12-02 23:29:50,273::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7174::DEBUG::2012-12-02 23:29:50,273::task::1151::TaskManager.Task::(prepare) Task=`6290899d-8792-4221-9246-a567580065e3`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7174::DEBUG::2012-12-02 23:29:50,273::task::568::TaskManager.Task::(_updateState) Task=`6290899d-8792-4221-9246-a567580065e3`::moving from state preparing -> state finished >Thread-7174::DEBUG::2012-12-02 23:29:50,273::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7174::DEBUG::2012-12-02 23:29:50,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7174::DEBUG::2012-12-02 23:29:50,274::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7174::DEBUG::2012-12-02 23:29:50,274::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7174::DEBUG::2012-12-02 23:29:50,274::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7174::DEBUG::2012-12-02 23:29:50,274::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7174::DEBUG::2012-12-02 23:29:50,274::task::957::TaskManager.Task::(_decref) Task=`6290899d-8792-4221-9246-a567580065e3`::ref 0 aborting False >Thread-7239::DEBUG::2012-12-02 23:29:57,085::task::568::TaskManager.Task::(_updateState) Task=`f94dbb66-fbde-49df-8da4-a809be5eb857`::moving from state init -> state preparing >Thread-7239::INFO::2012-12-02 23:29:57,085::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7239::INFO::2012-12-02 23:29:57,085::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00877594947815', 'lastCheck': 1354483789.350348, 'code': 0, 'valid': True}} >Thread-7239::DEBUG::2012-12-02 23:29:57,085::task::1151::TaskManager.Task::(prepare) Task=`f94dbb66-fbde-49df-8da4-a809be5eb857`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00877594947815', 'lastCheck': 1354483789.350348, 'code': 0, 'valid': True}} >Thread-7239::DEBUG::2012-12-02 23:29:57,086::task::568::TaskManager.Task::(_updateState) Task=`f94dbb66-fbde-49df-8da4-a809be5eb857`::moving from state preparing -> state finished >Thread-7239::DEBUG::2012-12-02 23:29:57,086::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7239::DEBUG::2012-12-02 23:29:57,086::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7239::DEBUG::2012-12-02 23:29:57,086::task::957::TaskManager.Task::(_decref) Task=`f94dbb66-fbde-49df-8da4-a809be5eb857`::ref 0 aborting False >Thread-7242::DEBUG::2012-12-02 23:30:00,484::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7242::DEBUG::2012-12-02 23:30:00,484::task::568::TaskManager.Task::(_updateState) Task=`1b0d728b-bfde-4a9f-b3d6-923d96ad495d`::moving from state init -> state preparing >Thread-7242::INFO::2012-12-02 23:30:00,484::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7242::INFO::2012-12-02 23:30:00,484::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7242::DEBUG::2012-12-02 23:30:00,484::task::1151::TaskManager.Task::(prepare) Task=`1b0d728b-bfde-4a9f-b3d6-923d96ad495d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7242::DEBUG::2012-12-02 23:30:00,485::task::568::TaskManager.Task::(_updateState) Task=`1b0d728b-bfde-4a9f-b3d6-923d96ad495d`::moving from state preparing -> state finished >Thread-7242::DEBUG::2012-12-02 23:30:00,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7242::DEBUG::2012-12-02 23:30:00,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7242::DEBUG::2012-12-02 23:30:00,485::task::957::TaskManager.Task::(_decref) Task=`1b0d728b-bfde-4a9f-b3d6-923d96ad495d`::ref 0 aborting False >Thread-7243::DEBUG::2012-12-02 23:30:00,627::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7243::DEBUG::2012-12-02 23:30:00,628::task::568::TaskManager.Task::(_updateState) Task=`108ba9aa-0476-43da-8212-27786424f840`::moving from state init -> state preparing >Thread-7243::INFO::2012-12-02 23:30:00,628::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7243::DEBUG::2012-12-02 23:30:00,628::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3b6715d5-d96f-453b-ad01-a2ffdfb12190`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7243::DEBUG::2012-12-02 23:30:00,628::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7243::DEBUG::2012-12-02 23:30:00,629::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7243::DEBUG::2012-12-02 23:30:00,629::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3b6715d5-d96f-453b-ad01-a2ffdfb12190`::Granted request >Thread-7243::DEBUG::2012-12-02 23:30:00,629::task::794::TaskManager.Task::(resourceAcquired) Task=`108ba9aa-0476-43da-8212-27786424f840`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7243::DEBUG::2012-12-02 23:30:00,629::task::957::TaskManager.Task::(_decref) Task=`108ba9aa-0476-43da-8212-27786424f840`::ref 1 aborting False >Thread-7243::DEBUG::2012-12-02 23:30:00,635::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7243::DEBUG::2012-12-02 23:30:00,636::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7243::DEBUG::2012-12-02 23:30:00,842::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7243::WARNING::2012-12-02 23:30:00,843::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7243::DEBUG::2012-12-02 23:30:00,843::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7243::WARNING::2012-12-02 23:30:00,872::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7243::INFO::2012-12-02 23:30:00,872::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7243::DEBUG::2012-12-02 23:30:00,873::task::1151::TaskManager.Task::(prepare) Task=`108ba9aa-0476-43da-8212-27786424f840`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7243::DEBUG::2012-12-02 23:30:00,873::task::568::TaskManager.Task::(_updateState) Task=`108ba9aa-0476-43da-8212-27786424f840`::moving from state preparing -> state finished >Thread-7243::DEBUG::2012-12-02 23:30:00,873::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7243::DEBUG::2012-12-02 23:30:00,873::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7243::DEBUG::2012-12-02 23:30:00,873::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7243::DEBUG::2012-12-02 23:30:00,873::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7243::DEBUG::2012-12-02 23:30:00,874::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7243::DEBUG::2012-12-02 23:30:00,874::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7243::DEBUG::2012-12-02 23:30:00,874::task::957::TaskManager.Task::(_decref) Task=`108ba9aa-0476-43da-8212-27786424f840`::ref 0 aborting False >Thread-7308::DEBUG::2012-12-02 23:30:08,002::task::568::TaskManager.Task::(_updateState) Task=`3354688e-8594-4fa4-b4f6-d863795ad060`::moving from state init -> state preparing >Thread-7308::INFO::2012-12-02 23:30:08,002::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7308::INFO::2012-12-02 23:30:08,002::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0106508731842', 'lastCheck': 1354483799.375416, 'code': 0, 'valid': True}} >Thread-7308::DEBUG::2012-12-02 23:30:08,002::task::1151::TaskManager.Task::(prepare) Task=`3354688e-8594-4fa4-b4f6-d863795ad060`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0106508731842', 'lastCheck': 1354483799.375416, 'code': 0, 'valid': True}} >Thread-7308::DEBUG::2012-12-02 23:30:08,003::task::568::TaskManager.Task::(_updateState) Task=`3354688e-8594-4fa4-b4f6-d863795ad060`::moving from state preparing -> state finished >Thread-7308::DEBUG::2012-12-02 23:30:08,003::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7308::DEBUG::2012-12-02 23:30:08,003::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7308::DEBUG::2012-12-02 23:30:08,003::task::957::TaskManager.Task::(_decref) Task=`3354688e-8594-4fa4-b4f6-d863795ad060`::ref 0 aborting False >Thread-7311::DEBUG::2012-12-02 23:30:11,121::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7311::DEBUG::2012-12-02 23:30:11,122::task::568::TaskManager.Task::(_updateState) Task=`8a5028cb-89cf-4e14-bf3e-14318276025d`::moving from state init -> state preparing >Thread-7311::INFO::2012-12-02 23:30:11,122::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7311::INFO::2012-12-02 23:30:11,122::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7311::DEBUG::2012-12-02 23:30:11,122::task::1151::TaskManager.Task::(prepare) Task=`8a5028cb-89cf-4e14-bf3e-14318276025d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7311::DEBUG::2012-12-02 23:30:11,122::task::568::TaskManager.Task::(_updateState) Task=`8a5028cb-89cf-4e14-bf3e-14318276025d`::moving from state preparing -> state finished >Thread-7311::DEBUG::2012-12-02 23:30:11,123::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7311::DEBUG::2012-12-02 23:30:11,123::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7311::DEBUG::2012-12-02 23:30:11,123::task::957::TaskManager.Task::(_decref) Task=`8a5028cb-89cf-4e14-bf3e-14318276025d`::ref 0 aborting False >Thread-7312::DEBUG::2012-12-02 23:30:11,265::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7312::DEBUG::2012-12-02 23:30:11,265::task::568::TaskManager.Task::(_updateState) Task=`5044e57d-aaa2-42b2-b311-25a8df077ab6`::moving from state init -> state preparing >Thread-7312::INFO::2012-12-02 23:30:11,265::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7312::DEBUG::2012-12-02 23:30:11,266::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d6f74390-de85-4933-9133-ae126f3beaf4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7312::DEBUG::2012-12-02 23:30:11,266::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7312::DEBUG::2012-12-02 23:30:11,266::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7312::DEBUG::2012-12-02 23:30:11,266::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d6f74390-de85-4933-9133-ae126f3beaf4`::Granted request >Thread-7312::DEBUG::2012-12-02 23:30:11,266::task::794::TaskManager.Task::(resourceAcquired) Task=`5044e57d-aaa2-42b2-b311-25a8df077ab6`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7312::DEBUG::2012-12-02 23:30:11,266::task::957::TaskManager.Task::(_decref) Task=`5044e57d-aaa2-42b2-b311-25a8df077ab6`::ref 1 aborting False >Thread-7312::DEBUG::2012-12-02 23:30:11,273::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7312::DEBUG::2012-12-02 23:30:11,273::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7312::DEBUG::2012-12-02 23:30:11,477::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7312::WARNING::2012-12-02 23:30:11,479::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7312::DEBUG::2012-12-02 23:30:11,479::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7312::WARNING::2012-12-02 23:30:11,507::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7312::INFO::2012-12-02 23:30:11,508::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7312::DEBUG::2012-12-02 23:30:11,508::task::1151::TaskManager.Task::(prepare) Task=`5044e57d-aaa2-42b2-b311-25a8df077ab6`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7312::DEBUG::2012-12-02 23:30:11,508::task::568::TaskManager.Task::(_updateState) Task=`5044e57d-aaa2-42b2-b311-25a8df077ab6`::moving from state preparing -> state finished >Thread-7312::DEBUG::2012-12-02 23:30:11,508::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7312::DEBUG::2012-12-02 23:30:11,508::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7312::DEBUG::2012-12-02 23:30:11,508::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7312::DEBUG::2012-12-02 23:30:11,509::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7312::DEBUG::2012-12-02 23:30:11,509::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7312::DEBUG::2012-12-02 23:30:11,509::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7312::DEBUG::2012-12-02 23:30:11,509::task::957::TaskManager.Task::(_decref) Task=`5044e57d-aaa2-42b2-b311-25a8df077ab6`::ref 0 aborting False >Thread-7377::DEBUG::2012-12-02 23:30:18,972::task::568::TaskManager.Task::(_updateState) Task=`01c124ac-cb61-483e-8625-ce7bd15e7f30`::moving from state init -> state preparing >Thread-7377::INFO::2012-12-02 23:30:18,973::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7377::INFO::2012-12-02 23:30:18,973::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00816702842712', 'lastCheck': 1354483809.3955979, 'code': 0, 'valid': True}} >Thread-7377::DEBUG::2012-12-02 23:30:18,973::task::1151::TaskManager.Task::(prepare) Task=`01c124ac-cb61-483e-8625-ce7bd15e7f30`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00816702842712', 'lastCheck': 1354483809.3955979, 'code': 0, 'valid': True}} >Thread-7377::DEBUG::2012-12-02 23:30:18,973::task::568::TaskManager.Task::(_updateState) Task=`01c124ac-cb61-483e-8625-ce7bd15e7f30`::moving from state preparing -> state finished >Thread-7377::DEBUG::2012-12-02 23:30:18,973::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7377::DEBUG::2012-12-02 23:30:18,973::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7377::DEBUG::2012-12-02 23:30:18,974::task::957::TaskManager.Task::(_decref) Task=`01c124ac-cb61-483e-8625-ce7bd15e7f30`::ref 0 aborting False >Thread-7380::DEBUG::2012-12-02 23:30:21,745::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7380::DEBUG::2012-12-02 23:30:21,745::task::568::TaskManager.Task::(_updateState) Task=`956bc48d-bd7b-471e-bd54-08a7280aaefb`::moving from state init -> state preparing >Thread-7380::INFO::2012-12-02 23:30:21,745::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7380::INFO::2012-12-02 23:30:21,745::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7380::DEBUG::2012-12-02 23:30:21,745::task::1151::TaskManager.Task::(prepare) Task=`956bc48d-bd7b-471e-bd54-08a7280aaefb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7380::DEBUG::2012-12-02 23:30:21,746::task::568::TaskManager.Task::(_updateState) Task=`956bc48d-bd7b-471e-bd54-08a7280aaefb`::moving from state preparing -> state finished >Thread-7380::DEBUG::2012-12-02 23:30:21,746::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7380::DEBUG::2012-12-02 23:30:21,746::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7380::DEBUG::2012-12-02 23:30:21,746::task::957::TaskManager.Task::(_decref) Task=`956bc48d-bd7b-471e-bd54-08a7280aaefb`::ref 0 aborting False >Thread-7381::DEBUG::2012-12-02 23:30:21,888::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7381::DEBUG::2012-12-02 23:30:21,889::task::568::TaskManager.Task::(_updateState) Task=`60326fef-fcef-46e7-9313-8b25b68a2d67`::moving from state init -> state preparing >Thread-7381::INFO::2012-12-02 23:30:21,889::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7381::DEBUG::2012-12-02 23:30:21,889::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`bf535bf1-fff6-44f9-a148-32712ce2d3e3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7381::DEBUG::2012-12-02 23:30:21,889::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7381::DEBUG::2012-12-02 23:30:21,890::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7381::DEBUG::2012-12-02 23:30:21,890::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`bf535bf1-fff6-44f9-a148-32712ce2d3e3`::Granted request >Thread-7381::DEBUG::2012-12-02 23:30:21,890::task::794::TaskManager.Task::(resourceAcquired) Task=`60326fef-fcef-46e7-9313-8b25b68a2d67`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7381::DEBUG::2012-12-02 23:30:21,890::task::957::TaskManager.Task::(_decref) Task=`60326fef-fcef-46e7-9313-8b25b68a2d67`::ref 1 aborting False >Thread-7381::DEBUG::2012-12-02 23:30:21,896::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7381::DEBUG::2012-12-02 23:30:21,896::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7381::DEBUG::2012-12-02 23:30:22,103::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7381::WARNING::2012-12-02 23:30:22,105::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7381::DEBUG::2012-12-02 23:30:22,105::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7381::WARNING::2012-12-02 23:30:22,134::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7381::INFO::2012-12-02 23:30:22,135::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7381::DEBUG::2012-12-02 23:30:22,135::task::1151::TaskManager.Task::(prepare) Task=`60326fef-fcef-46e7-9313-8b25b68a2d67`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7381::DEBUG::2012-12-02 23:30:22,135::task::568::TaskManager.Task::(_updateState) Task=`60326fef-fcef-46e7-9313-8b25b68a2d67`::moving from state preparing -> state finished >Thread-7381::DEBUG::2012-12-02 23:30:22,135::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7381::DEBUG::2012-12-02 23:30:22,135::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7381::DEBUG::2012-12-02 23:30:22,136::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7381::DEBUG::2012-12-02 23:30:22,136::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7381::DEBUG::2012-12-02 23:30:22,136::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7381::DEBUG::2012-12-02 23:30:22,136::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7381::DEBUG::2012-12-02 23:30:22,136::task::957::TaskManager.Task::(_decref) Task=`60326fef-fcef-46e7-9313-8b25b68a2d67`::ref 0 aborting False >Thread-7446::DEBUG::2012-12-02 23:30:29,947::task::568::TaskManager.Task::(_updateState) Task=`7bc02e5b-623e-4c11-aeeb-a4f7d639b2eb`::moving from state init -> state preparing >Thread-7446::INFO::2012-12-02 23:30:29,947::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7446::INFO::2012-12-02 23:30:29,947::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00890183448792', 'lastCheck': 1354483829.436276, 'code': 0, 'valid': True}} >Thread-7446::DEBUG::2012-12-02 23:30:29,948::task::1151::TaskManager.Task::(prepare) Task=`7bc02e5b-623e-4c11-aeeb-a4f7d639b2eb`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00890183448792', 'lastCheck': 1354483829.436276, 'code': 0, 'valid': True}} >Thread-7446::DEBUG::2012-12-02 23:30:29,948::task::568::TaskManager.Task::(_updateState) Task=`7bc02e5b-623e-4c11-aeeb-a4f7d639b2eb`::moving from state preparing -> state finished >Thread-7446::DEBUG::2012-12-02 23:30:29,948::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7446::DEBUG::2012-12-02 23:30:29,948::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7446::DEBUG::2012-12-02 23:30:29,948::task::957::TaskManager.Task::(_decref) Task=`7bc02e5b-623e-4c11-aeeb-a4f7d639b2eb`::ref 0 aborting False >Thread-7449::DEBUG::2012-12-02 23:30:32,387::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7449::DEBUG::2012-12-02 23:30:32,388::task::568::TaskManager.Task::(_updateState) Task=`74525379-cca9-4e87-8d88-5339d8aee9f6`::moving from state init -> state preparing >Thread-7449::INFO::2012-12-02 23:30:32,388::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7449::INFO::2012-12-02 23:30:32,388::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7449::DEBUG::2012-12-02 23:30:32,388::task::1151::TaskManager.Task::(prepare) Task=`74525379-cca9-4e87-8d88-5339d8aee9f6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7449::DEBUG::2012-12-02 23:30:32,388::task::568::TaskManager.Task::(_updateState) Task=`74525379-cca9-4e87-8d88-5339d8aee9f6`::moving from state preparing -> state finished >Thread-7449::DEBUG::2012-12-02 23:30:32,388::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7449::DEBUG::2012-12-02 23:30:32,389::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7449::DEBUG::2012-12-02 23:30:32,389::task::957::TaskManager.Task::(_decref) Task=`74525379-cca9-4e87-8d88-5339d8aee9f6`::ref 0 aborting False >Thread-7450::DEBUG::2012-12-02 23:30:32,523::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7450::DEBUG::2012-12-02 23:30:32,524::task::568::TaskManager.Task::(_updateState) Task=`ae4dab5a-566b-4cd2-94a9-7eecd031fdc2`::moving from state init -> state preparing >Thread-7450::INFO::2012-12-02 23:30:32,524::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7450::DEBUG::2012-12-02 23:30:32,524::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e25b903a-d3d2-4fcf-a256-26a6a83e8407`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7450::DEBUG::2012-12-02 23:30:32,524::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7450::DEBUG::2012-12-02 23:30:32,524::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7450::DEBUG::2012-12-02 23:30:32,524::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e25b903a-d3d2-4fcf-a256-26a6a83e8407`::Granted request >Thread-7450::DEBUG::2012-12-02 23:30:32,525::task::794::TaskManager.Task::(resourceAcquired) Task=`ae4dab5a-566b-4cd2-94a9-7eecd031fdc2`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7450::DEBUG::2012-12-02 23:30:32,525::task::957::TaskManager.Task::(_decref) Task=`ae4dab5a-566b-4cd2-94a9-7eecd031fdc2`::ref 1 aborting False >Thread-7450::DEBUG::2012-12-02 23:30:32,529::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7450::DEBUG::2012-12-02 23:30:32,530::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7450::DEBUG::2012-12-02 23:30:32,735::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7450::WARNING::2012-12-02 23:30:32,737::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7450::DEBUG::2012-12-02 23:30:32,737::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7450::WARNING::2012-12-02 23:30:32,767::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7450::INFO::2012-12-02 23:30:32,767::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7450::DEBUG::2012-12-02 23:30:32,767::task::1151::TaskManager.Task::(prepare) Task=`ae4dab5a-566b-4cd2-94a9-7eecd031fdc2`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7450::DEBUG::2012-12-02 23:30:32,767::task::568::TaskManager.Task::(_updateState) Task=`ae4dab5a-566b-4cd2-94a9-7eecd031fdc2`::moving from state preparing -> state finished >Thread-7450::DEBUG::2012-12-02 23:30:32,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7450::DEBUG::2012-12-02 23:30:32,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7450::DEBUG::2012-12-02 23:30:32,768::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7450::DEBUG::2012-12-02 23:30:32,768::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7450::DEBUG::2012-12-02 23:30:32,768::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7450::DEBUG::2012-12-02 23:30:32,768::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7450::DEBUG::2012-12-02 23:30:32,769::task::957::TaskManager.Task::(_decref) Task=`ae4dab5a-566b-4cd2-94a9-7eecd031fdc2`::ref 0 aborting False >Thread-7515::DEBUG::2012-12-02 23:30:40,931::task::568::TaskManager.Task::(_updateState) Task=`b0809649-1179-4d7d-a44f-f6902ac3bf47`::moving from state init -> state preparing >Thread-7515::INFO::2012-12-02 23:30:40,931::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7515::INFO::2012-12-02 23:30:40,931::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0102691650391', 'lastCheck': 1354483839.456423, 'code': 0, 'valid': True}} >Thread-7515::DEBUG::2012-12-02 23:30:40,931::task::1151::TaskManager.Task::(prepare) Task=`b0809649-1179-4d7d-a44f-f6902ac3bf47`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0102691650391', 'lastCheck': 1354483839.456423, 'code': 0, 'valid': True}} >Thread-7515::DEBUG::2012-12-02 23:30:40,932::task::568::TaskManager.Task::(_updateState) Task=`b0809649-1179-4d7d-a44f-f6902ac3bf47`::moving from state preparing -> state finished >Thread-7515::DEBUG::2012-12-02 23:30:40,932::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7515::DEBUG::2012-12-02 23:30:40,932::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7515::DEBUG::2012-12-02 23:30:40,932::task::957::TaskManager.Task::(_decref) Task=`b0809649-1179-4d7d-a44f-f6902ac3bf47`::ref 0 aborting False >Thread-7517::DEBUG::2012-12-02 23:30:42,979::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7517::DEBUG::2012-12-02 23:30:42,980::task::568::TaskManager.Task::(_updateState) Task=`7cecd98a-bdb9-45b9-9f5c-b4cfb2ac7731`::moving from state init -> state preparing >Thread-7517::INFO::2012-12-02 23:30:42,980::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7517::INFO::2012-12-02 23:30:42,980::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7517::DEBUG::2012-12-02 23:30:42,980::task::1151::TaskManager.Task::(prepare) Task=`7cecd98a-bdb9-45b9-9f5c-b4cfb2ac7731`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7517::DEBUG::2012-12-02 23:30:42,980::task::568::TaskManager.Task::(_updateState) Task=`7cecd98a-bdb9-45b9-9f5c-b4cfb2ac7731`::moving from state preparing -> state finished >Thread-7517::DEBUG::2012-12-02 23:30:42,981::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7517::DEBUG::2012-12-02 23:30:42,981::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7517::DEBUG::2012-12-02 23:30:42,981::task::957::TaskManager.Task::(_decref) Task=`7cecd98a-bdb9-45b9-9f5c-b4cfb2ac7731`::ref 0 aborting False >Thread-7518::DEBUG::2012-12-02 23:30:43,124::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7518::DEBUG::2012-12-02 23:30:43,124::task::568::TaskManager.Task::(_updateState) Task=`a9d5e369-1063-408c-aca6-f9bc6e914201`::moving from state init -> state preparing >Thread-7518::INFO::2012-12-02 23:30:43,124::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7518::DEBUG::2012-12-02 23:30:43,125::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c39796cb-3d91-4143-8b3e-6500dd031365`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7518::DEBUG::2012-12-02 23:30:43,125::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7518::DEBUG::2012-12-02 23:30:43,125::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7518::DEBUG::2012-12-02 23:30:43,125::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c39796cb-3d91-4143-8b3e-6500dd031365`::Granted request >Thread-7518::DEBUG::2012-12-02 23:30:43,125::task::794::TaskManager.Task::(resourceAcquired) Task=`a9d5e369-1063-408c-aca6-f9bc6e914201`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7518::DEBUG::2012-12-02 23:30:43,126::task::957::TaskManager.Task::(_decref) Task=`a9d5e369-1063-408c-aca6-f9bc6e914201`::ref 1 aborting False >Thread-7518::DEBUG::2012-12-02 23:30:43,132::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7518::DEBUG::2012-12-02 23:30:43,132::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7518::DEBUG::2012-12-02 23:30:43,340::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7518::WARNING::2012-12-02 23:30:43,342::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7518::DEBUG::2012-12-02 23:30:43,342::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7570::WARNING::2012-12-02 23:30:43,367::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7573::WARNING::2012-12-02 23:30:43,368::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7575::WARNING::2012-12-02 23:30:43,369::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7576::WARNING::2012-12-02 23:30:43,370::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7577::WARNING::2012-12-02 23:30:43,371::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7579::WARNING::2012-12-02 23:30:43,371::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7578::WARNING::2012-12-02 23:30:43,371::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7580::WARNING::2012-12-02 23:30:43,372::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7518::WARNING::2012-12-02 23:30:43,389::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7518::INFO::2012-12-02 23:30:43,389::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7518::DEBUG::2012-12-02 23:30:43,390::task::1151::TaskManager.Task::(prepare) Task=`a9d5e369-1063-408c-aca6-f9bc6e914201`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7518::DEBUG::2012-12-02 23:30:43,390::task::568::TaskManager.Task::(_updateState) Task=`a9d5e369-1063-408c-aca6-f9bc6e914201`::moving from state preparing -> state finished >Thread-7518::DEBUG::2012-12-02 23:30:43,390::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7518::DEBUG::2012-12-02 23:30:43,390::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7518::DEBUG::2012-12-02 23:30:43,390::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7518::DEBUG::2012-12-02 23:30:43,390::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7518::DEBUG::2012-12-02 23:30:43,391::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7518::DEBUG::2012-12-02 23:30:43,391::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7518::DEBUG::2012-12-02 23:30:43,391::task::957::TaskManager.Task::(_decref) Task=`a9d5e369-1063-408c-aca6-f9bc6e914201`::ref 0 aborting False >Thread-7584::DEBUG::2012-12-02 23:30:51,876::task::568::TaskManager.Task::(_updateState) Task=`21988a6c-2e82-4781-a243-8778dba0588e`::moving from state init -> state preparing >Thread-7584::INFO::2012-12-02 23:30:51,877::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7584::INFO::2012-12-02 23:30:51,877::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0124561786652', 'lastCheck': 1354483849.479795, 'code': 0, 'valid': True}} >Thread-7584::DEBUG::2012-12-02 23:30:51,877::task::1151::TaskManager.Task::(prepare) Task=`21988a6c-2e82-4781-a243-8778dba0588e`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0124561786652', 'lastCheck': 1354483849.479795, 'code': 0, 'valid': True}} >Thread-7584::DEBUG::2012-12-02 23:30:51,877::task::568::TaskManager.Task::(_updateState) Task=`21988a6c-2e82-4781-a243-8778dba0588e`::moving from state preparing -> state finished >Thread-7584::DEBUG::2012-12-02 23:30:51,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7584::DEBUG::2012-12-02 23:30:51,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7584::DEBUG::2012-12-02 23:30:51,878::task::957::TaskManager.Task::(_decref) Task=`21988a6c-2e82-4781-a243-8778dba0588e`::ref 0 aborting False >Thread-7586::DEBUG::2012-12-02 23:30:53,629::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7586::DEBUG::2012-12-02 23:30:53,630::task::568::TaskManager.Task::(_updateState) Task=`53ac7e22-d114-4ab0-835c-15b95ec5b619`::moving from state init -> state preparing >Thread-7586::INFO::2012-12-02 23:30:53,630::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7586::INFO::2012-12-02 23:30:53,630::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7586::DEBUG::2012-12-02 23:30:53,630::task::1151::TaskManager.Task::(prepare) Task=`53ac7e22-d114-4ab0-835c-15b95ec5b619`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7586::DEBUG::2012-12-02 23:30:53,630::task::568::TaskManager.Task::(_updateState) Task=`53ac7e22-d114-4ab0-835c-15b95ec5b619`::moving from state preparing -> state finished >Thread-7586::DEBUG::2012-12-02 23:30:53,631::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7586::DEBUG::2012-12-02 23:30:53,631::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7586::DEBUG::2012-12-02 23:30:53,631::task::957::TaskManager.Task::(_decref) Task=`53ac7e22-d114-4ab0-835c-15b95ec5b619`::ref 0 aborting False >Thread-7587::DEBUG::2012-12-02 23:30:53,770::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7587::DEBUG::2012-12-02 23:30:53,771::task::568::TaskManager.Task::(_updateState) Task=`d3ab4e88-ee9b-414b-8f46-53345c86be0d`::moving from state init -> state preparing >Thread-7587::INFO::2012-12-02 23:30:53,771::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7587::DEBUG::2012-12-02 23:30:53,771::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`649d90be-6a5c-4173-96b9-303f4eae92e8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7587::DEBUG::2012-12-02 23:30:53,771::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7587::DEBUG::2012-12-02 23:30:53,772::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7587::DEBUG::2012-12-02 23:30:53,772::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`649d90be-6a5c-4173-96b9-303f4eae92e8`::Granted request >Thread-7587::DEBUG::2012-12-02 23:30:53,772::task::794::TaskManager.Task::(resourceAcquired) Task=`d3ab4e88-ee9b-414b-8f46-53345c86be0d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7587::DEBUG::2012-12-02 23:30:53,772::task::957::TaskManager.Task::(_decref) Task=`d3ab4e88-ee9b-414b-8f46-53345c86be0d`::ref 1 aborting False >Thread-7587::DEBUG::2012-12-02 23:30:53,778::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7587::DEBUG::2012-12-02 23:30:53,778::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7587::DEBUG::2012-12-02 23:30:53,984::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7587::WARNING::2012-12-02 23:30:53,986::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7587::DEBUG::2012-12-02 23:30:53,986::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7587::WARNING::2012-12-02 23:30:54,020::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7587::INFO::2012-12-02 23:30:54,020::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7587::DEBUG::2012-12-02 23:30:54,020::task::1151::TaskManager.Task::(prepare) Task=`d3ab4e88-ee9b-414b-8f46-53345c86be0d`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7587::DEBUG::2012-12-02 23:30:54,021::task::568::TaskManager.Task::(_updateState) Task=`d3ab4e88-ee9b-414b-8f46-53345c86be0d`::moving from state preparing -> state finished >Thread-7587::DEBUG::2012-12-02 23:30:54,021::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7587::DEBUG::2012-12-02 23:30:54,021::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7587::DEBUG::2012-12-02 23:30:54,021::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7587::DEBUG::2012-12-02 23:30:54,021::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7587::DEBUG::2012-12-02 23:30:54,021::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7587::DEBUG::2012-12-02 23:30:54,021::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7587::DEBUG::2012-12-02 23:30:54,022::task::957::TaskManager.Task::(_decref) Task=`d3ab4e88-ee9b-414b-8f46-53345c86be0d`::ref 0 aborting False >Thread-5152::DEBUG::2012-12-02 23:30:59,480::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:30:59,480::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5152::DEBUG::2012-12-02 23:30:59,480::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5152::DEBUG::2012-12-02 23:30:59,686::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5152::WARNING::2012-12-02 23:30:59,688::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5152::DEBUG::2012-12-02 23:30:59,688::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5152::DEBUG::2012-12-02 23:30:59,722::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:30:59,722::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-5152::DEBUG::2012-12-02 23:30:59,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-5152::DEBUG::2012-12-02 23:30:59,734::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-5152::WARNING::2012-12-02 23:30:59,735::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-5152::WARNING::2012-12-02 23:30:59,735::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-7714::DEBUG::2012-12-02 23:31:02,809::task::568::TaskManager.Task::(_updateState) Task=`57d775bc-b758-4d4f-ae90-efd74a308611`::moving from state init -> state preparing >Thread-7714::INFO::2012-12-02 23:31:02,810::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7714::INFO::2012-12-02 23:31:02,810::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00338387489319', 'lastCheck': 1354483859.750422, 'code': 0, 'valid': True}} >Thread-7714::DEBUG::2012-12-02 23:31:02,810::task::1151::TaskManager.Task::(prepare) Task=`57d775bc-b758-4d4f-ae90-efd74a308611`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00338387489319', 'lastCheck': 1354483859.750422, 'code': 0, 'valid': True}} >Thread-7714::DEBUG::2012-12-02 23:31:02,810::task::568::TaskManager.Task::(_updateState) Task=`57d775bc-b758-4d4f-ae90-efd74a308611`::moving from state preparing -> state finished >Thread-7714::DEBUG::2012-12-02 23:31:02,810::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7714::DEBUG::2012-12-02 23:31:02,810::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7714::DEBUG::2012-12-02 23:31:02,810::task::957::TaskManager.Task::(_decref) Task=`57d775bc-b758-4d4f-ae90-efd74a308611`::ref 0 aborting False >Thread-7716::DEBUG::2012-12-02 23:31:04,234::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7716::DEBUG::2012-12-02 23:31:04,234::task::568::TaskManager.Task::(_updateState) Task=`92140baf-60b8-45bc-8cde-116e70d47f2f`::moving from state init -> state preparing >Thread-7716::INFO::2012-12-02 23:31:04,234::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7716::INFO::2012-12-02 23:31:04,234::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7716::DEBUG::2012-12-02 23:31:04,234::task::1151::TaskManager.Task::(prepare) Task=`92140baf-60b8-45bc-8cde-116e70d47f2f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7716::DEBUG::2012-12-02 23:31:04,235::task::568::TaskManager.Task::(_updateState) Task=`92140baf-60b8-45bc-8cde-116e70d47f2f`::moving from state preparing -> state finished >Thread-7716::DEBUG::2012-12-02 23:31:04,235::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7716::DEBUG::2012-12-02 23:31:04,235::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7716::DEBUG::2012-12-02 23:31:04,235::task::957::TaskManager.Task::(_decref) Task=`92140baf-60b8-45bc-8cde-116e70d47f2f`::ref 0 aborting False >Thread-7717::DEBUG::2012-12-02 23:31:04,380::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7717::DEBUG::2012-12-02 23:31:04,381::task::568::TaskManager.Task::(_updateState) Task=`72afcdec-c87d-4e2f-ab2b-2b6e6f23ec98`::moving from state init -> state preparing >Thread-7717::INFO::2012-12-02 23:31:04,381::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7717::DEBUG::2012-12-02 23:31:04,381::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3e76621b-036a-451e-be7a-80d0adb5ead1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7717::DEBUG::2012-12-02 23:31:04,381::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7717::DEBUG::2012-12-02 23:31:04,382::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7717::DEBUG::2012-12-02 23:31:04,382::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3e76621b-036a-451e-be7a-80d0adb5ead1`::Granted request >Thread-7717::DEBUG::2012-12-02 23:31:04,382::task::794::TaskManager.Task::(resourceAcquired) Task=`72afcdec-c87d-4e2f-ab2b-2b6e6f23ec98`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7717::DEBUG::2012-12-02 23:31:04,382::task::957::TaskManager.Task::(_decref) Task=`72afcdec-c87d-4e2f-ab2b-2b6e6f23ec98`::ref 1 aborting False >Thread-7717::DEBUG::2012-12-02 23:31:04,387::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7717::DEBUG::2012-12-02 23:31:04,388::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7717::DEBUG::2012-12-02 23:31:04,594::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7717::WARNING::2012-12-02 23:31:04,596::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7717::DEBUG::2012-12-02 23:31:04,596::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7717::WARNING::2012-12-02 23:31:04,627::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7717::INFO::2012-12-02 23:31:04,627::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7717::DEBUG::2012-12-02 23:31:04,627::task::1151::TaskManager.Task::(prepare) Task=`72afcdec-c87d-4e2f-ab2b-2b6e6f23ec98`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7717::DEBUG::2012-12-02 23:31:04,627::task::568::TaskManager.Task::(_updateState) Task=`72afcdec-c87d-4e2f-ab2b-2b6e6f23ec98`::moving from state preparing -> state finished >Thread-7717::DEBUG::2012-12-02 23:31:04,628::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7717::DEBUG::2012-12-02 23:31:04,628::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7717::DEBUG::2012-12-02 23:31:04,628::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7717::DEBUG::2012-12-02 23:31:04,628::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7717::DEBUG::2012-12-02 23:31:04,628::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7717::DEBUG::2012-12-02 23:31:04,628::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7717::DEBUG::2012-12-02 23:31:04,629::task::957::TaskManager.Task::(_decref) Task=`72afcdec-c87d-4e2f-ab2b-2b6e6f23ec98`::ref 0 aborting False >Thread-7783::DEBUG::2012-12-02 23:31:13,828::task::568::TaskManager.Task::(_updateState) Task=`4a89ec82-60e7-4bfe-9cca-05740f0c44d3`::moving from state init -> state preparing >Thread-7783::INFO::2012-12-02 23:31:13,828::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7783::INFO::2012-12-02 23:31:13,828::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00729179382324', 'lastCheck': 1354483869.7687421, 'code': 0, 'valid': True}} >Thread-7783::DEBUG::2012-12-02 23:31:13,828::task::1151::TaskManager.Task::(prepare) Task=`4a89ec82-60e7-4bfe-9cca-05740f0c44d3`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00729179382324', 'lastCheck': 1354483869.7687421, 'code': 0, 'valid': True}} >Thread-7783::DEBUG::2012-12-02 23:31:13,829::task::568::TaskManager.Task::(_updateState) Task=`4a89ec82-60e7-4bfe-9cca-05740f0c44d3`::moving from state preparing -> state finished >Thread-7783::DEBUG::2012-12-02 23:31:13,829::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7783::DEBUG::2012-12-02 23:31:13,829::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7783::DEBUG::2012-12-02 23:31:13,829::task::957::TaskManager.Task::(_decref) Task=`4a89ec82-60e7-4bfe-9cca-05740f0c44d3`::ref 0 aborting False >Thread-7785::DEBUG::2012-12-02 23:31:14,838::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7785::DEBUG::2012-12-02 23:31:14,838::task::568::TaskManager.Task::(_updateState) Task=`7f3a22f7-b992-40e4-8812-5971f304376a`::moving from state init -> state preparing >Thread-7785::INFO::2012-12-02 23:31:14,839::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7785::INFO::2012-12-02 23:31:14,839::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7785::DEBUG::2012-12-02 23:31:14,839::task::1151::TaskManager.Task::(prepare) Task=`7f3a22f7-b992-40e4-8812-5971f304376a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7785::DEBUG::2012-12-02 23:31:14,839::task::568::TaskManager.Task::(_updateState) Task=`7f3a22f7-b992-40e4-8812-5971f304376a`::moving from state preparing -> state finished >Thread-7785::DEBUG::2012-12-02 23:31:14,839::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7785::DEBUG::2012-12-02 23:31:14,839::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7785::DEBUG::2012-12-02 23:31:14,839::task::957::TaskManager.Task::(_decref) Task=`7f3a22f7-b992-40e4-8812-5971f304376a`::ref 0 aborting False >Thread-7786::DEBUG::2012-12-02 23:31:14,996::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7786::DEBUG::2012-12-02 23:31:14,997::task::568::TaskManager.Task::(_updateState) Task=`f747c2d7-d867-4428-9acf-f7beb7552f51`::moving from state init -> state preparing >Thread-7786::INFO::2012-12-02 23:31:14,997::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7786::DEBUG::2012-12-02 23:31:14,997::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e50e172a-7f55-47d3-a0da-d60ba79ac2cd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7786::DEBUG::2012-12-02 23:31:14,997::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7786::DEBUG::2012-12-02 23:31:14,997::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7786::DEBUG::2012-12-02 23:31:14,997::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e50e172a-7f55-47d3-a0da-d60ba79ac2cd`::Granted request >Thread-7786::DEBUG::2012-12-02 23:31:14,998::task::794::TaskManager.Task::(resourceAcquired) Task=`f747c2d7-d867-4428-9acf-f7beb7552f51`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7786::DEBUG::2012-12-02 23:31:14,998::task::957::TaskManager.Task::(_decref) Task=`f747c2d7-d867-4428-9acf-f7beb7552f51`::ref 1 aborting False >Thread-7786::DEBUG::2012-12-02 23:31:15,004::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7786::DEBUG::2012-12-02 23:31:15,005::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7786::DEBUG::2012-12-02 23:31:15,211::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7786::WARNING::2012-12-02 23:31:15,213::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7786::DEBUG::2012-12-02 23:31:15,213::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7786::WARNING::2012-12-02 23:31:15,242::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7786::INFO::2012-12-02 23:31:15,243::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7786::DEBUG::2012-12-02 23:31:15,243::task::1151::TaskManager.Task::(prepare) Task=`f747c2d7-d867-4428-9acf-f7beb7552f51`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7786::DEBUG::2012-12-02 23:31:15,243::task::568::TaskManager.Task::(_updateState) Task=`f747c2d7-d867-4428-9acf-f7beb7552f51`::moving from state preparing -> state finished >Thread-7786::DEBUG::2012-12-02 23:31:15,243::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7786::DEBUG::2012-12-02 23:31:15,243::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7786::DEBUG::2012-12-02 23:31:15,244::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7786::DEBUG::2012-12-02 23:31:15,244::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7786::DEBUG::2012-12-02 23:31:15,244::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7786::DEBUG::2012-12-02 23:31:15,244::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7786::DEBUG::2012-12-02 23:31:15,244::task::957::TaskManager.Task::(_decref) Task=`f747c2d7-d867-4428-9acf-f7beb7552f51`::ref 0 aborting False >Thread-7852::DEBUG::2012-12-02 23:31:24,851::task::568::TaskManager.Task::(_updateState) Task=`fb787ad5-9ab6-4762-9a74-c4703ba55f74`::moving from state init -> state preparing >Thread-7852::INFO::2012-12-02 23:31:24,852::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7852::INFO::2012-12-02 23:31:24,852::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00756001472473', 'lastCheck': 1354483879.786763, 'code': 0, 'valid': True}} >Thread-7852::DEBUG::2012-12-02 23:31:24,852::task::1151::TaskManager.Task::(prepare) Task=`fb787ad5-9ab6-4762-9a74-c4703ba55f74`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00756001472473', 'lastCheck': 1354483879.786763, 'code': 0, 'valid': True}} >Thread-7852::DEBUG::2012-12-02 23:31:24,852::task::568::TaskManager.Task::(_updateState) Task=`fb787ad5-9ab6-4762-9a74-c4703ba55f74`::moving from state preparing -> state finished >Thread-7852::DEBUG::2012-12-02 23:31:24,852::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7852::DEBUG::2012-12-02 23:31:24,852::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7852::DEBUG::2012-12-02 23:31:24,853::task::957::TaskManager.Task::(_decref) Task=`fb787ad5-9ab6-4762-9a74-c4703ba55f74`::ref 0 aborting False >Thread-7854::DEBUG::2012-12-02 23:31:25,472::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7854::DEBUG::2012-12-02 23:31:25,472::task::568::TaskManager.Task::(_updateState) Task=`766cd2b2-123d-4728-a807-cf140083e37f`::moving from state init -> state preparing >Thread-7854::INFO::2012-12-02 23:31:25,473::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7854::INFO::2012-12-02 23:31:25,473::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7854::DEBUG::2012-12-02 23:31:25,473::task::1151::TaskManager.Task::(prepare) Task=`766cd2b2-123d-4728-a807-cf140083e37f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7854::DEBUG::2012-12-02 23:31:25,473::task::568::TaskManager.Task::(_updateState) Task=`766cd2b2-123d-4728-a807-cf140083e37f`::moving from state preparing -> state finished >Thread-7854::DEBUG::2012-12-02 23:31:25,473::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7854::DEBUG::2012-12-02 23:31:25,473::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7854::DEBUG::2012-12-02 23:31:25,473::task::957::TaskManager.Task::(_decref) Task=`766cd2b2-123d-4728-a807-cf140083e37f`::ref 0 aborting False >Thread-7855::DEBUG::2012-12-02 23:31:25,619::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7855::DEBUG::2012-12-02 23:31:25,620::task::568::TaskManager.Task::(_updateState) Task=`02baa8bc-46d8-4019-bf23-2e8eea40f23b`::moving from state init -> state preparing >Thread-7855::INFO::2012-12-02 23:31:25,620::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7855::DEBUG::2012-12-02 23:31:25,620::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`42bb198b-6648-43a8-a95d-24a2f2585b47`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7855::DEBUG::2012-12-02 23:31:25,620::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7855::DEBUG::2012-12-02 23:31:25,621::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7855::DEBUG::2012-12-02 23:31:25,621::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`42bb198b-6648-43a8-a95d-24a2f2585b47`::Granted request >Thread-7855::DEBUG::2012-12-02 23:31:25,621::task::794::TaskManager.Task::(resourceAcquired) Task=`02baa8bc-46d8-4019-bf23-2e8eea40f23b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7855::DEBUG::2012-12-02 23:31:25,621::task::957::TaskManager.Task::(_decref) Task=`02baa8bc-46d8-4019-bf23-2e8eea40f23b`::ref 1 aborting False >Thread-7855::DEBUG::2012-12-02 23:31:25,627::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7855::DEBUG::2012-12-02 23:31:25,627::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7855::DEBUG::2012-12-02 23:31:25,835::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7855::WARNING::2012-12-02 23:31:25,837::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7855::DEBUG::2012-12-02 23:31:25,837::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7855::WARNING::2012-12-02 23:31:25,866::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7855::INFO::2012-12-02 23:31:25,867::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7855::DEBUG::2012-12-02 23:31:25,867::task::1151::TaskManager.Task::(prepare) Task=`02baa8bc-46d8-4019-bf23-2e8eea40f23b`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7855::DEBUG::2012-12-02 23:31:25,867::task::568::TaskManager.Task::(_updateState) Task=`02baa8bc-46d8-4019-bf23-2e8eea40f23b`::moving from state preparing -> state finished >Thread-7855::DEBUG::2012-12-02 23:31:25,867::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7855::DEBUG::2012-12-02 23:31:25,868::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7855::DEBUG::2012-12-02 23:31:25,868::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7855::DEBUG::2012-12-02 23:31:25,868::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7855::DEBUG::2012-12-02 23:31:25,868::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7855::DEBUG::2012-12-02 23:31:25,868::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7855::DEBUG::2012-12-02 23:31:25,868::task::957::TaskManager.Task::(_decref) Task=`02baa8bc-46d8-4019-bf23-2e8eea40f23b`::ref 0 aborting False >Thread-7921::DEBUG::2012-12-02 23:31:35,803::task::568::TaskManager.Task::(_updateState) Task=`a743a071-857b-4ae2-af0d-7a3ab7c05e64`::moving from state init -> state preparing >Thread-7921::INFO::2012-12-02 23:31:35,803::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7921::INFO::2012-12-02 23:31:35,803::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0073549747467', 'lastCheck': 1354483889.804013, 'code': 0, 'valid': True}} >Thread-7921::DEBUG::2012-12-02 23:31:35,804::task::1151::TaskManager.Task::(prepare) Task=`a743a071-857b-4ae2-af0d-7a3ab7c05e64`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0073549747467', 'lastCheck': 1354483889.804013, 'code': 0, 'valid': True}} >Thread-7921::DEBUG::2012-12-02 23:31:35,804::task::568::TaskManager.Task::(_updateState) Task=`a743a071-857b-4ae2-af0d-7a3ab7c05e64`::moving from state preparing -> state finished >Thread-7921::DEBUG::2012-12-02 23:31:35,804::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7921::DEBUG::2012-12-02 23:31:35,804::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7921::DEBUG::2012-12-02 23:31:35,804::task::957::TaskManager.Task::(_decref) Task=`a743a071-857b-4ae2-af0d-7a3ab7c05e64`::ref 0 aborting False >Thread-7923::DEBUG::2012-12-02 23:31:36,085::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7923::DEBUG::2012-12-02 23:31:36,086::task::568::TaskManager.Task::(_updateState) Task=`91cb6406-3933-4403-86b0-274ba4ab15df`::moving from state init -> state preparing >Thread-7923::INFO::2012-12-02 23:31:36,086::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7923::INFO::2012-12-02 23:31:36,086::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7923::DEBUG::2012-12-02 23:31:36,086::task::1151::TaskManager.Task::(prepare) Task=`91cb6406-3933-4403-86b0-274ba4ab15df`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7923::DEBUG::2012-12-02 23:31:36,086::task::568::TaskManager.Task::(_updateState) Task=`91cb6406-3933-4403-86b0-274ba4ab15df`::moving from state preparing -> state finished >Thread-7923::DEBUG::2012-12-02 23:31:36,086::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7923::DEBUG::2012-12-02 23:31:36,087::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7923::DEBUG::2012-12-02 23:31:36,087::task::957::TaskManager.Task::(_decref) Task=`91cb6406-3933-4403-86b0-274ba4ab15df`::ref 0 aborting False >Thread-7924::DEBUG::2012-12-02 23:31:36,243::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7924::DEBUG::2012-12-02 23:31:36,243::task::568::TaskManager.Task::(_updateState) Task=`12020f73-a5e2-4e16-b670-a7c313be700b`::moving from state init -> state preparing >Thread-7924::INFO::2012-12-02 23:31:36,243::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7924::DEBUG::2012-12-02 23:31:36,244::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4c487552-b9a0-47d3-b4f1-e952efa2ce59`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7924::DEBUG::2012-12-02 23:31:36,244::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7924::DEBUG::2012-12-02 23:31:36,244::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7924::DEBUG::2012-12-02 23:31:36,244::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4c487552-b9a0-47d3-b4f1-e952efa2ce59`::Granted request >Thread-7924::DEBUG::2012-12-02 23:31:36,244::task::794::TaskManager.Task::(resourceAcquired) Task=`12020f73-a5e2-4e16-b670-a7c313be700b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7924::DEBUG::2012-12-02 23:31:36,245::task::957::TaskManager.Task::(_decref) Task=`12020f73-a5e2-4e16-b670-a7c313be700b`::ref 1 aborting False >Thread-7924::DEBUG::2012-12-02 23:31:36,249::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7924::DEBUG::2012-12-02 23:31:36,249::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7924::DEBUG::2012-12-02 23:31:36,455::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7924::WARNING::2012-12-02 23:31:36,456::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7924::DEBUG::2012-12-02 23:31:36,456::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7924::WARNING::2012-12-02 23:31:36,486::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7924::INFO::2012-12-02 23:31:36,486::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7924::DEBUG::2012-12-02 23:31:36,486::task::1151::TaskManager.Task::(prepare) Task=`12020f73-a5e2-4e16-b670-a7c313be700b`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7924::DEBUG::2012-12-02 23:31:36,486::task::568::TaskManager.Task::(_updateState) Task=`12020f73-a5e2-4e16-b670-a7c313be700b`::moving from state preparing -> state finished >Thread-7924::DEBUG::2012-12-02 23:31:36,487::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7924::DEBUG::2012-12-02 23:31:36,487::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7924::DEBUG::2012-12-02 23:31:36,487::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7924::DEBUG::2012-12-02 23:31:36,487::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7924::DEBUG::2012-12-02 23:31:36,487::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7924::DEBUG::2012-12-02 23:31:36,487::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7924::DEBUG::2012-12-02 23:31:36,488::task::957::TaskManager.Task::(_decref) Task=`12020f73-a5e2-4e16-b670-a7c313be700b`::ref 0 aborting False >Thread-7990::DEBUG::2012-12-02 23:31:46,676::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7990::DEBUG::2012-12-02 23:31:46,676::task::568::TaskManager.Task::(_updateState) Task=`b7cf1b64-2697-4128-bbf1-b1d3369d7de8`::moving from state init -> state preparing >Thread-7990::INFO::2012-12-02 23:31:46,676::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7990::INFO::2012-12-02 23:31:46,677::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7990::DEBUG::2012-12-02 23:31:46,677::task::1151::TaskManager.Task::(prepare) Task=`b7cf1b64-2697-4128-bbf1-b1d3369d7de8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7990::DEBUG::2012-12-02 23:31:46,677::task::568::TaskManager.Task::(_updateState) Task=`b7cf1b64-2697-4128-bbf1-b1d3369d7de8`::moving from state preparing -> state finished >Thread-7990::DEBUG::2012-12-02 23:31:46,677::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7990::DEBUG::2012-12-02 23:31:46,677::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7990::DEBUG::2012-12-02 23:31:46,677::task::957::TaskManager.Task::(_decref) Task=`b7cf1b64-2697-4128-bbf1-b1d3369d7de8`::ref 0 aborting False >Thread-7991::DEBUG::2012-12-02 23:31:46,784::task::568::TaskManager.Task::(_updateState) Task=`a4c901a2-e5db-43ae-9f20-cafce9446f8a`::moving from state init -> state preparing >Thread-7991::INFO::2012-12-02 23:31:46,785::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7991::INFO::2012-12-02 23:31:46,785::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00728797912598', 'lastCheck': 1354483899.8214991, 'code': 0, 'valid': True}} >Thread-7991::DEBUG::2012-12-02 23:31:46,785::task::1151::TaskManager.Task::(prepare) Task=`a4c901a2-e5db-43ae-9f20-cafce9446f8a`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00728797912598', 'lastCheck': 1354483899.8214991, 'code': 0, 'valid': True}} >Thread-7991::DEBUG::2012-12-02 23:31:46,785::task::568::TaskManager.Task::(_updateState) Task=`a4c901a2-e5db-43ae-9f20-cafce9446f8a`::moving from state preparing -> state finished >Thread-7991::DEBUG::2012-12-02 23:31:46,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7991::DEBUG::2012-12-02 23:31:46,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7991::DEBUG::2012-12-02 23:31:46,785::task::957::TaskManager.Task::(_decref) Task=`a4c901a2-e5db-43ae-9f20-cafce9446f8a`::ref 0 aborting False >Thread-7992::DEBUG::2012-12-02 23:31:46,817::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7992::DEBUG::2012-12-02 23:31:46,818::task::568::TaskManager.Task::(_updateState) Task=`5c28b4e5-85fe-4e3a-b80e-f657d779614f`::moving from state init -> state preparing >Thread-7992::INFO::2012-12-02 23:31:46,818::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7992::DEBUG::2012-12-02 23:31:46,818::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`73d81e1c-9b74-40f5-a80e-600da0f5401c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7992::DEBUG::2012-12-02 23:31:46,818::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7992::DEBUG::2012-12-02 23:31:46,818::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7992::DEBUG::2012-12-02 23:31:46,819::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`73d81e1c-9b74-40f5-a80e-600da0f5401c`::Granted request >Thread-7992::DEBUG::2012-12-02 23:31:46,819::task::794::TaskManager.Task::(resourceAcquired) Task=`5c28b4e5-85fe-4e3a-b80e-f657d779614f`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7992::DEBUG::2012-12-02 23:31:46,819::task::957::TaskManager.Task::(_decref) Task=`5c28b4e5-85fe-4e3a-b80e-f657d779614f`::ref 1 aborting False >Thread-7992::DEBUG::2012-12-02 23:31:46,825::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7992::DEBUG::2012-12-02 23:31:46,826::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7992::DEBUG::2012-12-02 23:31:47,032::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7992::WARNING::2012-12-02 23:31:47,033::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7992::DEBUG::2012-12-02 23:31:47,034::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7992::WARNING::2012-12-02 23:31:47,064::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7992::INFO::2012-12-02 23:31:47,065::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7992::DEBUG::2012-12-02 23:31:47,065::task::1151::TaskManager.Task::(prepare) Task=`5c28b4e5-85fe-4e3a-b80e-f657d779614f`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-7992::DEBUG::2012-12-02 23:31:47,065::task::568::TaskManager.Task::(_updateState) Task=`5c28b4e5-85fe-4e3a-b80e-f657d779614f`::moving from state preparing -> state finished >Thread-7992::DEBUG::2012-12-02 23:31:47,065::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7992::DEBUG::2012-12-02 23:31:47,065::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7992::DEBUG::2012-12-02 23:31:47,066::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7992::DEBUG::2012-12-02 23:31:47,066::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7992::DEBUG::2012-12-02 23:31:47,066::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7992::DEBUG::2012-12-02 23:31:47,066::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7992::DEBUG::2012-12-02 23:31:47,066::task::957::TaskManager.Task::(_decref) Task=`5c28b4e5-85fe-4e3a-b80e-f657d779614f`::ref 0 aborting False >Thread-8059::DEBUG::2012-12-02 23:31:57,254::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8059::DEBUG::2012-12-02 23:31:57,255::task::568::TaskManager.Task::(_updateState) Task=`d7e57a3e-27b4-4776-975b-e65579233171`::moving from state init -> state preparing >Thread-8059::INFO::2012-12-02 23:31:57,255::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8059::INFO::2012-12-02 23:31:57,255::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8059::DEBUG::2012-12-02 23:31:57,255::task::1151::TaskManager.Task::(prepare) Task=`d7e57a3e-27b4-4776-975b-e65579233171`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8059::DEBUG::2012-12-02 23:31:57,255::task::568::TaskManager.Task::(_updateState) Task=`d7e57a3e-27b4-4776-975b-e65579233171`::moving from state preparing -> state finished >Thread-8059::DEBUG::2012-12-02 23:31:57,256::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8059::DEBUG::2012-12-02 23:31:57,256::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8059::DEBUG::2012-12-02 23:31:57,256::task::957::TaskManager.Task::(_decref) Task=`d7e57a3e-27b4-4776-975b-e65579233171`::ref 0 aborting False >Thread-8060::DEBUG::2012-12-02 23:31:57,399::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8060::DEBUG::2012-12-02 23:31:57,399::task::568::TaskManager.Task::(_updateState) Task=`786f2340-dafd-4fa7-bb73-777713aea4e3`::moving from state init -> state preparing >Thread-8060::INFO::2012-12-02 23:31:57,399::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8060::DEBUG::2012-12-02 23:31:57,400::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7f66ae17-69ba-4d3f-accf-caa4154c7301`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8060::DEBUG::2012-12-02 23:31:57,400::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8060::DEBUG::2012-12-02 23:31:57,400::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8060::DEBUG::2012-12-02 23:31:57,400::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7f66ae17-69ba-4d3f-accf-caa4154c7301`::Granted request >Thread-8060::DEBUG::2012-12-02 23:31:57,400::task::794::TaskManager.Task::(resourceAcquired) Task=`786f2340-dafd-4fa7-bb73-777713aea4e3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8060::DEBUG::2012-12-02 23:31:57,400::task::957::TaskManager.Task::(_decref) Task=`786f2340-dafd-4fa7-bb73-777713aea4e3`::ref 1 aborting False >Thread-8060::DEBUG::2012-12-02 23:31:57,406::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8060::DEBUG::2012-12-02 23:31:57,407::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8060::DEBUG::2012-12-02 23:31:57,612::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8060::WARNING::2012-12-02 23:31:57,613::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8060::DEBUG::2012-12-02 23:31:57,613::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8060::WARNING::2012-12-02 23:31:57,649::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8060::INFO::2012-12-02 23:31:57,649::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8060::DEBUG::2012-12-02 23:31:57,649::task::1151::TaskManager.Task::(prepare) Task=`786f2340-dafd-4fa7-bb73-777713aea4e3`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8060::DEBUG::2012-12-02 23:31:57,649::task::568::TaskManager.Task::(_updateState) Task=`786f2340-dafd-4fa7-bb73-777713aea4e3`::moving from state preparing -> state finished >Thread-8060::DEBUG::2012-12-02 23:31:57,649::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8060::DEBUG::2012-12-02 23:31:57,650::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8060::DEBUG::2012-12-02 23:31:57,650::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8060::DEBUG::2012-12-02 23:31:57,650::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8060::DEBUG::2012-12-02 23:31:57,650::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8060::DEBUG::2012-12-02 23:31:57,650::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8060::DEBUG::2012-12-02 23:31:57,650::task::957::TaskManager.Task::(_decref) Task=`786f2340-dafd-4fa7-bb73-777713aea4e3`::ref 0 aborting False >Thread-8122::DEBUG::2012-12-02 23:31:57,738::task::568::TaskManager.Task::(_updateState) Task=`c4a48c3a-cbcd-464a-a9d3-c2e92dab8fad`::moving from state init -> state preparing >Thread-8122::INFO::2012-12-02 23:31:57,738::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8122::INFO::2012-12-02 23:31:57,738::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0076060295105', 'lastCheck': 1354483909.839174, 'code': 0, 'valid': True}} >Thread-8122::DEBUG::2012-12-02 23:31:57,738::task::1151::TaskManager.Task::(prepare) Task=`c4a48c3a-cbcd-464a-a9d3-c2e92dab8fad`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0076060295105', 'lastCheck': 1354483909.839174, 'code': 0, 'valid': True}} >Thread-8122::DEBUG::2012-12-02 23:31:57,738::task::568::TaskManager.Task::(_updateState) Task=`c4a48c3a-cbcd-464a-a9d3-c2e92dab8fad`::moving from state preparing -> state finished >Thread-8122::DEBUG::2012-12-02 23:31:57,738::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8122::DEBUG::2012-12-02 23:31:57,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8122::DEBUG::2012-12-02 23:31:57,739::task::957::TaskManager.Task::(_decref) Task=`c4a48c3a-cbcd-464a-a9d3-c2e92dab8fad`::ref 0 aborting False >Thread-8128::DEBUG::2012-12-02 23:32:07,843::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8128::DEBUG::2012-12-02 23:32:07,844::task::568::TaskManager.Task::(_updateState) Task=`80d05b9d-8901-4804-9219-7bfaa0b317b3`::moving from state init -> state preparing >Thread-8128::INFO::2012-12-02 23:32:07,844::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8128::INFO::2012-12-02 23:32:07,844::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8128::DEBUG::2012-12-02 23:32:07,844::task::1151::TaskManager.Task::(prepare) Task=`80d05b9d-8901-4804-9219-7bfaa0b317b3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8128::DEBUG::2012-12-02 23:32:07,844::task::568::TaskManager.Task::(_updateState) Task=`80d05b9d-8901-4804-9219-7bfaa0b317b3`::moving from state preparing -> state finished >Thread-8128::DEBUG::2012-12-02 23:32:07,844::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8128::DEBUG::2012-12-02 23:32:07,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8128::DEBUG::2012-12-02 23:32:07,845::task::957::TaskManager.Task::(_decref) Task=`80d05b9d-8901-4804-9219-7bfaa0b317b3`::ref 0 aborting False >Thread-8129::DEBUG::2012-12-02 23:32:07,989::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8129::DEBUG::2012-12-02 23:32:07,990::task::568::TaskManager.Task::(_updateState) Task=`31d62f07-4439-4f37-9f90-48ed016804db`::moving from state init -> state preparing >Thread-8129::INFO::2012-12-02 23:32:07,990::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8129::DEBUG::2012-12-02 23:32:07,990::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8f4215c0-bd0e-494c-a2c0-48f97998fc79`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8129::DEBUG::2012-12-02 23:32:07,990::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8129::DEBUG::2012-12-02 23:32:07,990::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8129::DEBUG::2012-12-02 23:32:07,991::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8f4215c0-bd0e-494c-a2c0-48f97998fc79`::Granted request >Thread-8129::DEBUG::2012-12-02 23:32:07,991::task::794::TaskManager.Task::(resourceAcquired) Task=`31d62f07-4439-4f37-9f90-48ed016804db`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8129::DEBUG::2012-12-02 23:32:07,991::task::957::TaskManager.Task::(_decref) Task=`31d62f07-4439-4f37-9f90-48ed016804db`::ref 1 aborting False >Thread-8129::DEBUG::2012-12-02 23:32:07,997::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8129::DEBUG::2012-12-02 23:32:07,997::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8129::DEBUG::2012-12-02 23:32:08,204::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8129::WARNING::2012-12-02 23:32:08,206::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8129::DEBUG::2012-12-02 23:32:08,206::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8129::WARNING::2012-12-02 23:32:08,237::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8129::INFO::2012-12-02 23:32:08,237::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8129::DEBUG::2012-12-02 23:32:08,237::task::1151::TaskManager.Task::(prepare) Task=`31d62f07-4439-4f37-9f90-48ed016804db`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8129::DEBUG::2012-12-02 23:32:08,237::task::568::TaskManager.Task::(_updateState) Task=`31d62f07-4439-4f37-9f90-48ed016804db`::moving from state preparing -> state finished >Thread-8129::DEBUG::2012-12-02 23:32:08,237::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8129::DEBUG::2012-12-02 23:32:08,238::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8129::DEBUG::2012-12-02 23:32:08,238::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8129::DEBUG::2012-12-02 23:32:08,238::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8129::DEBUG::2012-12-02 23:32:08,238::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8129::DEBUG::2012-12-02 23:32:08,238::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8129::DEBUG::2012-12-02 23:32:08,238::task::957::TaskManager.Task::(_decref) Task=`31d62f07-4439-4f37-9f90-48ed016804db`::ref 0 aborting False >Thread-8191::DEBUG::2012-12-02 23:32:08,688::task::568::TaskManager.Task::(_updateState) Task=`734726c3-9065-4586-8935-c78ef7135b7d`::moving from state init -> state preparing >Thread-8191::INFO::2012-12-02 23:32:08,688::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8191::INFO::2012-12-02 23:32:08,689::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00893616676331', 'lastCheck': 1354483919.863282, 'code': 0, 'valid': True}} >Thread-8191::DEBUG::2012-12-02 23:32:08,689::task::1151::TaskManager.Task::(prepare) Task=`734726c3-9065-4586-8935-c78ef7135b7d`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00893616676331', 'lastCheck': 1354483919.863282, 'code': 0, 'valid': True}} >Thread-8191::DEBUG::2012-12-02 23:32:08,689::task::568::TaskManager.Task::(_updateState) Task=`734726c3-9065-4586-8935-c78ef7135b7d`::moving from state preparing -> state finished >Thread-8191::DEBUG::2012-12-02 23:32:08,689::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8191::DEBUG::2012-12-02 23:32:08,689::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8191::DEBUG::2012-12-02 23:32:08,689::task::957::TaskManager.Task::(_decref) Task=`734726c3-9065-4586-8935-c78ef7135b7d`::ref 0 aborting False >Thread-8197::DEBUG::2012-12-02 23:32:18,488::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8197::DEBUG::2012-12-02 23:32:18,488::task::568::TaskManager.Task::(_updateState) Task=`8b7e51fa-46e4-4191-9b9b-cd8b5a804592`::moving from state init -> state preparing >Thread-8197::INFO::2012-12-02 23:32:18,489::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8197::INFO::2012-12-02 23:32:18,489::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8197::DEBUG::2012-12-02 23:32:18,489::task::1151::TaskManager.Task::(prepare) Task=`8b7e51fa-46e4-4191-9b9b-cd8b5a804592`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8197::DEBUG::2012-12-02 23:32:18,489::task::568::TaskManager.Task::(_updateState) Task=`8b7e51fa-46e4-4191-9b9b-cd8b5a804592`::moving from state preparing -> state finished >Thread-8197::DEBUG::2012-12-02 23:32:18,489::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8197::DEBUG::2012-12-02 23:32:18,489::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8197::DEBUG::2012-12-02 23:32:18,489::task::957::TaskManager.Task::(_decref) Task=`8b7e51fa-46e4-4191-9b9b-cd8b5a804592`::ref 0 aborting False >Thread-8198::DEBUG::2012-12-02 23:32:18,627::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8198::DEBUG::2012-12-02 23:32:18,628::task::568::TaskManager.Task::(_updateState) Task=`4e5164b1-8af1-4e63-8a56-c066a33f8aa3`::moving from state init -> state preparing >Thread-8198::INFO::2012-12-02 23:32:18,628::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8198::DEBUG::2012-12-02 23:32:18,628::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`29c7f347-db49-4a24-9fd3-80e394c82882`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8198::DEBUG::2012-12-02 23:32:18,628::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8198::DEBUG::2012-12-02 23:32:18,628::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8198::DEBUG::2012-12-02 23:32:18,628::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`29c7f347-db49-4a24-9fd3-80e394c82882`::Granted request >Thread-8198::DEBUG::2012-12-02 23:32:18,629::task::794::TaskManager.Task::(resourceAcquired) Task=`4e5164b1-8af1-4e63-8a56-c066a33f8aa3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8198::DEBUG::2012-12-02 23:32:18,629::task::957::TaskManager.Task::(_decref) Task=`4e5164b1-8af1-4e63-8a56-c066a33f8aa3`::ref 1 aborting False >Thread-8198::DEBUG::2012-12-02 23:32:18,635::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8198::DEBUG::2012-12-02 23:32:18,636::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8198::DEBUG::2012-12-02 23:32:18,842::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8198::WARNING::2012-12-02 23:32:18,844::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8198::DEBUG::2012-12-02 23:32:18,844::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8198::WARNING::2012-12-02 23:32:18,874::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8198::INFO::2012-12-02 23:32:18,874::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8198::DEBUG::2012-12-02 23:32:18,874::task::1151::TaskManager.Task::(prepare) Task=`4e5164b1-8af1-4e63-8a56-c066a33f8aa3`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8198::DEBUG::2012-12-02 23:32:18,874::task::568::TaskManager.Task::(_updateState) Task=`4e5164b1-8af1-4e63-8a56-c066a33f8aa3`::moving from state preparing -> state finished >Thread-8198::DEBUG::2012-12-02 23:32:18,875::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8198::DEBUG::2012-12-02 23:32:18,875::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8198::DEBUG::2012-12-02 23:32:18,875::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8198::DEBUG::2012-12-02 23:32:18,875::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8198::DEBUG::2012-12-02 23:32:18,875::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8198::DEBUG::2012-12-02 23:32:18,875::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8198::DEBUG::2012-12-02 23:32:18,876::task::957::TaskManager.Task::(_decref) Task=`4e5164b1-8af1-4e63-8a56-c066a33f8aa3`::ref 0 aborting False >Thread-8260::DEBUG::2012-12-02 23:32:19,669::task::568::TaskManager.Task::(_updateState) Task=`b0e7d07a-07f0-4c11-9b96-4bb2cf8d78f5`::moving from state init -> state preparing >Thread-8260::INFO::2012-12-02 23:32:19,669::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8260::INFO::2012-12-02 23:32:19,670::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0066511631012', 'lastCheck': 1354483929.881027, 'code': 0, 'valid': True}} >Thread-8260::DEBUG::2012-12-02 23:32:19,670::task::1151::TaskManager.Task::(prepare) Task=`b0e7d07a-07f0-4c11-9b96-4bb2cf8d78f5`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0066511631012', 'lastCheck': 1354483929.881027, 'code': 0, 'valid': True}} >Thread-8260::DEBUG::2012-12-02 23:32:19,670::task::568::TaskManager.Task::(_updateState) Task=`b0e7d07a-07f0-4c11-9b96-4bb2cf8d78f5`::moving from state preparing -> state finished >Thread-8260::DEBUG::2012-12-02 23:32:19,670::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8260::DEBUG::2012-12-02 23:32:19,670::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8260::DEBUG::2012-12-02 23:32:19,670::task::957::TaskManager.Task::(_decref) Task=`b0e7d07a-07f0-4c11-9b96-4bb2cf8d78f5`::ref 0 aborting False >Thread-8266::DEBUG::2012-12-02 23:32:29,122::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8266::DEBUG::2012-12-02 23:32:29,122::task::568::TaskManager.Task::(_updateState) Task=`fd818546-fca7-44c4-81f2-ff62ea3726af`::moving from state init -> state preparing >Thread-8266::INFO::2012-12-02 23:32:29,122::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8266::INFO::2012-12-02 23:32:29,122::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8266::DEBUG::2012-12-02 23:32:29,123::task::1151::TaskManager.Task::(prepare) Task=`fd818546-fca7-44c4-81f2-ff62ea3726af`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8266::DEBUG::2012-12-02 23:32:29,123::task::568::TaskManager.Task::(_updateState) Task=`fd818546-fca7-44c4-81f2-ff62ea3726af`::moving from state preparing -> state finished >Thread-8266::DEBUG::2012-12-02 23:32:29,123::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8266::DEBUG::2012-12-02 23:32:29,123::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8266::DEBUG::2012-12-02 23:32:29,123::task::957::TaskManager.Task::(_decref) Task=`fd818546-fca7-44c4-81f2-ff62ea3726af`::ref 0 aborting False >Thread-8267::DEBUG::2012-12-02 23:32:29,265::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8267::DEBUG::2012-12-02 23:32:29,265::task::568::TaskManager.Task::(_updateState) Task=`545e3774-839d-4518-9ccd-b45d26681981`::moving from state init -> state preparing >Thread-8267::INFO::2012-12-02 23:32:29,265::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8267::DEBUG::2012-12-02 23:32:29,266::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`40bebb29-940e-453d-98b2-2204d6b5df8f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8267::DEBUG::2012-12-02 23:32:29,266::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8267::DEBUG::2012-12-02 23:32:29,266::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8267::DEBUG::2012-12-02 23:32:29,266::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`40bebb29-940e-453d-98b2-2204d6b5df8f`::Granted request >Thread-8267::DEBUG::2012-12-02 23:32:29,266::task::794::TaskManager.Task::(resourceAcquired) Task=`545e3774-839d-4518-9ccd-b45d26681981`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8267::DEBUG::2012-12-02 23:32:29,266::task::957::TaskManager.Task::(_decref) Task=`545e3774-839d-4518-9ccd-b45d26681981`::ref 1 aborting False >Thread-8267::DEBUG::2012-12-02 23:32:29,273::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8267::DEBUG::2012-12-02 23:32:29,273::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8267::DEBUG::2012-12-02 23:32:29,481::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8267::WARNING::2012-12-02 23:32:29,483::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8267::DEBUG::2012-12-02 23:32:29,483::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8267::WARNING::2012-12-02 23:32:29,513::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8267::INFO::2012-12-02 23:32:29,513::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8267::DEBUG::2012-12-02 23:32:29,513::task::1151::TaskManager.Task::(prepare) Task=`545e3774-839d-4518-9ccd-b45d26681981`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8267::DEBUG::2012-12-02 23:32:29,514::task::568::TaskManager.Task::(_updateState) Task=`545e3774-839d-4518-9ccd-b45d26681981`::moving from state preparing -> state finished >Thread-8267::DEBUG::2012-12-02 23:32:29,514::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8267::DEBUG::2012-12-02 23:32:29,514::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8267::DEBUG::2012-12-02 23:32:29,514::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8267::DEBUG::2012-12-02 23:32:29,514::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8267::DEBUG::2012-12-02 23:32:29,514::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8267::DEBUG::2012-12-02 23:32:29,515::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8267::DEBUG::2012-12-02 23:32:29,515::task::957::TaskManager.Task::(_decref) Task=`545e3774-839d-4518-9ccd-b45d26681981`::ref 0 aborting False >Thread-8329::DEBUG::2012-12-02 23:32:30,603::task::568::TaskManager.Task::(_updateState) Task=`de582509-9717-4ce3-9c18-7eb84001a869`::moving from state init -> state preparing >Thread-8329::INFO::2012-12-02 23:32:30,603::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8329::INFO::2012-12-02 23:32:30,603::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00772404670715', 'lastCheck': 1354483949.915617, 'code': 0, 'valid': True}} >Thread-8329::DEBUG::2012-12-02 23:32:30,603::task::1151::TaskManager.Task::(prepare) Task=`de582509-9717-4ce3-9c18-7eb84001a869`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00772404670715', 'lastCheck': 1354483949.915617, 'code': 0, 'valid': True}} >Thread-8329::DEBUG::2012-12-02 23:32:30,604::task::568::TaskManager.Task::(_updateState) Task=`de582509-9717-4ce3-9c18-7eb84001a869`::moving from state preparing -> state finished >Thread-8329::DEBUG::2012-12-02 23:32:30,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8329::DEBUG::2012-12-02 23:32:30,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8329::DEBUG::2012-12-02 23:32:30,604::task::957::TaskManager.Task::(_decref) Task=`de582509-9717-4ce3-9c18-7eb84001a869`::ref 0 aborting False >Thread-8335::DEBUG::2012-12-02 23:32:39,693::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8335::DEBUG::2012-12-02 23:32:39,693::task::568::TaskManager.Task::(_updateState) Task=`be6308e4-a24e-4c97-bb19-abdc741856cc`::moving from state init -> state preparing >Thread-8335::INFO::2012-12-02 23:32:39,693::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8335::INFO::2012-12-02 23:32:39,694::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8335::DEBUG::2012-12-02 23:32:39,694::task::1151::TaskManager.Task::(prepare) Task=`be6308e4-a24e-4c97-bb19-abdc741856cc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8335::DEBUG::2012-12-02 23:32:39,694::task::568::TaskManager.Task::(_updateState) Task=`be6308e4-a24e-4c97-bb19-abdc741856cc`::moving from state preparing -> state finished >Thread-8335::DEBUG::2012-12-02 23:32:39,694::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8335::DEBUG::2012-12-02 23:32:39,694::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8335::DEBUG::2012-12-02 23:32:39,694::task::957::TaskManager.Task::(_decref) Task=`be6308e4-a24e-4c97-bb19-abdc741856cc`::ref 0 aborting False >Thread-8336::DEBUG::2012-12-02 23:32:39,835::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8336::DEBUG::2012-12-02 23:32:39,835::task::568::TaskManager.Task::(_updateState) Task=`d80aecf0-590e-4e98-8f5b-8d3dd78f0dd0`::moving from state init -> state preparing >Thread-8336::INFO::2012-12-02 23:32:39,835::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8336::DEBUG::2012-12-02 23:32:39,835::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`59a33304-f9a5-4f98-bb9f-94e5207966a7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8336::DEBUG::2012-12-02 23:32:39,835::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8336::DEBUG::2012-12-02 23:32:39,836::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8336::DEBUG::2012-12-02 23:32:39,836::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`59a33304-f9a5-4f98-bb9f-94e5207966a7`::Granted request >Thread-8336::DEBUG::2012-12-02 23:32:39,836::task::794::TaskManager.Task::(resourceAcquired) Task=`d80aecf0-590e-4e98-8f5b-8d3dd78f0dd0`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8336::DEBUG::2012-12-02 23:32:39,836::task::957::TaskManager.Task::(_decref) Task=`d80aecf0-590e-4e98-8f5b-8d3dd78f0dd0`::ref 1 aborting False >Thread-8336::DEBUG::2012-12-02 23:32:39,843::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8336::DEBUG::2012-12-02 23:32:39,843::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8336::DEBUG::2012-12-02 23:32:40,051::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8336::WARNING::2012-12-02 23:32:40,052::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8336::DEBUG::2012-12-02 23:32:40,052::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8336::WARNING::2012-12-02 23:32:40,082::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8336::INFO::2012-12-02 23:32:40,083::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8336::DEBUG::2012-12-02 23:32:40,085::task::1151::TaskManager.Task::(prepare) Task=`d80aecf0-590e-4e98-8f5b-8d3dd78f0dd0`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8336::DEBUG::2012-12-02 23:32:40,085::task::568::TaskManager.Task::(_updateState) Task=`d80aecf0-590e-4e98-8f5b-8d3dd78f0dd0`::moving from state preparing -> state finished >Thread-8336::DEBUG::2012-12-02 23:32:40,087::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8336::DEBUG::2012-12-02 23:32:40,088::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8336::DEBUG::2012-12-02 23:32:40,090::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8336::DEBUG::2012-12-02 23:32:40,090::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8336::DEBUG::2012-12-02 23:32:40,090::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8336::DEBUG::2012-12-02 23:32:40,091::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8336::DEBUG::2012-12-02 23:32:40,091::task::957::TaskManager.Task::(_decref) Task=`d80aecf0-590e-4e98-8f5b-8d3dd78f0dd0`::ref 0 aborting False >Thread-8398::DEBUG::2012-12-02 23:32:41,537::task::568::TaskManager.Task::(_updateState) Task=`5cb8e2ed-e9c1-4d48-94da-9d320cd16084`::moving from state init -> state preparing >Thread-8398::INFO::2012-12-02 23:32:41,537::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8398::INFO::2012-12-02 23:32:41,537::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00784492492676', 'lastCheck': 1354483960.102212, 'code': 0, 'valid': True}} >Thread-8398::DEBUG::2012-12-02 23:32:41,537::task::1151::TaskManager.Task::(prepare) Task=`5cb8e2ed-e9c1-4d48-94da-9d320cd16084`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00784492492676', 'lastCheck': 1354483960.102212, 'code': 0, 'valid': True}} >Thread-8398::DEBUG::2012-12-02 23:32:41,538::task::568::TaskManager.Task::(_updateState) Task=`5cb8e2ed-e9c1-4d48-94da-9d320cd16084`::moving from state preparing -> state finished >Thread-8398::DEBUG::2012-12-02 23:32:41,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8398::DEBUG::2012-12-02 23:32:41,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8398::DEBUG::2012-12-02 23:32:41,538::task::957::TaskManager.Task::(_decref) Task=`5cb8e2ed-e9c1-4d48-94da-9d320cd16084`::ref 0 aborting False >Thread-8403::DEBUG::2012-12-02 23:32:50,298::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8403::DEBUG::2012-12-02 23:32:50,298::task::568::TaskManager.Task::(_updateState) Task=`ced9a43c-5e40-46d2-bf29-c8b812c8ba37`::moving from state init -> state preparing >Thread-8403::INFO::2012-12-02 23:32:50,298::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8403::INFO::2012-12-02 23:32:50,298::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8403::DEBUG::2012-12-02 23:32:50,299::task::1151::TaskManager.Task::(prepare) Task=`ced9a43c-5e40-46d2-bf29-c8b812c8ba37`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8403::DEBUG::2012-12-02 23:32:50,299::task::568::TaskManager.Task::(_updateState) Task=`ced9a43c-5e40-46d2-bf29-c8b812c8ba37`::moving from state preparing -> state finished >Thread-8403::DEBUG::2012-12-02 23:32:50,299::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8403::DEBUG::2012-12-02 23:32:50,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8403::DEBUG::2012-12-02 23:32:50,299::task::957::TaskManager.Task::(_decref) Task=`ced9a43c-5e40-46d2-bf29-c8b812c8ba37`::ref 0 aborting False >Thread-8405::DEBUG::2012-12-02 23:32:50,438::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8405::DEBUG::2012-12-02 23:32:50,438::task::568::TaskManager.Task::(_updateState) Task=`05036ddb-cd41-4af0-9a37-0978e26d1145`::moving from state init -> state preparing >Thread-8405::INFO::2012-12-02 23:32:50,438::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8405::DEBUG::2012-12-02 23:32:50,438::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8f943b48-c94f-4064-932f-d2183479ae48`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8405::DEBUG::2012-12-02 23:32:50,438::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8405::DEBUG::2012-12-02 23:32:50,439::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8405::DEBUG::2012-12-02 23:32:50,439::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8f943b48-c94f-4064-932f-d2183479ae48`::Granted request >Thread-8405::DEBUG::2012-12-02 23:32:50,439::task::794::TaskManager.Task::(resourceAcquired) Task=`05036ddb-cd41-4af0-9a37-0978e26d1145`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8405::DEBUG::2012-12-02 23:32:50,439::task::957::TaskManager.Task::(_decref) Task=`05036ddb-cd41-4af0-9a37-0978e26d1145`::ref 1 aborting False >Thread-8405::DEBUG::2012-12-02 23:32:50,446::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8405::DEBUG::2012-12-02 23:32:50,446::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8405::DEBUG::2012-12-02 23:32:50,654::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8405::WARNING::2012-12-02 23:32:50,655::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8405::DEBUG::2012-12-02 23:32:50,655::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8405::WARNING::2012-12-02 23:32:50,687::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8405::INFO::2012-12-02 23:32:50,687::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8405::DEBUG::2012-12-02 23:32:50,687::task::1151::TaskManager.Task::(prepare) Task=`05036ddb-cd41-4af0-9a37-0978e26d1145`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8405::DEBUG::2012-12-02 23:32:50,687::task::568::TaskManager.Task::(_updateState) Task=`05036ddb-cd41-4af0-9a37-0978e26d1145`::moving from state preparing -> state finished >Thread-8405::DEBUG::2012-12-02 23:32:50,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8405::DEBUG::2012-12-02 23:32:50,688::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8405::DEBUG::2012-12-02 23:32:50,688::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8405::DEBUG::2012-12-02 23:32:50,688::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8405::DEBUG::2012-12-02 23:32:50,688::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8405::DEBUG::2012-12-02 23:32:50,688::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8405::DEBUG::2012-12-02 23:32:50,688::task::957::TaskManager.Task::(_decref) Task=`05036ddb-cd41-4af0-9a37-0978e26d1145`::ref 0 aborting False >Thread-8467::DEBUG::2012-12-02 23:32:52,482::task::568::TaskManager.Task::(_updateState) Task=`b5653da7-dd5e-403f-8cbc-afa371bf1feb`::moving from state init -> state preparing >Thread-8467::INFO::2012-12-02 23:32:52,482::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8467::INFO::2012-12-02 23:32:52,483::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0116350650787', 'lastCheck': 1354483970.125561, 'code': 0, 'valid': True}} >Thread-8467::DEBUG::2012-12-02 23:32:52,483::task::1151::TaskManager.Task::(prepare) Task=`b5653da7-dd5e-403f-8cbc-afa371bf1feb`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0116350650787', 'lastCheck': 1354483970.125561, 'code': 0, 'valid': True}} >Thread-8467::DEBUG::2012-12-02 23:32:52,483::task::568::TaskManager.Task::(_updateState) Task=`b5653da7-dd5e-403f-8cbc-afa371bf1feb`::moving from state preparing -> state finished >Thread-8467::DEBUG::2012-12-02 23:32:52,483::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8467::DEBUG::2012-12-02 23:32:52,483::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8467::DEBUG::2012-12-02 23:32:52,483::task::957::TaskManager.Task::(_decref) Task=`b5653da7-dd5e-403f-8cbc-afa371bf1feb`::ref 0 aborting False >Thread-8472::DEBUG::2012-12-02 23:33:00,907::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8472::DEBUG::2012-12-02 23:33:00,907::task::568::TaskManager.Task::(_updateState) Task=`38d5f0c0-7ba9-4344-8cce-7a9734bcb79a`::moving from state init -> state preparing >Thread-8472::INFO::2012-12-02 23:33:00,907::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8472::INFO::2012-12-02 23:33:00,908::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8472::DEBUG::2012-12-02 23:33:00,908::task::1151::TaskManager.Task::(prepare) Task=`38d5f0c0-7ba9-4344-8cce-7a9734bcb79a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8472::DEBUG::2012-12-02 23:33:00,908::task::568::TaskManager.Task::(_updateState) Task=`38d5f0c0-7ba9-4344-8cce-7a9734bcb79a`::moving from state preparing -> state finished >Thread-8472::DEBUG::2012-12-02 23:33:00,908::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8472::DEBUG::2012-12-02 23:33:00,908::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8472::DEBUG::2012-12-02 23:33:00,908::task::957::TaskManager.Task::(_decref) Task=`38d5f0c0-7ba9-4344-8cce-7a9734bcb79a`::ref 0 aborting False >Thread-8473::DEBUG::2012-12-02 23:33:01,051::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8473::DEBUG::2012-12-02 23:33:01,052::task::568::TaskManager.Task::(_updateState) Task=`3a4cda68-a8f5-4ee4-9026-3e1f6424f23d`::moving from state init -> state preparing >Thread-8473::INFO::2012-12-02 23:33:01,052::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8473::DEBUG::2012-12-02 23:33:01,052::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f87c3a49-7993-4e14-8734-b264ac22d789`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8473::DEBUG::2012-12-02 23:33:01,052::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8473::DEBUG::2012-12-02 23:33:01,052::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8473::DEBUG::2012-12-02 23:33:01,053::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f87c3a49-7993-4e14-8734-b264ac22d789`::Granted request >Thread-8473::DEBUG::2012-12-02 23:33:01,053::task::794::TaskManager.Task::(resourceAcquired) Task=`3a4cda68-a8f5-4ee4-9026-3e1f6424f23d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8473::DEBUG::2012-12-02 23:33:01,053::task::957::TaskManager.Task::(_decref) Task=`3a4cda68-a8f5-4ee4-9026-3e1f6424f23d`::ref 1 aborting False >Thread-8473::DEBUG::2012-12-02 23:33:01,058::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8473::DEBUG::2012-12-02 23:33:01,058::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8473::DEBUG::2012-12-02 23:33:01,265::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8473::WARNING::2012-12-02 23:33:01,266::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8473::DEBUG::2012-12-02 23:33:01,267::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8473::WARNING::2012-12-02 23:33:01,298::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8473::INFO::2012-12-02 23:33:01,298::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8473::DEBUG::2012-12-02 23:33:01,298::task::1151::TaskManager.Task::(prepare) Task=`3a4cda68-a8f5-4ee4-9026-3e1f6424f23d`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8473::DEBUG::2012-12-02 23:33:01,299::task::568::TaskManager.Task::(_updateState) Task=`3a4cda68-a8f5-4ee4-9026-3e1f6424f23d`::moving from state preparing -> state finished >Thread-8473::DEBUG::2012-12-02 23:33:01,299::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8473::DEBUG::2012-12-02 23:33:01,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8473::DEBUG::2012-12-02 23:33:01,299::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8473::DEBUG::2012-12-02 23:33:01,299::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8473::DEBUG::2012-12-02 23:33:01,299::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8473::DEBUG::2012-12-02 23:33:01,299::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8473::DEBUG::2012-12-02 23:33:01,300::task::957::TaskManager.Task::(_decref) Task=`3a4cda68-a8f5-4ee4-9026-3e1f6424f23d`::ref 0 aborting False >Thread-8536::DEBUG::2012-12-02 23:33:03,385::task::568::TaskManager.Task::(_updateState) Task=`9dd7b226-8cbb-4c2a-8964-199ba218c0a0`::moving from state init -> state preparing >Thread-8536::INFO::2012-12-02 23:33:03,385::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8536::INFO::2012-12-02 23:33:03,386::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00914692878723', 'lastCheck': 1354483980.148988, 'code': 0, 'valid': True}} >Thread-8536::DEBUG::2012-12-02 23:33:03,386::task::1151::TaskManager.Task::(prepare) Task=`9dd7b226-8cbb-4c2a-8964-199ba218c0a0`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00914692878723', 'lastCheck': 1354483980.148988, 'code': 0, 'valid': True}} >Thread-8536::DEBUG::2012-12-02 23:33:03,386::task::568::TaskManager.Task::(_updateState) Task=`9dd7b226-8cbb-4c2a-8964-199ba218c0a0`::moving from state preparing -> state finished >Thread-8536::DEBUG::2012-12-02 23:33:03,386::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8536::DEBUG::2012-12-02 23:33:03,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8536::DEBUG::2012-12-02 23:33:03,386::task::957::TaskManager.Task::(_decref) Task=`9dd7b226-8cbb-4c2a-8964-199ba218c0a0`::ref 0 aborting False >Thread-8541::DEBUG::2012-12-02 23:33:11,509::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8541::DEBUG::2012-12-02 23:33:11,509::task::568::TaskManager.Task::(_updateState) Task=`8e834992-2f8a-4fa8-bbef-172b1b764b25`::moving from state init -> state preparing >Thread-8541::INFO::2012-12-02 23:33:11,509::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8541::INFO::2012-12-02 23:33:11,509::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8541::DEBUG::2012-12-02 23:33:11,510::task::1151::TaskManager.Task::(prepare) Task=`8e834992-2f8a-4fa8-bbef-172b1b764b25`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8541::DEBUG::2012-12-02 23:33:11,510::task::568::TaskManager.Task::(_updateState) Task=`8e834992-2f8a-4fa8-bbef-172b1b764b25`::moving from state preparing -> state finished >Thread-8541::DEBUG::2012-12-02 23:33:11,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8541::DEBUG::2012-12-02 23:33:11,510::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8541::DEBUG::2012-12-02 23:33:11,510::task::957::TaskManager.Task::(_decref) Task=`8e834992-2f8a-4fa8-bbef-172b1b764b25`::ref 0 aborting False >Thread-8542::DEBUG::2012-12-02 23:33:11,649::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8542::DEBUG::2012-12-02 23:33:11,650::task::568::TaskManager.Task::(_updateState) Task=`32d6ba3d-fc26-4bdb-a808-d3b44c5775a5`::moving from state init -> state preparing >Thread-8542::INFO::2012-12-02 23:33:11,650::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8542::DEBUG::2012-12-02 23:33:11,650::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a7b51c9-7ae5-4eb0-ab8f-9b3eda045553`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8542::DEBUG::2012-12-02 23:33:11,650::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8542::DEBUG::2012-12-02 23:33:11,650::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8542::DEBUG::2012-12-02 23:33:11,651::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a7b51c9-7ae5-4eb0-ab8f-9b3eda045553`::Granted request >Thread-8542::DEBUG::2012-12-02 23:33:11,651::task::794::TaskManager.Task::(resourceAcquired) Task=`32d6ba3d-fc26-4bdb-a808-d3b44c5775a5`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8542::DEBUG::2012-12-02 23:33:11,651::task::957::TaskManager.Task::(_decref) Task=`32d6ba3d-fc26-4bdb-a808-d3b44c5775a5`::ref 1 aborting False >Thread-8542::DEBUG::2012-12-02 23:33:11,657::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8542::DEBUG::2012-12-02 23:33:11,657::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8542::DEBUG::2012-12-02 23:33:11,864::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8542::WARNING::2012-12-02 23:33:11,865::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8542::DEBUG::2012-12-02 23:33:11,865::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8542::WARNING::2012-12-02 23:33:11,895::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8542::INFO::2012-12-02 23:33:11,896::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8542::DEBUG::2012-12-02 23:33:11,896::task::1151::TaskManager.Task::(prepare) Task=`32d6ba3d-fc26-4bdb-a808-d3b44c5775a5`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8542::DEBUG::2012-12-02 23:33:11,896::task::568::TaskManager.Task::(_updateState) Task=`32d6ba3d-fc26-4bdb-a808-d3b44c5775a5`::moving from state preparing -> state finished >Thread-8542::DEBUG::2012-12-02 23:33:11,896::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8542::DEBUG::2012-12-02 23:33:11,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8542::DEBUG::2012-12-02 23:33:11,897::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8542::DEBUG::2012-12-02 23:33:11,897::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8542::DEBUG::2012-12-02 23:33:11,897::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8542::DEBUG::2012-12-02 23:33:11,897::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8542::DEBUG::2012-12-02 23:33:11,897::task::957::TaskManager.Task::(_decref) Task=`32d6ba3d-fc26-4bdb-a808-d3b44c5775a5`::ref 0 aborting False >Thread-8605::DEBUG::2012-12-02 23:33:14,334::task::568::TaskManager.Task::(_updateState) Task=`ec0d5536-df80-4f4d-b66e-678f363aedae`::moving from state init -> state preparing >Thread-8605::INFO::2012-12-02 23:33:14,334::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8605::INFO::2012-12-02 23:33:14,334::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00740480422974', 'lastCheck': 1354483990.1678519, 'code': 0, 'valid': True}} >Thread-8605::DEBUG::2012-12-02 23:33:14,334::task::1151::TaskManager.Task::(prepare) Task=`ec0d5536-df80-4f4d-b66e-678f363aedae`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00740480422974', 'lastCheck': 1354483990.1678519, 'code': 0, 'valid': True}} >Thread-8605::DEBUG::2012-12-02 23:33:14,334::task::568::TaskManager.Task::(_updateState) Task=`ec0d5536-df80-4f4d-b66e-678f363aedae`::moving from state preparing -> state finished >Thread-8605::DEBUG::2012-12-02 23:33:14,335::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8605::DEBUG::2012-12-02 23:33:14,335::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8605::DEBUG::2012-12-02 23:33:14,335::task::957::TaskManager.Task::(_decref) Task=`ec0d5536-df80-4f4d-b66e-678f363aedae`::ref 0 aborting False >Thread-8610::DEBUG::2012-12-02 23:33:22,097::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8610::DEBUG::2012-12-02 23:33:22,097::task::568::TaskManager.Task::(_updateState) Task=`f79daf15-6184-4e24-ba09-817e195ce3a3`::moving from state init -> state preparing >Thread-8610::INFO::2012-12-02 23:33:22,097::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8610::INFO::2012-12-02 23:33:22,097::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8610::DEBUG::2012-12-02 23:33:22,097::task::1151::TaskManager.Task::(prepare) Task=`f79daf15-6184-4e24-ba09-817e195ce3a3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8610::DEBUG::2012-12-02 23:33:22,098::task::568::TaskManager.Task::(_updateState) Task=`f79daf15-6184-4e24-ba09-817e195ce3a3`::moving from state preparing -> state finished >Thread-8610::DEBUG::2012-12-02 23:33:22,098::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8610::DEBUG::2012-12-02 23:33:22,098::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8610::DEBUG::2012-12-02 23:33:22,098::task::957::TaskManager.Task::(_decref) Task=`f79daf15-6184-4e24-ba09-817e195ce3a3`::ref 0 aborting False >Thread-8611::DEBUG::2012-12-02 23:33:22,236::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8611::DEBUG::2012-12-02 23:33:22,237::task::568::TaskManager.Task::(_updateState) Task=`ecee47a0-db93-48b1-b4da-91b0931c240f`::moving from state init -> state preparing >Thread-8611::INFO::2012-12-02 23:33:22,237::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8611::DEBUG::2012-12-02 23:33:22,237::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`45657871-b697-400f-938c-1835ec6ae799`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8611::DEBUG::2012-12-02 23:33:22,237::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8611::DEBUG::2012-12-02 23:33:22,237::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8611::DEBUG::2012-12-02 23:33:22,237::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`45657871-b697-400f-938c-1835ec6ae799`::Granted request >Thread-8611::DEBUG::2012-12-02 23:33:22,238::task::794::TaskManager.Task::(resourceAcquired) Task=`ecee47a0-db93-48b1-b4da-91b0931c240f`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8611::DEBUG::2012-12-02 23:33:22,238::task::957::TaskManager.Task::(_decref) Task=`ecee47a0-db93-48b1-b4da-91b0931c240f`::ref 1 aborting False >Thread-8611::DEBUG::2012-12-02 23:33:22,244::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8611::DEBUG::2012-12-02 23:33:22,245::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8611::DEBUG::2012-12-02 23:33:22,453::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8611::WARNING::2012-12-02 23:33:22,454::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8611::DEBUG::2012-12-02 23:33:22,455::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8611::WARNING::2012-12-02 23:33:22,483::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8611::INFO::2012-12-02 23:33:22,483::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8611::DEBUG::2012-12-02 23:33:22,484::task::1151::TaskManager.Task::(prepare) Task=`ecee47a0-db93-48b1-b4da-91b0931c240f`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8611::DEBUG::2012-12-02 23:33:22,484::task::568::TaskManager.Task::(_updateState) Task=`ecee47a0-db93-48b1-b4da-91b0931c240f`::moving from state preparing -> state finished >Thread-8611::DEBUG::2012-12-02 23:33:22,484::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8611::DEBUG::2012-12-02 23:33:22,484::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8611::DEBUG::2012-12-02 23:33:22,484::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8611::DEBUG::2012-12-02 23:33:22,485::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8611::DEBUG::2012-12-02 23:33:22,485::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8611::DEBUG::2012-12-02 23:33:22,485::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8611::DEBUG::2012-12-02 23:33:22,485::task::957::TaskManager.Task::(_decref) Task=`ecee47a0-db93-48b1-b4da-91b0931c240f`::ref 0 aborting False >Thread-8674::DEBUG::2012-12-02 23:33:25,281::task::568::TaskManager.Task::(_updateState) Task=`86805df7-af20-4bad-bcef-73145e4b5922`::moving from state init -> state preparing >Thread-8674::INFO::2012-12-02 23:33:25,282::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8674::INFO::2012-12-02 23:33:25,282::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00745391845703', 'lastCheck': 1354484000.185535, 'code': 0, 'valid': True}} >Thread-8674::DEBUG::2012-12-02 23:33:25,282::task::1151::TaskManager.Task::(prepare) Task=`86805df7-af20-4bad-bcef-73145e4b5922`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00745391845703', 'lastCheck': 1354484000.185535, 'code': 0, 'valid': True}} >Thread-8674::DEBUG::2012-12-02 23:33:25,282::task::568::TaskManager.Task::(_updateState) Task=`86805df7-af20-4bad-bcef-73145e4b5922`::moving from state preparing -> state finished >Thread-8674::DEBUG::2012-12-02 23:33:25,282::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8674::DEBUG::2012-12-02 23:33:25,282::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8674::DEBUG::2012-12-02 23:33:25,283::task::957::TaskManager.Task::(_decref) Task=`86805df7-af20-4bad-bcef-73145e4b5922`::ref 0 aborting False >Thread-8679::DEBUG::2012-12-02 23:33:32,715::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8679::DEBUG::2012-12-02 23:33:32,715::task::568::TaskManager.Task::(_updateState) Task=`11df4d6d-a00b-48e6-ad44-b79971d26529`::moving from state init -> state preparing >Thread-8679::INFO::2012-12-02 23:33:32,715::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8679::INFO::2012-12-02 23:33:32,716::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8679::DEBUG::2012-12-02 23:33:32,716::task::1151::TaskManager.Task::(prepare) Task=`11df4d6d-a00b-48e6-ad44-b79971d26529`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8679::DEBUG::2012-12-02 23:33:32,716::task::568::TaskManager.Task::(_updateState) Task=`11df4d6d-a00b-48e6-ad44-b79971d26529`::moving from state preparing -> state finished >Thread-8679::DEBUG::2012-12-02 23:33:32,716::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8679::DEBUG::2012-12-02 23:33:32,716::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8679::DEBUG::2012-12-02 23:33:32,716::task::957::TaskManager.Task::(_decref) Task=`11df4d6d-a00b-48e6-ad44-b79971d26529`::ref 0 aborting False >Thread-8680::DEBUG::2012-12-02 23:33:32,858::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8680::DEBUG::2012-12-02 23:33:32,858::task::568::TaskManager.Task::(_updateState) Task=`9db7f7a2-4269-4b67-91d9-f397cb4e9b04`::moving from state init -> state preparing >Thread-8680::INFO::2012-12-02 23:33:32,858::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8680::DEBUG::2012-12-02 23:33:32,859::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`65b68b67-8689-4249-a2ad-645ae7ad22cb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8680::DEBUG::2012-12-02 23:33:32,859::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8680::DEBUG::2012-12-02 23:33:32,859::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8680::DEBUG::2012-12-02 23:33:32,859::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`65b68b67-8689-4249-a2ad-645ae7ad22cb`::Granted request >Thread-8680::DEBUG::2012-12-02 23:33:32,859::task::794::TaskManager.Task::(resourceAcquired) Task=`9db7f7a2-4269-4b67-91d9-f397cb4e9b04`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8680::DEBUG::2012-12-02 23:33:32,860::task::957::TaskManager.Task::(_decref) Task=`9db7f7a2-4269-4b67-91d9-f397cb4e9b04`::ref 1 aborting False >Thread-8680::DEBUG::2012-12-02 23:33:32,866::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8680::DEBUG::2012-12-02 23:33:32,866::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8680::DEBUG::2012-12-02 23:33:33,074::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8680::WARNING::2012-12-02 23:33:33,075::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8680::DEBUG::2012-12-02 23:33:33,076::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8680::WARNING::2012-12-02 23:33:33,103::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8680::INFO::2012-12-02 23:33:33,104::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8680::DEBUG::2012-12-02 23:33:33,104::task::1151::TaskManager.Task::(prepare) Task=`9db7f7a2-4269-4b67-91d9-f397cb4e9b04`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8680::DEBUG::2012-12-02 23:33:33,104::task::568::TaskManager.Task::(_updateState) Task=`9db7f7a2-4269-4b67-91d9-f397cb4e9b04`::moving from state preparing -> state finished >Thread-8680::DEBUG::2012-12-02 23:33:33,104::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8680::DEBUG::2012-12-02 23:33:33,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8680::DEBUG::2012-12-02 23:33:33,104::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8680::DEBUG::2012-12-02 23:33:33,105::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8680::DEBUG::2012-12-02 23:33:33,105::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8680::DEBUG::2012-12-02 23:33:33,105::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8680::DEBUG::2012-12-02 23:33:33,105::task::957::TaskManager.Task::(_decref) Task=`9db7f7a2-4269-4b67-91d9-f397cb4e9b04`::ref 0 aborting False >Thread-8743::DEBUG::2012-12-02 23:33:36,236::task::568::TaskManager.Task::(_updateState) Task=`dcb6a188-4cc8-409b-90cc-d1bc5655de0c`::moving from state init -> state preparing >Thread-8743::INFO::2012-12-02 23:33:36,236::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8743::INFO::2012-12-02 23:33:36,236::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00645208358765', 'lastCheck': 1354484010.2024081, 'code': 0, 'valid': True}} >Thread-8743::DEBUG::2012-12-02 23:33:36,237::task::1151::TaskManager.Task::(prepare) Task=`dcb6a188-4cc8-409b-90cc-d1bc5655de0c`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00645208358765', 'lastCheck': 1354484010.2024081, 'code': 0, 'valid': True}} >Thread-8743::DEBUG::2012-12-02 23:33:36,237::task::568::TaskManager.Task::(_updateState) Task=`dcb6a188-4cc8-409b-90cc-d1bc5655de0c`::moving from state preparing -> state finished >Thread-8743::DEBUG::2012-12-02 23:33:36,237::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8743::DEBUG::2012-12-02 23:33:36,237::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8743::DEBUG::2012-12-02 23:33:36,237::task::957::TaskManager.Task::(_decref) Task=`dcb6a188-4cc8-409b-90cc-d1bc5655de0c`::ref 0 aborting False >Thread-8748::DEBUG::2012-12-02 23:33:43,337::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8748::DEBUG::2012-12-02 23:33:43,338::task::568::TaskManager.Task::(_updateState) Task=`e331dd42-07fc-42ac-873f-0ad32e26084c`::moving from state init -> state preparing >Thread-8748::INFO::2012-12-02 23:33:43,338::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8748::INFO::2012-12-02 23:33:43,338::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8748::DEBUG::2012-12-02 23:33:43,338::task::1151::TaskManager.Task::(prepare) Task=`e331dd42-07fc-42ac-873f-0ad32e26084c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8748::DEBUG::2012-12-02 23:33:43,338::task::568::TaskManager.Task::(_updateState) Task=`e331dd42-07fc-42ac-873f-0ad32e26084c`::moving from state preparing -> state finished >Thread-8748::DEBUG::2012-12-02 23:33:43,338::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8748::DEBUG::2012-12-02 23:33:43,338::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8748::DEBUG::2012-12-02 23:33:43,339::task::957::TaskManager.Task::(_decref) Task=`e331dd42-07fc-42ac-873f-0ad32e26084c`::ref 0 aborting False >Thread-8749::DEBUG::2012-12-02 23:33:43,475::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8749::DEBUG::2012-12-02 23:33:43,476::task::568::TaskManager.Task::(_updateState) Task=`d0dd3e01-c928-4a00-9bb2-cd933eb1a641`::moving from state init -> state preparing >Thread-8749::INFO::2012-12-02 23:33:43,476::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8749::DEBUG::2012-12-02 23:33:43,476::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d69534d4-79a7-4fd7-90eb-81e802573164`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8749::DEBUG::2012-12-02 23:33:43,476::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8749::DEBUG::2012-12-02 23:33:43,476::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8749::DEBUG::2012-12-02 23:33:43,476::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d69534d4-79a7-4fd7-90eb-81e802573164`::Granted request >Thread-8749::DEBUG::2012-12-02 23:33:43,477::task::794::TaskManager.Task::(resourceAcquired) Task=`d0dd3e01-c928-4a00-9bb2-cd933eb1a641`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8749::DEBUG::2012-12-02 23:33:43,477::task::957::TaskManager.Task::(_decref) Task=`d0dd3e01-c928-4a00-9bb2-cd933eb1a641`::ref 1 aborting False >Thread-8749::DEBUG::2012-12-02 23:33:43,483::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8749::DEBUG::2012-12-02 23:33:43,484::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8749::DEBUG::2012-12-02 23:33:43,690::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8749::WARNING::2012-12-02 23:33:43,692::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8749::DEBUG::2012-12-02 23:33:43,692::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8749::WARNING::2012-12-02 23:33:43,721::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8749::INFO::2012-12-02 23:33:43,721::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8749::DEBUG::2012-12-02 23:33:43,722::task::1151::TaskManager.Task::(prepare) Task=`d0dd3e01-c928-4a00-9bb2-cd933eb1a641`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-8749::DEBUG::2012-12-02 23:33:43,722::task::568::TaskManager.Task::(_updateState) Task=`d0dd3e01-c928-4a00-9bb2-cd933eb1a641`::moving from state preparing -> state finished >Thread-8749::DEBUG::2012-12-02 23:33:43,722::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8749::DEBUG::2012-12-02 23:33:43,722::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8749::DEBUG::2012-12-02 23:33:43,722::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8749::DEBUG::2012-12-02 23:33:43,722::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8749::DEBUG::2012-12-02 23:33:43,723::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8749::DEBUG::2012-12-02 23:33:43,723::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8749::DEBUG::2012-12-02 23:33:43,723::task::957::TaskManager.Task::(_decref) Task=`d0dd3e01-c928-4a00-9bb2-cd933eb1a641`::ref 0 aborting False >Thread-8812::DEBUG::2012-12-02 23:33:47,225::task::568::TaskManager.Task::(_updateState) Task=`c80b5f8e-f2ca-4eeb-ba6b-1f530a5f5f52`::moving from state init -> state preparing >Thread-8812::INFO::2012-12-02 23:33:47,225::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8812::INFO::2012-12-02 23:33:47,225::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00682401657104', 'lastCheck': 1354484020.219697, 'code': 0, 'valid': True}} >Thread-8812::DEBUG::2012-12-02 23:33:47,225::task::1151::TaskManager.Task::(prepare) Task=`c80b5f8e-f2ca-4eeb-ba6b-1f530a5f5f52`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00682401657104', 'lastCheck': 1354484020.219697, 'code': 0, 'valid': True}} >Thread-8812::DEBUG::2012-12-02 23:33:47,225::task::568::TaskManager.Task::(_updateState) Task=`c80b5f8e-f2ca-4eeb-ba6b-1f530a5f5f52`::moving from state preparing -> state finished >Thread-8812::DEBUG::2012-12-02 23:33:47,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8812::DEBUG::2012-12-02 23:33:47,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8812::DEBUG::2012-12-02 23:33:47,226::task::957::TaskManager.Task::(_decref) Task=`c80b5f8e-f2ca-4eeb-ba6b-1f530a5f5f52`::ref 0 aborting False >Thread-8814::DEBUG::2012-12-02 23:33:48,481::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8814::DEBUG::2012-12-02 23:33:48,481::task::568::TaskManager.Task::(_updateState) Task=`2b294a3f-b870-4679-846e-7e7996c5e03a`::moving from state init -> state preparing >Thread-8814::INFO::2012-12-02 23:33:48,481::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-8814::INFO::2012-12-02 23:33:48,482::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-8814::DEBUG::2012-12-02 23:33:48,482::task::1151::TaskManager.Task::(prepare) Task=`2b294a3f-b870-4679-846e-7e7996c5e03a`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-8814::DEBUG::2012-12-02 23:33:48,482::task::568::TaskManager.Task::(_updateState) Task=`2b294a3f-b870-4679-846e-7e7996c5e03a`::moving from state preparing -> state finished >Thread-8814::DEBUG::2012-12-02 23:33:48,482::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8814::DEBUG::2012-12-02 23:33:48,482::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8814::DEBUG::2012-12-02 23:33:48,482::task::957::TaskManager.Task::(_decref) Task=`2b294a3f-b870-4679-846e-7e7996c5e03a`::ref 0 aborting False >Thread-8815::DEBUG::2012-12-02 23:33:48,669::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8815::DEBUG::2012-12-02 23:33:48,670::task::568::TaskManager.Task::(_updateState) Task=`3f928e6e-9341-4b4d-8ee9-2cdf70670b62`::moving from state init -> state preparing >Thread-8815::INFO::2012-12-02 23:33:48,670::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-8815::DEBUG::2012-12-02 23:33:48,676::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328' (cwd None) >Thread-8815::DEBUG::2012-12-02 23:33:48,794::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8815::DEBUG::2012-12-02 23:33:48,794::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8815::DEBUG::2012-12-02 23:33:48,794::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8815::DEBUG::2012-12-02 23:33:48,794::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8815::DEBUG::2012-12-02 23:33:48,795::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8815::DEBUG::2012-12-02 23:33:48,795::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8815::INFO::2012-12-02 23:33:48,795::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-8815::DEBUG::2012-12-02 23:33:48,795::task::1151::TaskManager.Task::(prepare) Task=`3f928e6e-9341-4b4d-8ee9-2cdf70670b62`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-8815::DEBUG::2012-12-02 23:33:48,795::task::568::TaskManager.Task::(_updateState) Task=`3f928e6e-9341-4b4d-8ee9-2cdf70670b62`::moving from state preparing -> state finished >Thread-8815::DEBUG::2012-12-02 23:33:48,795::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8815::DEBUG::2012-12-02 23:33:48,795::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8815::DEBUG::2012-12-02 23:33:48,796::task::957::TaskManager.Task::(_decref) Task=`3f928e6e-9341-4b4d-8ee9-2cdf70670b62`::ref 0 aborting False >Thread-8816::DEBUG::2012-12-02 23:33:49,340::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8816::DEBUG::2012-12-02 23:33:49,341::task::568::TaskManager.Task::(_updateState) Task=`1a578584-a0bb-43a2-b5b9-5431ef560282`::moving from state init -> state preparing >Thread-8816::INFO::2012-12-02 23:33:49,341::logUtils::37::dispatcher::(wrapper) Run and protect: createStorageDomain(storageType=1, sdUUID='5971512c-ee51-475e-b956-3872c38ce193', domainName='aaa', typeSpecificArg='multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', domClass=1, domVersion='3', options=None) >Thread-8816::DEBUG::2012-12-02 23:33:49,341::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-8816::DEBUG::2012-12-02 23:33:49,341::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8816::DEBUG::2012-12-02 23:33:49,341::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-8816::DEBUG::2012-12-02 23:33:49,341::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8816::DEBUG::2012-12-02 23:33:49,342::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-8816::DEBUG::2012-12-02 23:33:49,370::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-8816::DEBUG::2012-12-02 23:33:49,370::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8816::DEBUG::2012-12-02 23:33:50,002::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-8816::DEBUG::2012-12-02 23:33:50,104::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-8816::DEBUG::2012-12-02 23:33:50,104::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,107::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,107::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,108::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,108::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,108::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,108::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8816::DEBUG::2012-12-02 23:33:50,108::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8816::DEBUG::2012-12-02 23:33:50,110::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-8816::DEBUG::2012-12-02 23:33:50,318::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-8816::WARNING::2012-12-02 23:33:50,319::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-8816::DEBUG::2012-12-02 23:33:50,320::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8816::INFO::2012-12-02 23:33:50,350::nfsSD::64::Storage.StorageDomain::(create) sdUUID=5971512c-ee51-475e-b956-3872c38ce193 domainName=aaa remotePath=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328 domClass=1 >Thread-8816::DEBUG::2012-12-02 23:33:50,561::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-8816::DEBUG::2012-12-02 23:33:50,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-8816::WARNING::2012-12-02 23:33:50,562::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-8816::DEBUG::2012-12-02 23:33:50,563::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-8816::DEBUG::2012-12-02 23:33:50,563::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-8816::DEBUG::2012-12-02 23:33:50,563::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dc3b77637ab57f3d6d5fa35ee341f87a2f3334fb'] >Thread-8816::DEBUG::2012-12-02 23:33:50,599::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-8816::DEBUG::2012-12-02 23:33:50,612::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-8816::DEBUG::2012-12-02 23:33:50,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-8816::DEBUG::2012-12-02 23:33:50,623::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dc3b77637ab57f3d6d5fa35ee341f87a2f3334fb'] >Thread-8816::DEBUG::2012-12-02 23:33:50,627::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-8816::DEBUG::2012-12-02 23:33:50,627::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '5971512c-ee51-475e-b956-3872c38ce193_imageNS' >Thread-8816::DEBUG::2012-12-02 23:33:50,628::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '5971512c-ee51-475e-b956-3872c38ce193_volumeNS' >Thread-8816::DEBUG::2012-12-02 23:33:50,839::sd::389::Storage.StorageDomain::(initSPMlease) lease initialized successfully >Thread-8816::INFO::2012-12-02 23:33:50,839::logUtils::39::dispatcher::(wrapper) Run and protect: createStorageDomain, Return response: None >Thread-8816::DEBUG::2012-12-02 23:33:50,839::task::1151::TaskManager.Task::(prepare) Task=`1a578584-a0bb-43a2-b5b9-5431ef560282`::finished: None >Thread-8816::DEBUG::2012-12-02 23:33:50,839::task::568::TaskManager.Task::(_updateState) Task=`1a578584-a0bb-43a2-b5b9-5431ef560282`::moving from state preparing -> state finished >Thread-8816::DEBUG::2012-12-02 23:33:50,839::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8816::DEBUG::2012-12-02 23:33:50,839::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8816::DEBUG::2012-12-02 23:33:50,840::task::957::TaskManager.Task::(_decref) Task=`1a578584-a0bb-43a2-b5b9-5431ef560282`::ref 0 aborting False >Thread-8880::DEBUG::2012-12-02 23:33:51,003::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8880::DEBUG::2012-12-02 23:33:51,003::task::568::TaskManager.Task::(_updateState) Task=`a3ce151c-a181-4310-98b9-78ffec180737`::moving from state init -> state preparing >Thread-8880::INFO::2012-12-02 23:33:51,003::logUtils::37::dispatcher::(wrapper) Run and protect: getStorageDomainStats(sdUUID='5971512c-ee51-475e-b956-3872c38ce193', options=None) >Thread-8880::DEBUG::2012-12-02 23:33:51,003::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`461a0f16-90c8-480b-b6d3-7c78d6a7accd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8880::DEBUG::2012-12-02 23:33:51,004::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' for lock type 'shared' >Thread-8880::DEBUG::2012-12-02 23:33:51,004::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free. Now locking as 'shared' (1 active user) >Thread-8880::DEBUG::2012-12-02 23:33:51,004::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`461a0f16-90c8-480b-b6d3-7c78d6a7accd`::Granted request >Thread-8880::DEBUG::2012-12-02 23:33:51,004::task::794::TaskManager.Task::(resourceAcquired) Task=`a3ce151c-a181-4310-98b9-78ffec180737`::_resourcesAcquired: Storage.5971512c-ee51-475e-b956-3872c38ce193 (shared) >Thread-8880::DEBUG::2012-12-02 23:33:51,004::task::957::TaskManager.Task::(_decref) Task=`a3ce151c-a181-4310-98b9-78ffec180737`::ref 1 aborting False >Thread-8880::INFO::2012-12-02 23:33:51,006::logUtils::39::dispatcher::(wrapper) Run and protect: getStorageDomainStats, Return response: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '45385515008', 'disktotal': '53687091200', 'mdafree': 0}} >Thread-8880::DEBUG::2012-12-02 23:33:51,006::task::1151::TaskManager.Task::(prepare) Task=`a3ce151c-a181-4310-98b9-78ffec180737`::finished: {'stats': {'mdasize': 0, 'mdathreshold': True, 'mdavalid': True, 'diskfree': '45385515008', 'disktotal': '53687091200', 'mdafree': 0}} >Thread-8880::DEBUG::2012-12-02 23:33:51,006::task::568::TaskManager.Task::(_updateState) Task=`a3ce151c-a181-4310-98b9-78ffec180737`::moving from state preparing -> state finished >Thread-8880::DEBUG::2012-12-02 23:33:51,006::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.5971512c-ee51-475e-b956-3872c38ce193': < ResourceRef 'Storage.5971512c-ee51-475e-b956-3872c38ce193', isValid: 'True' obj: 'None'>} >Thread-8880::DEBUG::2012-12-02 23:33:51,006::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8880::DEBUG::2012-12-02 23:33:51,007::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' >Thread-8880::DEBUG::2012-12-02 23:33:51,007::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' (0 active users) >Thread-8880::DEBUG::2012-12-02 23:33:51,007::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free, finding out if anyone is waiting for it. >Thread-8880::DEBUG::2012-12-02 23:33:51,007::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193', Clearing records. >Thread-8880::DEBUG::2012-12-02 23:33:51,007::task::957::TaskManager.Task::(_decref) Task=`a3ce151c-a181-4310-98b9-78ffec180737`::ref 0 aborting False >Thread-8881::DEBUG::2012-12-02 23:33:51,440::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8881::DEBUG::2012-12-02 23:33:51,441::task::568::TaskManager.Task::(_updateState) Task=`cec99f92-316d-49bd-953f-8a6311764cba`::moving from state init -> state preparing >Thread-8881::INFO::2012-12-02 23:33:51,441::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None) >Thread-8881::DEBUG::2012-12-02 23:33:51,441::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328' (cwd None) >Thread-8881::DEBUG::2012-12-02 23:33:51,468::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-8881::DEBUG::2012-12-02 23:33:51,468::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8881::DEBUG::2012-12-02 23:33:51,468::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-8881::DEBUG::2012-12-02 23:33:51,469::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8881::DEBUG::2012-12-02 23:33:51,469::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-8881::DEBUG::2012-12-02 23:33:51,498::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-8881::DEBUG::2012-12-02 23:33:51,499::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8881::DEBUG::2012-12-02 23:33:52,131::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-8881::DEBUG::2012-12-02 23:33:52,221::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-8881::DEBUG::2012-12-02 23:33:52,222::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8881::DEBUG::2012-12-02 23:33:52,222::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8881::DEBUG::2012-12-02 23:33:52,222::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8881::DEBUG::2012-12-02 23:33:52,222::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8881::DEBUG::2012-12-02 23:33:52,222::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8881::DEBUG::2012-12-02 23:33:52,222::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8881::DEBUG::2012-12-02 23:33:52,223::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8881::INFO::2012-12-02 23:33:52,223::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-8881::DEBUG::2012-12-02 23:33:52,223::task::1151::TaskManager.Task::(prepare) Task=`cec99f92-316d-49bd-953f-8a6311764cba`::finished: {'statuslist': [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]} >Thread-8881::DEBUG::2012-12-02 23:33:52,223::task::568::TaskManager.Task::(_updateState) Task=`cec99f92-316d-49bd-953f-8a6311764cba`::moving from state preparing -> state finished >Thread-8881::DEBUG::2012-12-02 23:33:52,223::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8881::DEBUG::2012-12-02 23:33:52,223::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8881::DEBUG::2012-12-02 23:33:52,224::task::957::TaskManager.Task::(_decref) Task=`cec99f92-316d-49bd-953f-8a6311764cba`::ref 0 aborting False >Thread-8883::DEBUG::2012-12-02 23:33:52,728::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8883::DEBUG::2012-12-02 23:33:52,729::task::568::TaskManager.Task::(_updateState) Task=`f928fd90-39a0-41a4-9fa4-a2899918418a`::moving from state init -> state preparing >Thread-8883::INFO::2012-12-02 23:33:52,729::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-8883::INFO::2012-12-02 23:33:52,729::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-8883::DEBUG::2012-12-02 23:33:52,729::task::1151::TaskManager.Task::(prepare) Task=`f928fd90-39a0-41a4-9fa4-a2899918418a`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-8883::DEBUG::2012-12-02 23:33:52,729::task::568::TaskManager.Task::(_updateState) Task=`f928fd90-39a0-41a4-9fa4-a2899918418a`::moving from state preparing -> state finished >Thread-8883::DEBUG::2012-12-02 23:33:52,730::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8883::DEBUG::2012-12-02 23:33:52,730::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8883::DEBUG::2012-12-02 23:33:52,730::task::957::TaskManager.Task::(_decref) Task=`f928fd90-39a0-41a4-9fa4-a2899918418a`::ref 0 aborting False >Thread-8884::DEBUG::2012-12-02 23:33:52,888::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8884::DEBUG::2012-12-02 23:33:52,889::task::568::TaskManager.Task::(_updateState) Task=`1eca481a-6545-40a6-8b30-1429bc4b76b7`::moving from state init -> state preparing >Thread-8884::INFO::2012-12-02 23:33:52,889::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-8884::DEBUG::2012-12-02 23:33:52,895::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328' (cwd None) >Thread-8884::DEBUG::2012-12-02 23:33:53,024::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8884::DEBUG::2012-12-02 23:33:53,025::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8884::DEBUG::2012-12-02 23:33:53,025::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8884::DEBUG::2012-12-02 23:33:53,025::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8884::DEBUG::2012-12-02 23:33:53,025::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8884::DEBUG::2012-12-02 23:33:53,025::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8884::INFO::2012-12-02 23:33:53,026::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-8884::DEBUG::2012-12-02 23:33:53,026::task::1151::TaskManager.Task::(prepare) Task=`1eca481a-6545-40a6-8b30-1429bc4b76b7`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-8884::DEBUG::2012-12-02 23:33:53,026::task::568::TaskManager.Task::(_updateState) Task=`1eca481a-6545-40a6-8b30-1429bc4b76b7`::moving from state preparing -> state finished >Thread-8884::DEBUG::2012-12-02 23:33:53,026::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8884::DEBUG::2012-12-02 23:33:53,026::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8884::DEBUG::2012-12-02 23:33:53,026::task::957::TaskManager.Task::(_decref) Task=`1eca481a-6545-40a6-8b30-1429bc4b76b7`::ref 0 aborting False >Thread-8885::DEBUG::2012-12-02 23:33:53,175::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8885::DEBUG::2012-12-02 23:33:53,175::task::568::TaskManager.Task::(_updateState) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::moving from state init -> state preparing >Thread-8885::INFO::2012-12-02 23:33:53,175::logUtils::37::dispatcher::(wrapper) Run and protect: attachStorageDomain(sdUUID='5971512c-ee51-475e-b956-3872c38ce193', spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8885::DEBUG::2012-12-02 23:33:53,175::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f573b460-9e5e-45bf-abfd-7656647b8ed8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8885::DEBUG::2012-12-02 23:33:53,176::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-8885::DEBUG::2012-12-02 23:33:53,176::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-8885::DEBUG::2012-12-02 23:33:53,176::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f573b460-9e5e-45bf-abfd-7656647b8ed8`::Granted request >Thread-8885::DEBUG::2012-12-02 23:33:53,176::task::794::TaskManager.Task::(resourceAcquired) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-8885::DEBUG::2012-12-02 23:33:53,176::task::957::TaskManager.Task::(_decref) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::ref 1 aborting False >Thread-8885::DEBUG::2012-12-02 23:33:53,176::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`23943e0e-641b-4205-a69c-8bcbed6305bb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8885::DEBUG::2012-12-02 23:33:53,176::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' for lock type 'exclusive' >Thread-8885::DEBUG::2012-12-02 23:33:53,176::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free. Now locking as 'exclusive' (1 active user) >Thread-8885::DEBUG::2012-12-02 23:33:53,177::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`23943e0e-641b-4205-a69c-8bcbed6305bb`::Granted request >Thread-8885::DEBUG::2012-12-02 23:33:53,177::task::794::TaskManager.Task::(resourceAcquired) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::_resourcesAcquired: Storage.5971512c-ee51-475e-b956-3872c38ce193 (exclusive) >Thread-8885::DEBUG::2012-12-02 23:33:53,177::task::957::TaskManager.Task::(_decref) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::ref 1 aborting False >Thread-8885::INFO::2012-12-02 23:33:53,177::sp::912::Storage.StoragePool::(attachSD) sdUUID=5971512c-ee51-475e-b956-3872c38ce193 spUUID=479cf195-bb99-45c5-b400-433d810eaa48 >Thread-8885::DEBUG::2012-12-02 23:33:53,181::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-8885::DEBUG::2012-12-02 23:33:53,181::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8885::DEBUG::2012-12-02 23:33:53,181::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-8885::DEBUG::2012-12-02 23:33:53,181::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8885::DEBUG::2012-12-02 23:33:53,182::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-8885::DEBUG::2012-12-02 23:33:53,208::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-8885::DEBUG::2012-12-02 23:33:53,208::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8885::DEBUG::2012-12-02 23:33:53,841::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-8885::DEBUG::2012-12-02 23:33:53,915::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-8885::DEBUG::2012-12-02 23:33:53,915::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8885::DEBUG::2012-12-02 23:33:53,916::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8885::DEBUG::2012-12-02 23:33:53,916::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8885::DEBUG::2012-12-02 23:33:53,916::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8885::DEBUG::2012-12-02 23:33:53,916::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-8885::DEBUG::2012-12-02 23:33:53,916::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-8885::DEBUG::2012-12-02 23:33:53,916::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8885::INFO::2012-12-02 23:33:53,917::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-8888::DEBUG::2012-12-02 23:33:58,144::task::568::TaskManager.Task::(_updateState) Task=`846856fa-b983-4584-ba4e-13524391d4ee`::moving from state init -> state preparing >Thread-8888::INFO::2012-12-02 23:33:58,144::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8888::INFO::2012-12-02 23:33:58,144::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00755620002747', 'lastCheck': 1354484030.375973, 'code': 0, 'valid': True}} >Thread-8888::DEBUG::2012-12-02 23:33:58,144::task::1151::TaskManager.Task::(prepare) Task=`846856fa-b983-4584-ba4e-13524391d4ee`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00755620002747', 'lastCheck': 1354484030.375973, 'code': 0, 'valid': True}} >Thread-8888::DEBUG::2012-12-02 23:33:58,144::task::568::TaskManager.Task::(_updateState) Task=`846856fa-b983-4584-ba4e-13524391d4ee`::moving from state preparing -> state finished >Thread-8888::DEBUG::2012-12-02 23:33:58,145::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8888::DEBUG::2012-12-02 23:33:58,145::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8888::DEBUG::2012-12-02 23:33:58,145::task::957::TaskManager.Task::(_decref) Task=`846856fa-b983-4584-ba4e-13524391d4ee`::ref 0 aborting False >Thread-8894::DEBUG::2012-12-02 23:34:09,079::task::568::TaskManager.Task::(_updateState) Task=`96b2c45a-bb9c-4f34-bbdd-c5874c702131`::moving from state init -> state preparing >Thread-8894::INFO::2012-12-02 23:34:09,080::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8894::INFO::2012-12-02 23:34:09,080::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00951910018921', 'lastCheck': 1354484040.400476, 'code': 0, 'valid': True}} >Thread-8894::DEBUG::2012-12-02 23:34:09,080::task::1151::TaskManager.Task::(prepare) Task=`96b2c45a-bb9c-4f34-bbdd-c5874c702131`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00951910018921', 'lastCheck': 1354484040.400476, 'code': 0, 'valid': True}} >Thread-8894::DEBUG::2012-12-02 23:34:09,080::task::568::TaskManager.Task::(_updateState) Task=`96b2c45a-bb9c-4f34-bbdd-c5874c702131`::moving from state preparing -> state finished >Thread-8894::DEBUG::2012-12-02 23:34:09,080::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8894::DEBUG::2012-12-02 23:34:09,080::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8894::DEBUG::2012-12-02 23:34:09,081::task::957::TaskManager.Task::(_decref) Task=`96b2c45a-bb9c-4f34-bbdd-c5874c702131`::ref 0 aborting False >Thread-8885::DEBUG::2012-12-02 23:34:14,917::safelease::178::SANLock::(acquireHostId) Host id for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully acquired (id: 2) >Thread-8885::INFO::2012-12-02 23:34:14,918::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-8885::DEBUG::2012-12-02 23:34:15,239::safelease::235::SANLock::(acquire) Cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully acquired (id: 2) >Thread-8885::DEBUG::2012-12-02 23:34:15,246::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dc3b77637ab57f3d6d5fa35ee341f87a2f3334fb'] >Thread-8885::DEBUG::2012-12-02 23:34:15,246::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-8885::DEBUG::2012-12-02 23:34:15,246::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-8885::DEBUG::2012-12-02 23:34:15,247::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=8d74be082d97dc7dc025801b95e2c1e302a15747'] >Thread-8885::DEBUG::2012-12-02 23:34:15,276::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-8885::DEBUG::2012-12-02 23:34:15,276::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-8885::DEBUG::2012-12-02 23:34:15,276::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-8885::DEBUG::2012-12-02 23:34:15,276::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=89e0caa8df7ea3cfe3449af99797735ba34a6030'] >Thread-8885::DEBUG::2012-12-02 23:34:15,312::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-8885::INFO::2012-12-02 23:34:15,312::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/5971512c-ee51-475e-b956-3872c38ce193 >Thread-8885::DEBUG::2012-12-02 23:34:15,313::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-8885::DEBUG::2012-12-02 23:34:15,313::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-8885::DEBUG::2012-12-02 23:34:15,313::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-8885::DEBUG::2012-12-02 23:34:15,313::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-8885::DEBUG::2012-12-02 23:34:15,313::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=148af2e9c9d9dfed2227286a51a8b73853083e95'] >Thread-8885::DEBUG::2012-12-02 23:34:15,349::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-8885::INFO::2012-12-02 23:34:15,349::safelease::239::SANLock::(release) Releasing cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 >Thread-8885::DEBUG::2012-12-02 23:34:15,381::safelease::250::SANLock::(release) Cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully released >Thread-8885::INFO::2012-12-02 23:34:15,381::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-8885::DEBUG::2012-12-02 23:34:18,382::safelease::193::SANLock::(releaseHostId) Host id for domain 5971512c-ee51-475e-b956-3872c38ce193 released successfully (id: 2) >Thread-8885::DEBUG::2012-12-02 23:34:18,382::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-8885::DEBUG::2012-12-02 23:34:18,382::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-8885::DEBUG::2012-12-02 23:34:18,382::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8885::DEBUG::2012-12-02 23:34:18,384::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8885::DEBUG::2012-12-02 23:34:18,598::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8885::WARNING::2012-12-02 23:34:18,600::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8885::DEBUG::2012-12-02 23:34:18,600::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8885::ERROR::2012-12-02 23:34:18,636::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8885::DEBUG::2012-12-02 23:34:18,636::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-8885::INFO::2012-12-02 23:34:18,636::logUtils::39::dispatcher::(wrapper) Run and protect: attachStorageDomain, Return response: None >Thread-8885::DEBUG::2012-12-02 23:34:18,636::task::1151::TaskManager.Task::(prepare) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::finished: None >Thread-8885::DEBUG::2012-12-02 23:34:18,636::task::568::TaskManager.Task::(_updateState) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::moving from state preparing -> state finished >Thread-8885::DEBUG::2012-12-02 23:34:18,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.5971512c-ee51-475e-b956-3872c38ce193': < ResourceRef 'Storage.5971512c-ee51-475e-b956-3872c38ce193', isValid: 'True' obj: 'None'>, 'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8885::DEBUG::2012-12-02 23:34:18,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8885::DEBUG::2012-12-02 23:34:18,637::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' >Thread-8885::DEBUG::2012-12-02 23:34:18,637::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' (0 active users) >Thread-8885::DEBUG::2012-12-02 23:34:18,637::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free, finding out if anyone is waiting for it. >Thread-8885::DEBUG::2012-12-02 23:34:18,637::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193', Clearing records. >Thread-8885::DEBUG::2012-12-02 23:34:18,638::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8885::DEBUG::2012-12-02 23:34:18,638::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8885::DEBUG::2012-12-02 23:34:18,638::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8885::DEBUG::2012-12-02 23:34:18,638::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8885::DEBUG::2012-12-02 23:34:18,638::task::957::TaskManager.Task::(_decref) Task=`af048052-6251-47b0-bfe9-600aa0a45612`::ref 0 aborting False >Thread-8962::DEBUG::2012-12-02 23:34:18,788::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8962::DEBUG::2012-12-02 23:34:18,788::task::568::TaskManager.Task::(_updateState) Task=`1990a5cf-f832-47c4-a50e-30d836863704`::moving from state init -> state preparing >Thread-8962::INFO::2012-12-02 23:34:18,788::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8962::INFO::2012-12-02 23:34:18,788::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8962::DEBUG::2012-12-02 23:34:18,789::task::1151::TaskManager.Task::(prepare) Task=`1990a5cf-f832-47c4-a50e-30d836863704`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8962::DEBUG::2012-12-02 23:34:18,789::task::568::TaskManager.Task::(_updateState) Task=`1990a5cf-f832-47c4-a50e-30d836863704`::moving from state preparing -> state finished >Thread-8962::DEBUG::2012-12-02 23:34:18,789::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8962::DEBUG::2012-12-02 23:34:18,789::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8962::DEBUG::2012-12-02 23:34:18,789::task::957::TaskManager.Task::(_decref) Task=`1990a5cf-f832-47c4-a50e-30d836863704`::ref 0 aborting False >Thread-8963::DEBUG::2012-12-02 23:34:18,942::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8963::DEBUG::2012-12-02 23:34:18,943::task::568::TaskManager.Task::(_updateState) Task=`a12bf32a-7bde-492a-b754-b9816972ebcb`::moving from state init -> state preparing >Thread-8963::INFO::2012-12-02 23:34:18,943::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8963::DEBUG::2012-12-02 23:34:18,943::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e3ae51b8-1469-43a9-984b-45541c792cbf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8963::DEBUG::2012-12-02 23:34:18,943::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8963::DEBUG::2012-12-02 23:34:18,944::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8963::DEBUG::2012-12-02 23:34:18,944::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e3ae51b8-1469-43a9-984b-45541c792cbf`::Granted request >Thread-8963::DEBUG::2012-12-02 23:34:18,944::task::794::TaskManager.Task::(resourceAcquired) Task=`a12bf32a-7bde-492a-b754-b9816972ebcb`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8963::DEBUG::2012-12-02 23:34:18,944::task::957::TaskManager.Task::(_decref) Task=`a12bf32a-7bde-492a-b754-b9816972ebcb`::ref 1 aborting False >Thread-8963::DEBUG::2012-12-02 23:34:18,949::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8963::DEBUG::2012-12-02 23:34:18,949::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8963::DEBUG::2012-12-02 23:34:19,157::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8963::WARNING::2012-12-02 23:34:19,158::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8963::DEBUG::2012-12-02 23:34:19,159::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8963::WARNING::2012-12-02 23:34:19,188::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8963::INFO::2012-12-02 23:34:19,188::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-8963::DEBUG::2012-12-02 23:34:19,188::task::1151::TaskManager.Task::(prepare) Task=`a12bf32a-7bde-492a-b754-b9816972ebcb`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-8963::DEBUG::2012-12-02 23:34:19,189::task::568::TaskManager.Task::(_updateState) Task=`a12bf32a-7bde-492a-b754-b9816972ebcb`::moving from state preparing -> state finished >Thread-8963::DEBUG::2012-12-02 23:34:19,189::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8963::DEBUG::2012-12-02 23:34:19,189::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8963::DEBUG::2012-12-02 23:34:19,189::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8963::DEBUG::2012-12-02 23:34:19,189::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8963::DEBUG::2012-12-02 23:34:19,189::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8963::DEBUG::2012-12-02 23:34:19,190::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8963::DEBUG::2012-12-02 23:34:19,190::task::957::TaskManager.Task::(_decref) Task=`a12bf32a-7bde-492a-b754-b9816972ebcb`::ref 0 aborting False >Thread-9026::DEBUG::2012-12-02 23:34:20,109::task::568::TaskManager.Task::(_updateState) Task=`e4689614-4281-4d2e-9ac2-fc1e393c91b8`::moving from state init -> state preparing >Thread-9026::INFO::2012-12-02 23:34:20,109::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9026::INFO::2012-12-02 23:34:20,109::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00607490539551', 'lastCheck': 1354484050.417995, 'code': 0, 'valid': True}} >Thread-9026::DEBUG::2012-12-02 23:34:20,110::task::1151::TaskManager.Task::(prepare) Task=`e4689614-4281-4d2e-9ac2-fc1e393c91b8`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00607490539551', 'lastCheck': 1354484050.417995, 'code': 0, 'valid': True}} >Thread-9026::DEBUG::2012-12-02 23:34:20,110::task::568::TaskManager.Task::(_updateState) Task=`e4689614-4281-4d2e-9ac2-fc1e393c91b8`::moving from state preparing -> state finished >Thread-9026::DEBUG::2012-12-02 23:34:20,110::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9026::DEBUG::2012-12-02 23:34:20,110::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9026::DEBUG::2012-12-02 23:34:20,110::task::957::TaskManager.Task::(_decref) Task=`e4689614-4281-4d2e-9ac2-fc1e393c91b8`::ref 0 aborting False >Thread-9032::DEBUG::2012-12-02 23:34:29,406::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9032::DEBUG::2012-12-02 23:34:29,406::task::568::TaskManager.Task::(_updateState) Task=`43a09717-016c-4b7a-9cc5-98a40f8e67aa`::moving from state init -> state preparing >Thread-9032::INFO::2012-12-02 23:34:29,406::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9032::INFO::2012-12-02 23:34:29,407::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9032::DEBUG::2012-12-02 23:34:29,407::task::1151::TaskManager.Task::(prepare) Task=`43a09717-016c-4b7a-9cc5-98a40f8e67aa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9032::DEBUG::2012-12-02 23:34:29,407::task::568::TaskManager.Task::(_updateState) Task=`43a09717-016c-4b7a-9cc5-98a40f8e67aa`::moving from state preparing -> state finished >Thread-9032::DEBUG::2012-12-02 23:34:29,407::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9032::DEBUG::2012-12-02 23:34:29,407::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9032::DEBUG::2012-12-02 23:34:29,407::task::957::TaskManager.Task::(_decref) Task=`43a09717-016c-4b7a-9cc5-98a40f8e67aa`::ref 0 aborting False >Thread-9033::DEBUG::2012-12-02 23:34:29,565::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9033::DEBUG::2012-12-02 23:34:29,565::task::568::TaskManager.Task::(_updateState) Task=`dde87830-7b3e-496a-b271-d44d62c1bd6d`::moving from state init -> state preparing >Thread-9033::INFO::2012-12-02 23:34:29,566::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9033::DEBUG::2012-12-02 23:34:29,566::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d64a1af7-46ab-460e-9da8-c00853833743`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9033::DEBUG::2012-12-02 23:34:29,566::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9033::DEBUG::2012-12-02 23:34:29,566::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9033::DEBUG::2012-12-02 23:34:29,566::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d64a1af7-46ab-460e-9da8-c00853833743`::Granted request >Thread-9033::DEBUG::2012-12-02 23:34:29,567::task::794::TaskManager.Task::(resourceAcquired) Task=`dde87830-7b3e-496a-b271-d44d62c1bd6d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9033::DEBUG::2012-12-02 23:34:29,567::task::957::TaskManager.Task::(_decref) Task=`dde87830-7b3e-496a-b271-d44d62c1bd6d`::ref 1 aborting False >Thread-9033::DEBUG::2012-12-02 23:34:29,573::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9033::DEBUG::2012-12-02 23:34:29,574::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9033::DEBUG::2012-12-02 23:34:29,782::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9033::WARNING::2012-12-02 23:34:29,784::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9033::DEBUG::2012-12-02 23:34:29,784::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9033::WARNING::2012-12-02 23:34:29,814::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9033::INFO::2012-12-02 23:34:29,814::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-9033::DEBUG::2012-12-02 23:34:29,815::task::1151::TaskManager.Task::(prepare) Task=`dde87830-7b3e-496a-b271-d44d62c1bd6d`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-9033::DEBUG::2012-12-02 23:34:29,815::task::568::TaskManager.Task::(_updateState) Task=`dde87830-7b3e-496a-b271-d44d62c1bd6d`::moving from state preparing -> state finished >Thread-9033::DEBUG::2012-12-02 23:34:29,815::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9033::DEBUG::2012-12-02 23:34:29,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9033::DEBUG::2012-12-02 23:34:29,815::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9033::DEBUG::2012-12-02 23:34:29,815::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9033::DEBUG::2012-12-02 23:34:29,815::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9033::DEBUG::2012-12-02 23:34:29,816::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9033::DEBUG::2012-12-02 23:34:29,816::task::957::TaskManager.Task::(_decref) Task=`dde87830-7b3e-496a-b271-d44d62c1bd6d`::ref 0 aborting False >Thread-9096::DEBUG::2012-12-02 23:34:31,037::task::568::TaskManager.Task::(_updateState) Task=`1f0ff78c-46a8-4808-be09-d5c40551872e`::moving from state init -> state preparing >Thread-9096::INFO::2012-12-02 23:34:31,037::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9096::INFO::2012-12-02 23:34:31,037::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00673484802246', 'lastCheck': 1354484070.4547369, 'code': 0, 'valid': True}} >Thread-9096::DEBUG::2012-12-02 23:34:31,037::task::1151::TaskManager.Task::(prepare) Task=`1f0ff78c-46a8-4808-be09-d5c40551872e`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00673484802246', 'lastCheck': 1354484070.4547369, 'code': 0, 'valid': True}} >Thread-9096::DEBUG::2012-12-02 23:34:31,037::task::568::TaskManager.Task::(_updateState) Task=`1f0ff78c-46a8-4808-be09-d5c40551872e`::moving from state preparing -> state finished >Thread-9096::DEBUG::2012-12-02 23:34:31,037::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9096::DEBUG::2012-12-02 23:34:31,038::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9096::DEBUG::2012-12-02 23:34:31,038::task::957::TaskManager.Task::(_decref) Task=`1f0ff78c-46a8-4808-be09-d5c40551872e`::ref 0 aborting False >Thread-9102::DEBUG::2012-12-02 23:34:40,043::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9102::DEBUG::2012-12-02 23:34:40,044::task::568::TaskManager.Task::(_updateState) Task=`de4b247b-7095-479e-8f91-a9469c75c58f`::moving from state init -> state preparing >Thread-9102::INFO::2012-12-02 23:34:40,044::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9102::INFO::2012-12-02 23:34:40,044::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9102::DEBUG::2012-12-02 23:34:40,044::task::1151::TaskManager.Task::(prepare) Task=`de4b247b-7095-479e-8f91-a9469c75c58f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9102::DEBUG::2012-12-02 23:34:40,045::task::568::TaskManager.Task::(_updateState) Task=`de4b247b-7095-479e-8f91-a9469c75c58f`::moving from state preparing -> state finished >Thread-9102::DEBUG::2012-12-02 23:34:40,045::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9102::DEBUG::2012-12-02 23:34:40,045::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9102::DEBUG::2012-12-02 23:34:40,045::task::957::TaskManager.Task::(_decref) Task=`de4b247b-7095-479e-8f91-a9469c75c58f`::ref 0 aborting False >Thread-9103::DEBUG::2012-12-02 23:34:40,180::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9103::DEBUG::2012-12-02 23:34:40,180::task::568::TaskManager.Task::(_updateState) Task=`55da85fd-120b-4284-904e-20e2f7d4fb13`::moving from state init -> state preparing >Thread-9103::INFO::2012-12-02 23:34:40,180::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9103::DEBUG::2012-12-02 23:34:40,180::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7c131d26-44b6-4644-b277-45ba9779f454`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9103::DEBUG::2012-12-02 23:34:40,181::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9103::DEBUG::2012-12-02 23:34:40,181::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9103::DEBUG::2012-12-02 23:34:40,181::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7c131d26-44b6-4644-b277-45ba9779f454`::Granted request >Thread-9103::DEBUG::2012-12-02 23:34:40,181::task::794::TaskManager.Task::(resourceAcquired) Task=`55da85fd-120b-4284-904e-20e2f7d4fb13`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9103::DEBUG::2012-12-02 23:34:40,181::task::957::TaskManager.Task::(_decref) Task=`55da85fd-120b-4284-904e-20e2f7d4fb13`::ref 1 aborting False >Thread-9103::DEBUG::2012-12-02 23:34:40,187::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9103::DEBUG::2012-12-02 23:34:40,187::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9103::DEBUG::2012-12-02 23:34:40,395::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9103::WARNING::2012-12-02 23:34:40,396::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9103::DEBUG::2012-12-02 23:34:40,396::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9103::WARNING::2012-12-02 23:34:40,427::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9103::INFO::2012-12-02 23:34:40,427::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-9103::DEBUG::2012-12-02 23:34:40,427::task::1151::TaskManager.Task::(prepare) Task=`55da85fd-120b-4284-904e-20e2f7d4fb13`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-9103::DEBUG::2012-12-02 23:34:40,428::task::568::TaskManager.Task::(_updateState) Task=`55da85fd-120b-4284-904e-20e2f7d4fb13`::moving from state preparing -> state finished >Thread-9103::DEBUG::2012-12-02 23:34:40,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9103::DEBUG::2012-12-02 23:34:40,428::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9103::DEBUG::2012-12-02 23:34:40,428::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9103::DEBUG::2012-12-02 23:34:40,428::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9103::DEBUG::2012-12-02 23:34:40,428::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9103::DEBUG::2012-12-02 23:34:40,428::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9103::DEBUG::2012-12-02 23:34:40,429::task::957::TaskManager.Task::(_decref) Task=`55da85fd-120b-4284-904e-20e2f7d4fb13`::ref 0 aborting False >Thread-9166::DEBUG::2012-12-02 23:34:41,943::task::568::TaskManager.Task::(_updateState) Task=`269d5912-3acb-4023-8e05-ca27750179da`::moving from state init -> state preparing >Thread-9166::INFO::2012-12-02 23:34:41,943::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9166::INFO::2012-12-02 23:34:41,944::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00544714927673', 'lastCheck': 1354484080.470057, 'code': 0, 'valid': True}} >Thread-9166::DEBUG::2012-12-02 23:34:41,944::task::1151::TaskManager.Task::(prepare) Task=`269d5912-3acb-4023-8e05-ca27750179da`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00544714927673', 'lastCheck': 1354484080.470057, 'code': 0, 'valid': True}} >Thread-9166::DEBUG::2012-12-02 23:34:41,944::task::568::TaskManager.Task::(_updateState) Task=`269d5912-3acb-4023-8e05-ca27750179da`::moving from state preparing -> state finished >Thread-9166::DEBUG::2012-12-02 23:34:41,944::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9166::DEBUG::2012-12-02 23:34:41,944::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9166::DEBUG::2012-12-02 23:34:41,944::task::957::TaskManager.Task::(_decref) Task=`269d5912-3acb-4023-8e05-ca27750179da`::ref 0 aborting False >Thread-9171::DEBUG::2012-12-02 23:34:50,628::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9171::DEBUG::2012-12-02 23:34:50,628::task::568::TaskManager.Task::(_updateState) Task=`0e8feccd-da73-45fd-bc36-816a6bde2936`::moving from state init -> state preparing >Thread-9171::INFO::2012-12-02 23:34:50,628::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-9171::INFO::2012-12-02 23:34:50,629::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-9171::DEBUG::2012-12-02 23:34:50,629::task::1151::TaskManager.Task::(prepare) Task=`0e8feccd-da73-45fd-bc36-816a6bde2936`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-9171::DEBUG::2012-12-02 23:34:50,629::task::568::TaskManager.Task::(_updateState) Task=`0e8feccd-da73-45fd-bc36-816a6bde2936`::moving from state preparing -> state finished >Thread-9171::DEBUG::2012-12-02 23:34:50,629::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9171::DEBUG::2012-12-02 23:34:50,629::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9171::DEBUG::2012-12-02 23:34:50,629::task::957::TaskManager.Task::(_decref) Task=`0e8feccd-da73-45fd-bc36-816a6bde2936`::ref 0 aborting False >Thread-9172::DEBUG::2012-12-02 23:34:50,653::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9172::DEBUG::2012-12-02 23:34:50,653::task::568::TaskManager.Task::(_updateState) Task=`8d4af403-f41a-40e8-8d53-563f65b7226a`::moving from state init -> state preparing >Thread-9172::INFO::2012-12-02 23:34:50,653::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9172::INFO::2012-12-02 23:34:50,654::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9172::DEBUG::2012-12-02 23:34:50,654::task::1151::TaskManager.Task::(prepare) Task=`8d4af403-f41a-40e8-8d53-563f65b7226a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9172::DEBUG::2012-12-02 23:34:50,654::task::568::TaskManager.Task::(_updateState) Task=`8d4af403-f41a-40e8-8d53-563f65b7226a`::moving from state preparing -> state finished >Thread-9172::DEBUG::2012-12-02 23:34:50,654::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9172::DEBUG::2012-12-02 23:34:50,654::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9172::DEBUG::2012-12-02 23:34:50,654::task::957::TaskManager.Task::(_decref) Task=`8d4af403-f41a-40e8-8d53-563f65b7226a`::ref 0 aborting False >Thread-9174::DEBUG::2012-12-02 23:34:50,790::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9175::DEBUG::2012-12-02 23:34:50,791::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9174::DEBUG::2012-12-02 23:34:50,791::task::568::TaskManager.Task::(_updateState) Task=`c2fcb93b-2aa3-48da-8443-b7a802a80b1f`::moving from state init -> state preparing >Thread-9175::DEBUG::2012-12-02 23:34:50,791::task::568::TaskManager.Task::(_updateState) Task=`52a79d8d-eb3d-42de-80dd-0737a30bafeb`::moving from state init -> state preparing >Thread-9174::INFO::2012-12-02 23:34:50,792::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-9175::INFO::2012-12-02 23:34:50,792::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9175::DEBUG::2012-12-02 23:34:50,793::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ed8bb66b-f400-4ba2-8185-c78e73d51467`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9175::DEBUG::2012-12-02 23:34:50,794::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9175::DEBUG::2012-12-02 23:34:50,796::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9175::DEBUG::2012-12-02 23:34:50,796::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ed8bb66b-f400-4ba2-8185-c78e73d51467`::Granted request >Thread-9175::DEBUG::2012-12-02 23:34:50,799::task::794::TaskManager.Task::(resourceAcquired) Task=`52a79d8d-eb3d-42de-80dd-0737a30bafeb`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9174::DEBUG::2012-12-02 23:34:50,799::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:50,799::task::957::TaskManager.Task::(_decref) Task=`52a79d8d-eb3d-42de-80dd-0737a30bafeb`::ref 1 aborting False >Thread-9174::DEBUG::2012-12-02 23:34:50,799::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9174::DEBUG::2012-12-02 23:34:50,800::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9174::DEBUG::2012-12-02 23:34:50,800::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9174::DEBUG::2012-12-02 23:34:50,802::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9174::DEBUG::2012-12-02 23:34:50,804::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9174::INFO::2012-12-02 23:34:50,805::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-9175::DEBUG::2012-12-02 23:34:50,807::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-9174::DEBUG::2012-12-02 23:34:50,807::task::1151::TaskManager.Task::(prepare) Task=`c2fcb93b-2aa3-48da-8443-b7a802a80b1f`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-9175::DEBUG::2012-12-02 23:34:50,807::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9174::DEBUG::2012-12-02 23:34:50,807::task::568::TaskManager.Task::(_updateState) Task=`c2fcb93b-2aa3-48da-8443-b7a802a80b1f`::moving from state preparing -> state finished >Thread-9175::DEBUG::2012-12-02 23:34:50,807::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-9174::DEBUG::2012-12-02 23:34:50,807::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9175::DEBUG::2012-12-02 23:34:50,808::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9174::DEBUG::2012-12-02 23:34:50,808::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9175::DEBUG::2012-12-02 23:34:50,808::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-9174::DEBUG::2012-12-02 23:34:50,808::task::957::TaskManager.Task::(_decref) Task=`c2fcb93b-2aa3-48da-8443-b7a802a80b1f`::ref 0 aborting False >Thread-9175::DEBUG::2012-12-02 23:34:50,832::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9175::DEBUG::2012-12-02 23:34:50,833::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9175::DEBUG::2012-12-02 23:34:51,467::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-9175::DEBUG::2012-12-02 23:34:51,555::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9175::DEBUG::2012-12-02 23:34:51,556::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,556::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,556::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,556::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,556::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,557::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,557::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9175::DEBUG::2012-12-02 23:34:51,557::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9175::DEBUG::2012-12-02 23:34:51,559::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9175::DEBUG::2012-12-02 23:34:51,767::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9175::WARNING::2012-12-02 23:34:51,769::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9175::DEBUG::2012-12-02 23:34:51,770::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9175::WARNING::2012-12-02 23:34:51,801::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9175::INFO::2012-12-02 23:34:51,801::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-9175::DEBUG::2012-12-02 23:34:51,801::task::1151::TaskManager.Task::(prepare) Task=`52a79d8d-eb3d-42de-80dd-0737a30bafeb`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Attached'}}} >Thread-9175::DEBUG::2012-12-02 23:34:51,801::task::568::TaskManager.Task::(_updateState) Task=`52a79d8d-eb3d-42de-80dd-0737a30bafeb`::moving from state preparing -> state finished >Thread-9175::DEBUG::2012-12-02 23:34:51,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9175::DEBUG::2012-12-02 23:34:51,802::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9175::DEBUG::2012-12-02 23:34:51,802::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9175::DEBUG::2012-12-02 23:34:51,802::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9175::DEBUG::2012-12-02 23:34:51,802::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9175::DEBUG::2012-12-02 23:34:51,802::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9175::DEBUG::2012-12-02 23:34:51,802::task::957::TaskManager.Task::(_decref) Task=`52a79d8d-eb3d-42de-80dd-0737a30bafeb`::ref 0 aborting False >Thread-9238::DEBUG::2012-12-02 23:34:52,019::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9238::DEBUG::2012-12-02 23:34:52,019::task::568::TaskManager.Task::(_updateState) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::moving from state init -> state preparing >Thread-9238::INFO::2012-12-02 23:34:52,019::logUtils::37::dispatcher::(wrapper) Run and protect: detachStorageDomain(sdUUID='5971512c-ee51-475e-b956-3872c38ce193', spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='00000000-0000-0000-0000-000000000000', masterVersion=2, options=None) >Thread-9238::DEBUG::2012-12-02 23:34:52,020::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3e1bd3f4-1d5e-48cd-a55b-ba3345094cc3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9238::DEBUG::2012-12-02 23:34:52,020::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-9238::DEBUG::2012-12-02 23:34:52,020::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-9238::DEBUG::2012-12-02 23:34:52,020::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3e1bd3f4-1d5e-48cd-a55b-ba3345094cc3`::Granted request >Thread-9238::DEBUG::2012-12-02 23:34:52,020::task::794::TaskManager.Task::(resourceAcquired) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-9238::DEBUG::2012-12-02 23:34:52,021::task::957::TaskManager.Task::(_decref) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::ref 1 aborting False >Thread-9238::DEBUG::2012-12-02 23:34:52,021::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`ceee67c5-104f-4c5c-9527-97e8579b2cec`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9238::DEBUG::2012-12-02 23:34:52,021::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' for lock type 'exclusive' >Thread-9238::DEBUG::2012-12-02 23:34:52,021::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free. Now locking as 'exclusive' (1 active user) >Thread-9238::DEBUG::2012-12-02 23:34:52,021::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`ceee67c5-104f-4c5c-9527-97e8579b2cec`::Granted request >Thread-9238::DEBUG::2012-12-02 23:34:52,022::task::794::TaskManager.Task::(resourceAcquired) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::_resourcesAcquired: Storage.5971512c-ee51-475e-b956-3872c38ce193 (exclusive) >Thread-9238::DEBUG::2012-12-02 23:34:52,022::task::957::TaskManager.Task::(_decref) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::ref 1 aborting False >Thread-9238::INFO::2012-12-02 23:34:52,022::sp::978::Storage.StoragePool::(detachSD) sdUUID=5971512c-ee51-475e-b956-3872c38ce193 spUUID=479cf195-bb99-45c5-b400-433d810eaa48 >Thread-9238::DEBUG::2012-12-02 23:34:52,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=148af2e9c9d9dfed2227286a51a8b73853083e95'] >Thread-9238::DEBUG::2012-12-02 23:34:52,036::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-9238::DEBUG::2012-12-02 23:34:52,036::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-9238::DEBUG::2012-12-02 23:34:52,036::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-9238::DEBUG::2012-12-02 23:34:52,036::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-9238::DEBUG::2012-12-02 23:34:52,037::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=ea2a83cb9e6d502aabc1445069efd526bb38c4e4'] >Thread-9238::DEBUG::2012-12-02 23:34:52,081::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-9238::WARNING::2012-12-02 23:34:52,081::sp::951::Storage.StoragePool::(forcedDetachSD) Force detaching domain `5971512c-ee51-475e-b956-3872c38ce193` >Thread-9238::DEBUG::2012-12-02 23:34:52,081::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-9238::DEBUG::2012-12-02 23:34:52,081::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-9238::DEBUG::2012-12-02 23:34:52,081::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-9238::DEBUG::2012-12-02 23:34:52,118::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-9238::DEBUG::2012-12-02 23:34:52,118::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-9238::DEBUG::2012-12-02 23:34:52,119::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9238::DEBUG::2012-12-02 23:34:52,119::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9238::DEBUG::2012-12-02 23:34:52,119::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9238::DEBUG::2012-12-02 23:34:52,326::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9238::WARNING::2012-12-02 23:34:52,328::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9238::DEBUG::2012-12-02 23:34:52,328::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9238::ERROR::2012-12-02 23:34:52,357::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9238::DEBUG::2012-12-02 23:34:52,357::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9238::DEBUG::2012-12-02 23:34:52,357::sp::968::Storage.StoragePool::(forcedDetachSD) Force detach for domain `5971512c-ee51-475e-b956-3872c38ce193` is done >Thread-9238::INFO::2012-12-02 23:34:52,357::logUtils::39::dispatcher::(wrapper) Run and protect: detachStorageDomain, Return response: None >Thread-9238::DEBUG::2012-12-02 23:34:52,357::task::1151::TaskManager.Task::(prepare) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::finished: None >Thread-9238::DEBUG::2012-12-02 23:34:52,357::task::568::TaskManager.Task::(_updateState) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::moving from state preparing -> state finished >Thread-9238::DEBUG::2012-12-02 23:34:52,358::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.5971512c-ee51-475e-b956-3872c38ce193': < ResourceRef 'Storage.5971512c-ee51-475e-b956-3872c38ce193', isValid: 'True' obj: 'None'>, 'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9238::DEBUG::2012-12-02 23:34:52,358::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9238::DEBUG::2012-12-02 23:34:52,358::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' >Thread-9238::DEBUG::2012-12-02 23:34:52,358::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' (0 active users) >Thread-9238::DEBUG::2012-12-02 23:34:52,358::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free, finding out if anyone is waiting for it. >Thread-9238::DEBUG::2012-12-02 23:34:52,358::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193', Clearing records. >Thread-9238::DEBUG::2012-12-02 23:34:52,359::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9238::DEBUG::2012-12-02 23:34:52,359::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9238::DEBUG::2012-12-02 23:34:52,359::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9238::DEBUG::2012-12-02 23:34:52,359::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9238::DEBUG::2012-12-02 23:34:52,359::task::957::TaskManager.Task::(_decref) Task=`ceb2f05f-44be-496c-b589-d3ff96494ad1`::ref 0 aborting False >Thread-9301::DEBUG::2012-12-02 23:34:52,540::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9301::DEBUG::2012-12-02 23:34:52,541::task::568::TaskManager.Task::(_updateState) Task=`a954b3f9-f867-4a81-8d81-befd6b07337a`::moving from state init -> state preparing >Thread-9301::INFO::2012-12-02 23:34:52,541::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-9301::DEBUG::2012-12-02 23:34:52,541::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d509808e-3d22-4d54-89f0-5af6403632e1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9301::DEBUG::2012-12-02 23:34:52,541::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9301::DEBUG::2012-12-02 23:34:52,541::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9301::DEBUG::2012-12-02 23:34:52,542::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d509808e-3d22-4d54-89f0-5af6403632e1`::Granted request >Thread-9301::DEBUG::2012-12-02 23:34:52,542::task::794::TaskManager.Task::(resourceAcquired) Task=`a954b3f9-f867-4a81-8d81-befd6b07337a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9301::DEBUG::2012-12-02 23:34:52,542::task::957::TaskManager.Task::(_decref) Task=`a954b3f9-f867-4a81-8d81-befd6b07337a`::ref 1 aborting False >Thread-9301::INFO::2012-12-02 23:34:52,542::fileSD::260::Storage.StorageDomain::(validate) sdUUID=225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-9301::DEBUG::2012-12-02 23:34:52,552::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-9301::DEBUG::2012-12-02 23:34:52,552::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:52,552::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:52,553::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:52,553::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:52,553::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:52,553::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:52,553::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-9301::DEBUG::2012-12-02 23:34:52,553::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9301::DEBUG::2012-12-02 23:34:52,553::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-9301::DEBUG::2012-12-02 23:34:52,554::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9301::DEBUG::2012-12-02 23:34:52,554::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-9301::DEBUG::2012-12-02 23:34:52,579::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9301::DEBUG::2012-12-02 23:34:52,579::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9302::DEBUG::2012-12-02 23:34:52,932::task::568::TaskManager.Task::(_updateState) Task=`70aba54b-b1b6-4a0d-bf24-0076ab7c4ab8`::moving from state init -> state preparing >Thread-9302::INFO::2012-12-02 23:34:52,932::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9302::INFO::2012-12-02 23:34:52,932::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00849294662476', 'lastCheck': 1354484090.4886889, 'code': 0, 'valid': True}} >Thread-9302::DEBUG::2012-12-02 23:34:52,933::task::1151::TaskManager.Task::(prepare) Task=`70aba54b-b1b6-4a0d-bf24-0076ab7c4ab8`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00849294662476', 'lastCheck': 1354484090.4886889, 'code': 0, 'valid': True}} >Thread-9302::DEBUG::2012-12-02 23:34:52,933::task::568::TaskManager.Task::(_updateState) Task=`70aba54b-b1b6-4a0d-bf24-0076ab7c4ab8`::moving from state preparing -> state finished >Thread-9302::DEBUG::2012-12-02 23:34:52,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9302::DEBUG::2012-12-02 23:34:52,933::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9302::DEBUG::2012-12-02 23:34:52,933::task::957::TaskManager.Task::(_decref) Task=`70aba54b-b1b6-4a0d-bf24-0076ab7c4ab8`::ref 0 aborting False >Thread-9301::DEBUG::2012-12-02 23:34:53,218::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-9301::DEBUG::2012-12-02 23:34:53,297::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9301::DEBUG::2012-12-02 23:34:53,297::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,297::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,298::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,298::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,298::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,298::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,298::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9301::DEBUG::2012-12-02 23:34:53,298::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,300::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9301::DEBUG::2012-12-02 23:34:53,507::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9301::WARNING::2012-12-02 23:34:53,509::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9301::DEBUG::2012-12-02 23:34:53,509::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,543::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-9301::DEBUG::2012-12-02 23:34:53,543::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-9301::DEBUG::2012-12-02 23:34:53,552::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=sdg2', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=2', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'ROLE=Master', 'SDUUID=225bca28-f0a0-4899-b137-32acd04f57d4', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b00a1b8eb074f50877a30cafc70d2fb098a142fe'] >Thread-9301::DEBUG::2012-12-02 23:34:53,555::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-9301::WARNING::2012-12-02 23:34:53,556::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_imageNS already registered >Thread-9301::WARNING::2012-12-02 23:34:53,556::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 225bca28-f0a0-4899-b137-32acd04f57d4_volumeNS already registered >Thread-9301::DEBUG::2012-12-02 23:34:53,556::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 225bca28-f0a0-4899-b137-32acd04f57d4 verified, version 2 >Thread-9301::DEBUG::2012-12-02 23:34:53,556::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-9301::DEBUG::2012-12-02 23:34:53,556::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9301::DEBUG::2012-12-02 23:34:53,557::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,557::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9301::DEBUG::2012-12-02 23:34:53,767::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9301::WARNING::2012-12-02 23:34:53,768::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9301::DEBUG::2012-12-02 23:34:53,769::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9301::ERROR::2012-12-02 23:34:53,798::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9301::DEBUG::2012-12-02 23:34:53,798::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9301::DEBUG::2012-12-02 23:34:53,798::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9301::DEBUG::2012-12-02 23:34:53,799::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9301::DEBUG::2012-12-02 23:34:54,007::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9301::WARNING::2012-12-02 23:34:54,008::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9301::DEBUG::2012-12-02 23:34:54,008::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9301::INFO::2012-12-02 23:34:54,036::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-9301::INFO::2012-12-02 23:34:54,036::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-9301::INFO::2012-12-02 23:34:54,036::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-9301::DEBUG::2012-12-02 23:34:54,037::task::1151::TaskManager.Task::(prepare) Task=`a954b3f9-f867-4a81-8d81-befd6b07337a`::finished: None >Thread-9301::DEBUG::2012-12-02 23:34:54,037::task::568::TaskManager.Task::(_updateState) Task=`a954b3f9-f867-4a81-8d81-befd6b07337a`::moving from state preparing -> state finished >Thread-9301::DEBUG::2012-12-02 23:34:54,037::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9301::DEBUG::2012-12-02 23:34:54,037::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9301::DEBUG::2012-12-02 23:34:54,037::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9301::DEBUG::2012-12-02 23:34:54,037::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9301::DEBUG::2012-12-02 23:34:54,038::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9301::DEBUG::2012-12-02 23:34:54,038::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9301::DEBUG::2012-12-02 23:34:54,038::task::957::TaskManager.Task::(_decref) Task=`a954b3f9-f867-4a81-8d81-befd6b07337a`::ref 0 aborting False >Thread-9489::DEBUG::2012-12-02 23:34:54,194::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9489::DEBUG::2012-12-02 23:34:54,195::task::568::TaskManager.Task::(_updateState) Task=`e57d3554-5973-4ec8-8930-04eb786e7413`::moving from state init -> state preparing >Thread-9489::INFO::2012-12-02 23:34:54,195::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-9489::DEBUG::2012-12-02 23:34:54,195::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328' (cwd None) >Thread-9489::DEBUG::2012-12-02 23:34:54,216::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-9489::DEBUG::2012-12-02 23:34:54,216::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9489::DEBUG::2012-12-02 23:34:54,216::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-9489::DEBUG::2012-12-02 23:34:54,217::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9489::DEBUG::2012-12-02 23:34:54,217::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-9489::DEBUG::2012-12-02 23:34:54,244::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9489::DEBUG::2012-12-02 23:34:54,244::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9489::DEBUG::2012-12-02 23:34:54,877::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-9489::DEBUG::2012-12-02 23:34:54,971::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9489::DEBUG::2012-12-02 23:34:54,972::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9489::DEBUG::2012-12-02 23:34:54,972::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9489::DEBUG::2012-12-02 23:34:54,972::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9489::DEBUG::2012-12-02 23:34:54,972::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9489::DEBUG::2012-12-02 23:34:54,972::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9489::DEBUG::2012-12-02 23:34:54,972::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9489::DEBUG::2012-12-02 23:34:54,972::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9489::INFO::2012-12-02 23:34:54,972::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-9489::DEBUG::2012-12-02 23:34:54,973::task::1151::TaskManager.Task::(prepare) Task=`e57d3554-5973-4ec8-8930-04eb786e7413`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-9489::DEBUG::2012-12-02 23:34:54,973::task::568::TaskManager.Task::(_updateState) Task=`e57d3554-5973-4ec8-8930-04eb786e7413`::moving from state preparing -> state finished >Thread-9489::DEBUG::2012-12-02 23:34:54,973::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9489::DEBUG::2012-12-02 23:34:54,973::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9489::DEBUG::2012-12-02 23:34:54,973::task::957::TaskManager.Task::(_decref) Task=`e57d3554-5973-4ec8-8930-04eb786e7413`::ref 0 aborting False >Thread-9494::DEBUG::2012-12-02 23:35:02,031::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9494::DEBUG::2012-12-02 23:35:02,031::task::568::TaskManager.Task::(_updateState) Task=`821e9619-be80-4e3a-abcc-967a5352ee3f`::moving from state init -> state preparing >Thread-9494::INFO::2012-12-02 23:35:02,031::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9494::INFO::2012-12-02 23:35:02,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9494::DEBUG::2012-12-02 23:35:02,032::task::1151::TaskManager.Task::(prepare) Task=`821e9619-be80-4e3a-abcc-967a5352ee3f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9494::DEBUG::2012-12-02 23:35:02,032::task::568::TaskManager.Task::(_updateState) Task=`821e9619-be80-4e3a-abcc-967a5352ee3f`::moving from state preparing -> state finished >Thread-9494::DEBUG::2012-12-02 23:35:02,032::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9494::DEBUG::2012-12-02 23:35:02,032::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9494::DEBUG::2012-12-02 23:35:02,032::task::957::TaskManager.Task::(_decref) Task=`821e9619-be80-4e3a-abcc-967a5352ee3f`::ref 0 aborting False >Thread-9495::DEBUG::2012-12-02 23:35:02,167::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9495::DEBUG::2012-12-02 23:35:02,167::task::568::TaskManager.Task::(_updateState) Task=`1a0e6510-ea62-477e-8008-cfe044987868`::moving from state init -> state preparing >Thread-9495::INFO::2012-12-02 23:35:02,167::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9495::DEBUG::2012-12-02 23:35:02,168::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7bf20030-e8fd-4153-88fd-cc87c73341f1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9495::DEBUG::2012-12-02 23:35:02,168::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9495::DEBUG::2012-12-02 23:35:02,168::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9495::DEBUG::2012-12-02 23:35:02,168::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7bf20030-e8fd-4153-88fd-cc87c73341f1`::Granted request >Thread-9495::DEBUG::2012-12-02 23:35:02,168::task::794::TaskManager.Task::(resourceAcquired) Task=`1a0e6510-ea62-477e-8008-cfe044987868`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9495::DEBUG::2012-12-02 23:35:02,168::task::957::TaskManager.Task::(_decref) Task=`1a0e6510-ea62-477e-8008-cfe044987868`::ref 1 aborting False >Thread-9495::DEBUG::2012-12-02 23:35:02,175::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9495::DEBUG::2012-12-02 23:35:02,176::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9495::DEBUG::2012-12-02 23:35:02,380::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9495::WARNING::2012-12-02 23:35:02,382::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9495::DEBUG::2012-12-02 23:35:02,382::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9495::WARNING::2012-12-02 23:35:02,411::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9495::INFO::2012-12-02 23:35:02,411::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9495::DEBUG::2012-12-02 23:35:02,411::task::1151::TaskManager.Task::(prepare) Task=`1a0e6510-ea62-477e-8008-cfe044987868`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9495::DEBUG::2012-12-02 23:35:02,412::task::568::TaskManager.Task::(_updateState) Task=`1a0e6510-ea62-477e-8008-cfe044987868`::moving from state preparing -> state finished >Thread-9495::DEBUG::2012-12-02 23:35:02,412::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9495::DEBUG::2012-12-02 23:35:02,412::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9495::DEBUG::2012-12-02 23:35:02,412::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9495::DEBUG::2012-12-02 23:35:02,412::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9495::DEBUG::2012-12-02 23:35:02,412::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9495::DEBUG::2012-12-02 23:35:02,413::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9495::DEBUG::2012-12-02 23:35:02,413::task::957::TaskManager.Task::(_decref) Task=`1a0e6510-ea62-477e-8008-cfe044987868`::ref 0 aborting False >Thread-9557::DEBUG::2012-12-02 23:35:03,922::task::568::TaskManager.Task::(_updateState) Task=`a2ba1a81-2125-48e1-a6b2-803c963b3ff8`::moving from state init -> state preparing >Thread-9557::INFO::2012-12-02 23:35:03,922::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9557::INFO::2012-12-02 23:35:03,922::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00753688812256', 'lastCheck': 1354484100.510792, 'code': 0, 'valid': True}} >Thread-9557::DEBUG::2012-12-02 23:35:03,922::task::1151::TaskManager.Task::(prepare) Task=`a2ba1a81-2125-48e1-a6b2-803c963b3ff8`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00753688812256', 'lastCheck': 1354484100.510792, 'code': 0, 'valid': True}} >Thread-9557::DEBUG::2012-12-02 23:35:03,922::task::568::TaskManager.Task::(_updateState) Task=`a2ba1a81-2125-48e1-a6b2-803c963b3ff8`::moving from state preparing -> state finished >Thread-9557::DEBUG::2012-12-02 23:35:03,922::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9557::DEBUG::2012-12-02 23:35:03,923::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9557::DEBUG::2012-12-02 23:35:03,923::task::957::TaskManager.Task::(_decref) Task=`a2ba1a81-2125-48e1-a6b2-803c963b3ff8`::ref 0 aborting False >Thread-9562::DEBUG::2012-12-02 23:35:12,630::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9562::DEBUG::2012-12-02 23:35:12,630::task::568::TaskManager.Task::(_updateState) Task=`cb2cf563-a4be-41b9-9f00-d015d9ec227c`::moving from state init -> state preparing >Thread-9562::INFO::2012-12-02 23:35:12,630::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9562::INFO::2012-12-02 23:35:12,630::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9562::DEBUG::2012-12-02 23:35:12,630::task::1151::TaskManager.Task::(prepare) Task=`cb2cf563-a4be-41b9-9f00-d015d9ec227c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9562::DEBUG::2012-12-02 23:35:12,631::task::568::TaskManager.Task::(_updateState) Task=`cb2cf563-a4be-41b9-9f00-d015d9ec227c`::moving from state preparing -> state finished >Thread-9562::DEBUG::2012-12-02 23:35:12,631::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9562::DEBUG::2012-12-02 23:35:12,631::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9562::DEBUG::2012-12-02 23:35:12,631::task::957::TaskManager.Task::(_decref) Task=`cb2cf563-a4be-41b9-9f00-d015d9ec227c`::ref 0 aborting False >Thread-9564::DEBUG::2012-12-02 23:35:12,773::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9564::DEBUG::2012-12-02 23:35:12,773::task::568::TaskManager.Task::(_updateState) Task=`e323baa3-04ad-49ac-9818-8654909e3d54`::moving from state init -> state preparing >Thread-9564::INFO::2012-12-02 23:35:12,774::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9564::DEBUG::2012-12-02 23:35:12,774::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4eac6e5c-41c9-4b8b-9909-60dd3c22b7af`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9564::DEBUG::2012-12-02 23:35:12,774::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9564::DEBUG::2012-12-02 23:35:12,774::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9564::DEBUG::2012-12-02 23:35:12,774::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4eac6e5c-41c9-4b8b-9909-60dd3c22b7af`::Granted request >Thread-9564::DEBUG::2012-12-02 23:35:12,775::task::794::TaskManager.Task::(resourceAcquired) Task=`e323baa3-04ad-49ac-9818-8654909e3d54`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9564::DEBUG::2012-12-02 23:35:12,775::task::957::TaskManager.Task::(_decref) Task=`e323baa3-04ad-49ac-9818-8654909e3d54`::ref 1 aborting False >Thread-9564::DEBUG::2012-12-02 23:35:12,779::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9564::DEBUG::2012-12-02 23:35:12,780::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9564::DEBUG::2012-12-02 23:35:12,985::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9564::WARNING::2012-12-02 23:35:12,986::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9564::DEBUG::2012-12-02 23:35:12,987::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9564::WARNING::2012-12-02 23:35:13,016::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9564::INFO::2012-12-02 23:35:13,016::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9564::DEBUG::2012-12-02 23:35:13,017::task::1151::TaskManager.Task::(prepare) Task=`e323baa3-04ad-49ac-9818-8654909e3d54`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9564::DEBUG::2012-12-02 23:35:13,017::task::568::TaskManager.Task::(_updateState) Task=`e323baa3-04ad-49ac-9818-8654909e3d54`::moving from state preparing -> state finished >Thread-9564::DEBUG::2012-12-02 23:35:13,017::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9564::DEBUG::2012-12-02 23:35:13,017::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9564::DEBUG::2012-12-02 23:35:13,017::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9564::DEBUG::2012-12-02 23:35:13,018::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9564::DEBUG::2012-12-02 23:35:13,018::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9564::DEBUG::2012-12-02 23:35:13,018::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9564::DEBUG::2012-12-02 23:35:13,018::task::957::TaskManager.Task::(_decref) Task=`e323baa3-04ad-49ac-9818-8654909e3d54`::ref 0 aborting False >Thread-9626::DEBUG::2012-12-02 23:35:14,859::task::568::TaskManager.Task::(_updateState) Task=`ee2d4096-b3cb-484e-9482-4eb2763ec019`::moving from state init -> state preparing >Thread-9626::INFO::2012-12-02 23:35:14,859::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9626::INFO::2012-12-02 23:35:14,859::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0064799785614', 'lastCheck': 1354484110.529428, 'code': 0, 'valid': True}} >Thread-9626::DEBUG::2012-12-02 23:35:14,859::task::1151::TaskManager.Task::(prepare) Task=`ee2d4096-b3cb-484e-9482-4eb2763ec019`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.0064799785614', 'lastCheck': 1354484110.529428, 'code': 0, 'valid': True}} >Thread-9626::DEBUG::2012-12-02 23:35:14,859::task::568::TaskManager.Task::(_updateState) Task=`ee2d4096-b3cb-484e-9482-4eb2763ec019`::moving from state preparing -> state finished >Thread-9626::DEBUG::2012-12-02 23:35:14,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9626::DEBUG::2012-12-02 23:35:14,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9626::DEBUG::2012-12-02 23:35:14,860::task::957::TaskManager.Task::(_decref) Task=`ee2d4096-b3cb-484e-9482-4eb2763ec019`::ref 0 aborting False >Thread-9631::DEBUG::2012-12-02 23:35:23,230::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9631::DEBUG::2012-12-02 23:35:23,230::task::568::TaskManager.Task::(_updateState) Task=`7f37116e-06a6-4d99-874b-474d65d7860b`::moving from state init -> state preparing >Thread-9631::INFO::2012-12-02 23:35:23,230::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9631::INFO::2012-12-02 23:35:23,231::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9631::DEBUG::2012-12-02 23:35:23,231::task::1151::TaskManager.Task::(prepare) Task=`7f37116e-06a6-4d99-874b-474d65d7860b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9631::DEBUG::2012-12-02 23:35:23,231::task::568::TaskManager.Task::(_updateState) Task=`7f37116e-06a6-4d99-874b-474d65d7860b`::moving from state preparing -> state finished >Thread-9631::DEBUG::2012-12-02 23:35:23,231::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9631::DEBUG::2012-12-02 23:35:23,231::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9631::DEBUG::2012-12-02 23:35:23,231::task::957::TaskManager.Task::(_decref) Task=`7f37116e-06a6-4d99-874b-474d65d7860b`::ref 0 aborting False >Thread-9632::DEBUG::2012-12-02 23:35:23,369::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9632::DEBUG::2012-12-02 23:35:23,370::task::568::TaskManager.Task::(_updateState) Task=`b6729de9-8629-4f02-9e4e-2af17f7be81a`::moving from state init -> state preparing >Thread-9632::INFO::2012-12-02 23:35:23,370::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9632::DEBUG::2012-12-02 23:35:23,370::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4f048186-56db-4a18-9f3c-fdd73ada28ff`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9632::DEBUG::2012-12-02 23:35:23,370::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9632::DEBUG::2012-12-02 23:35:23,370::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9632::DEBUG::2012-12-02 23:35:23,371::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4f048186-56db-4a18-9f3c-fdd73ada28ff`::Granted request >Thread-9632::DEBUG::2012-12-02 23:35:23,371::task::794::TaskManager.Task::(resourceAcquired) Task=`b6729de9-8629-4f02-9e4e-2af17f7be81a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9632::DEBUG::2012-12-02 23:35:23,371::task::957::TaskManager.Task::(_decref) Task=`b6729de9-8629-4f02-9e4e-2af17f7be81a`::ref 1 aborting False >Thread-9632::DEBUG::2012-12-02 23:35:23,377::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9632::DEBUG::2012-12-02 23:35:23,377::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9632::DEBUG::2012-12-02 23:35:23,581::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9632::WARNING::2012-12-02 23:35:23,583::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9632::DEBUG::2012-12-02 23:35:23,583::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9632::WARNING::2012-12-02 23:35:23,617::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9632::INFO::2012-12-02 23:35:23,618::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9632::DEBUG::2012-12-02 23:35:23,618::task::1151::TaskManager.Task::(prepare) Task=`b6729de9-8629-4f02-9e4e-2af17f7be81a`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9632::DEBUG::2012-12-02 23:35:23,618::task::568::TaskManager.Task::(_updateState) Task=`b6729de9-8629-4f02-9e4e-2af17f7be81a`::moving from state preparing -> state finished >Thread-9632::DEBUG::2012-12-02 23:35:23,618::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9632::DEBUG::2012-12-02 23:35:23,618::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9632::DEBUG::2012-12-02 23:35:23,618::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9632::DEBUG::2012-12-02 23:35:23,619::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9632::DEBUG::2012-12-02 23:35:23,619::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9632::DEBUG::2012-12-02 23:35:23,619::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9632::DEBUG::2012-12-02 23:35:23,619::task::957::TaskManager.Task::(_decref) Task=`b6729de9-8629-4f02-9e4e-2af17f7be81a`::ref 0 aborting False >Thread-9695::DEBUG::2012-12-02 23:35:25,806::task::568::TaskManager.Task::(_updateState) Task=`0b55d57a-e9d0-4067-80ae-b3db757ccdcc`::moving from state init -> state preparing >Thread-9695::INFO::2012-12-02 23:35:25,806::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9695::INFO::2012-12-02 23:35:25,807::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00615906715393', 'lastCheck': 1354484120.545728, 'code': 0, 'valid': True}} >Thread-9695::DEBUG::2012-12-02 23:35:25,807::task::1151::TaskManager.Task::(prepare) Task=`0b55d57a-e9d0-4067-80ae-b3db757ccdcc`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0.00615906715393', 'lastCheck': 1354484120.545728, 'code': 0, 'valid': True}} >Thread-9695::DEBUG::2012-12-02 23:35:25,807::task::568::TaskManager.Task::(_updateState) Task=`0b55d57a-e9d0-4067-80ae-b3db757ccdcc`::moving from state preparing -> state finished >Thread-9695::DEBUG::2012-12-02 23:35:25,807::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9695::DEBUG::2012-12-02 23:35:25,807::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9695::DEBUG::2012-12-02 23:35:25,807::task::957::TaskManager.Task::(_decref) Task=`0b55d57a-e9d0-4067-80ae-b3db757ccdcc`::ref 0 aborting False >Thread-5152::DEBUG::2012-12-02 23:35:30,553::fileSD::405::Storage.StorageDomain::(selftest) Unmounting stale file system /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252 >Thread-5152::DEBUG::2012-12-02 23:35:30,559::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252' (cwd None) >Thread-5152::ERROR::2012-12-02 23:35:30,571::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `225bca28-f0a0-4899-b137-32acd04f57d4` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 406, in selftest > mount.getMountFromTarget(self.mountpoint).umount() > File "/usr/share/vdsm/storage/mount.py", line 218, in umount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 206, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (16, ';umount.nfs: /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252: device is busy\numount.nfs: /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252: device is busy\n') >Thread-5152::DEBUG::2012-12-02 23:35:30,572::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `225bca28-f0a0-4899-b137-32acd04f57d4` changed its status to Invalid >Thread-9699::DEBUG::2012-12-02 23:35:30,572::misc::1183::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Emitting event >Thread-9699::DEBUG::2012-12-02 23:35:30,573::misc::1194::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Calling registered method `<functools.partial object at 0x7f19184d5998>` >Thread-9699::DEBUG::2012-12-02 23:35:30,573::misc::1204::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Event emitted >Thread-9702::DEBUG::2012-12-02 23:35:33,798::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9702::DEBUG::2012-12-02 23:35:33,799::task::568::TaskManager.Task::(_updateState) Task=`1c642946-506d-4e99-b9f5-a2686e67519a`::moving from state init -> state preparing >Thread-9702::INFO::2012-12-02 23:35:33,799::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9702::INFO::2012-12-02 23:35:33,799::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9702::DEBUG::2012-12-02 23:35:33,799::task::1151::TaskManager.Task::(prepare) Task=`1c642946-506d-4e99-b9f5-a2686e67519a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9702::DEBUG::2012-12-02 23:35:33,799::task::568::TaskManager.Task::(_updateState) Task=`1c642946-506d-4e99-b9f5-a2686e67519a`::moving from state preparing -> state finished >Thread-9702::DEBUG::2012-12-02 23:35:33,799::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9702::DEBUG::2012-12-02 23:35:33,800::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9702::DEBUG::2012-12-02 23:35:33,800::task::957::TaskManager.Task::(_decref) Task=`1c642946-506d-4e99-b9f5-a2686e67519a`::ref 0 aborting False >Thread-9703::DEBUG::2012-12-02 23:35:33,941::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9703::DEBUG::2012-12-02 23:35:33,942::task::568::TaskManager.Task::(_updateState) Task=`567557d5-24b1-4c65-9f3e-69a3ab56e2b8`::moving from state init -> state preparing >Thread-9703::INFO::2012-12-02 23:35:33,942::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9703::DEBUG::2012-12-02 23:35:33,942::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ddba2865-7a0d-44b8-84b6-520a47332175`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9703::DEBUG::2012-12-02 23:35:33,942::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9703::DEBUG::2012-12-02 23:35:33,943::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9703::DEBUG::2012-12-02 23:35:33,943::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ddba2865-7a0d-44b8-84b6-520a47332175`::Granted request >Thread-9703::DEBUG::2012-12-02 23:35:33,943::task::794::TaskManager.Task::(resourceAcquired) Task=`567557d5-24b1-4c65-9f3e-69a3ab56e2b8`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9703::DEBUG::2012-12-02 23:35:33,943::task::957::TaskManager.Task::(_decref) Task=`567557d5-24b1-4c65-9f3e-69a3ab56e2b8`::ref 1 aborting False >Thread-9703::DEBUG::2012-12-02 23:35:33,948::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9703::DEBUG::2012-12-02 23:35:33,949::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9703::DEBUG::2012-12-02 23:35:34,154::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9703::WARNING::2012-12-02 23:35:34,156::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9703::DEBUG::2012-12-02 23:35:34,156::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9703::WARNING::2012-12-02 23:35:34,185::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9703::INFO::2012-12-02 23:35:34,186::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'alerts': []}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9703::DEBUG::2012-12-02 23:35:34,186::task::1151::TaskManager.Task::(prepare) Task=`567557d5-24b1-4c65-9f3e-69a3ab56e2b8`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'alerts': []}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9703::DEBUG::2012-12-02 23:35:34,186::task::568::TaskManager.Task::(_updateState) Task=`567557d5-24b1-4c65-9f3e-69a3ab56e2b8`::moving from state preparing -> state finished >Thread-9703::DEBUG::2012-12-02 23:35:34,186::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9703::DEBUG::2012-12-02 23:35:34,186::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9703::DEBUG::2012-12-02 23:35:34,187::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9703::DEBUG::2012-12-02 23:35:34,187::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9703::DEBUG::2012-12-02 23:35:34,187::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9703::DEBUG::2012-12-02 23:35:34,187::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9703::DEBUG::2012-12-02 23:35:34,187::task::957::TaskManager.Task::(_decref) Task=`567557d5-24b1-4c65-9f3e-69a3ab56e2b8`::ref 0 aborting False >Thread-9766::DEBUG::2012-12-02 23:35:36,817::task::568::TaskManager.Task::(_updateState) Task=`3492095d-fe73-45e1-ba6a-b1814f3e0072`::moving from state init -> state preparing >Thread-9766::INFO::2012-12-02 23:35:36,817::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9766::INFO::2012-12-02 23:35:36,817::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484130.5724821, 'code': 200, 'valid': False}} >Thread-9766::DEBUG::2012-12-02 23:35:36,818::task::1151::TaskManager.Task::(prepare) Task=`3492095d-fe73-45e1-ba6a-b1814f3e0072`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484130.5724821, 'code': 200, 'valid': False}} >Thread-9766::DEBUG::2012-12-02 23:35:36,818::task::568::TaskManager.Task::(_updateState) Task=`3492095d-fe73-45e1-ba6a-b1814f3e0072`::moving from state preparing -> state finished >Thread-9766::DEBUG::2012-12-02 23:35:36,818::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9766::DEBUG::2012-12-02 23:35:36,818::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9766::DEBUG::2012-12-02 23:35:36,818::task::957::TaskManager.Task::(_decref) Task=`3492095d-fe73-45e1-ba6a-b1814f3e0072`::ref 0 aborting False >Thread-5152::ERROR::2012-12-02 23:35:40,582::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `225bca28-f0a0-4899-b137-32acd04f57d4` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-9771::DEBUG::2012-12-02 23:35:44,399::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9771::DEBUG::2012-12-02 23:35:44,399::task::568::TaskManager.Task::(_updateState) Task=`3e27b8df-80c3-4761-b98d-eaca90412b22`::moving from state init -> state preparing >Thread-9771::INFO::2012-12-02 23:35:44,400::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9771::INFO::2012-12-02 23:35:44,400::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9771::DEBUG::2012-12-02 23:35:44,400::task::1151::TaskManager.Task::(prepare) Task=`3e27b8df-80c3-4761-b98d-eaca90412b22`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9771::DEBUG::2012-12-02 23:35:44,400::task::568::TaskManager.Task::(_updateState) Task=`3e27b8df-80c3-4761-b98d-eaca90412b22`::moving from state preparing -> state finished >Thread-9771::DEBUG::2012-12-02 23:35:44,400::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9771::DEBUG::2012-12-02 23:35:44,400::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9771::DEBUG::2012-12-02 23:35:44,401::task::957::TaskManager.Task::(_decref) Task=`3e27b8df-80c3-4761-b98d-eaca90412b22`::ref 0 aborting False >Thread-9772::DEBUG::2012-12-02 23:35:44,543::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9772::DEBUG::2012-12-02 23:35:44,543::task::568::TaskManager.Task::(_updateState) Task=`50e4cea2-9a66-4fe4-bb08-4d3ae3f85f49`::moving from state init -> state preparing >Thread-9772::INFO::2012-12-02 23:35:44,544::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9772::DEBUG::2012-12-02 23:35:44,544::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`40a4d276-eae0-411e-9907-8f5931a45113`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9772::DEBUG::2012-12-02 23:35:44,544::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9772::DEBUG::2012-12-02 23:35:44,544::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9772::DEBUG::2012-12-02 23:35:44,544::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`40a4d276-eae0-411e-9907-8f5931a45113`::Granted request >Thread-9772::DEBUG::2012-12-02 23:35:44,545::task::794::TaskManager.Task::(resourceAcquired) Task=`50e4cea2-9a66-4fe4-bb08-4d3ae3f85f49`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9772::DEBUG::2012-12-02 23:35:44,545::task::957::TaskManager.Task::(_decref) Task=`50e4cea2-9a66-4fe4-bb08-4d3ae3f85f49`::ref 1 aborting False >Thread-9772::DEBUG::2012-12-02 23:35:44,549::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9772::DEBUG::2012-12-02 23:35:44,550::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9772::DEBUG::2012-12-02 23:35:44,755::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9772::WARNING::2012-12-02 23:35:44,756::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9772::DEBUG::2012-12-02 23:35:44,757::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9772::WARNING::2012-12-02 23:35:44,787::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9772::INFO::2012-12-02 23:35:44,787::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'alerts': []}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9772::DEBUG::2012-12-02 23:35:44,787::task::1151::TaskManager.Task::(prepare) Task=`50e4cea2-9a66-4fe4-bb08-4d3ae3f85f49`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'alerts': []}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9772::DEBUG::2012-12-02 23:35:44,788::task::568::TaskManager.Task::(_updateState) Task=`50e4cea2-9a66-4fe4-bb08-4d3ae3f85f49`::moving from state preparing -> state finished >Thread-9772::DEBUG::2012-12-02 23:35:44,788::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9772::DEBUG::2012-12-02 23:35:44,788::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9772::DEBUG::2012-12-02 23:35:44,788::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9772::DEBUG::2012-12-02 23:35:44,788::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9772::DEBUG::2012-12-02 23:35:44,788::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9772::DEBUG::2012-12-02 23:35:44,788::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9772::DEBUG::2012-12-02 23:35:44,789::task::957::TaskManager.Task::(_decref) Task=`50e4cea2-9a66-4fe4-bb08-4d3ae3f85f49`::ref 0 aborting False >Thread-9835::DEBUG::2012-12-02 23:35:47,762::task::568::TaskManager.Task::(_updateState) Task=`a8373f0a-6dde-4c72-8884-aee0708cec66`::moving from state init -> state preparing >Thread-9835::INFO::2012-12-02 23:35:47,762::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9835::INFO::2012-12-02 23:35:47,762::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484140.5823481, 'code': 200, 'valid': False}} >Thread-9835::DEBUG::2012-12-02 23:35:47,762::task::1151::TaskManager.Task::(prepare) Task=`a8373f0a-6dde-4c72-8884-aee0708cec66`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484140.5823481, 'code': 200, 'valid': False}} >Thread-9835::DEBUG::2012-12-02 23:35:47,762::task::568::TaskManager.Task::(_updateState) Task=`a8373f0a-6dde-4c72-8884-aee0708cec66`::moving from state preparing -> state finished >Thread-9835::DEBUG::2012-12-02 23:35:47,762::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9835::DEBUG::2012-12-02 23:35:47,763::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9835::DEBUG::2012-12-02 23:35:47,763::task::957::TaskManager.Task::(_decref) Task=`a8373f0a-6dde-4c72-8884-aee0708cec66`::ref 0 aborting False >Thread-5152::ERROR::2012-12-02 23:35:50,588::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `225bca28-f0a0-4899-b137-32acd04f57d4` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252/225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-9840::DEBUG::2012-12-02 23:35:54,998::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9840::DEBUG::2012-12-02 23:35:54,999::task::568::TaskManager.Task::(_updateState) Task=`2e8fca26-7cb4-4e42-bb8b-d2672cdd1921`::moving from state init -> state preparing >Thread-9840::INFO::2012-12-02 23:35:54,999::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9840::INFO::2012-12-02 23:35:54,999::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9840::DEBUG::2012-12-02 23:35:54,999::task::1151::TaskManager.Task::(prepare) Task=`2e8fca26-7cb4-4e42-bb8b-d2672cdd1921`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9840::DEBUG::2012-12-02 23:35:55,000::task::568::TaskManager.Task::(_updateState) Task=`2e8fca26-7cb4-4e42-bb8b-d2672cdd1921`::moving from state preparing -> state finished >Thread-9840::DEBUG::2012-12-02 23:35:55,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9840::DEBUG::2012-12-02 23:35:55,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9840::DEBUG::2012-12-02 23:35:55,000::task::957::TaskManager.Task::(_decref) Task=`2e8fca26-7cb4-4e42-bb8b-d2672cdd1921`::ref 0 aborting False >Thread-9841::DEBUG::2012-12-02 23:35:55,139::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9841::DEBUG::2012-12-02 23:35:55,139::task::568::TaskManager.Task::(_updateState) Task=`d2a3ae38-34a7-4620-a9f3-91b8a49bffb8`::moving from state init -> state preparing >Thread-9841::INFO::2012-12-02 23:35:55,140::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9841::DEBUG::2012-12-02 23:35:55,140::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`35b95796-4318-490e-ac4f-aa7fb947e470`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9841::DEBUG::2012-12-02 23:35:55,140::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9841::DEBUG::2012-12-02 23:35:55,140::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9841::DEBUG::2012-12-02 23:35:55,140::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`35b95796-4318-490e-ac4f-aa7fb947e470`::Granted request >Thread-9841::DEBUG::2012-12-02 23:35:55,141::task::794::TaskManager.Task::(resourceAcquired) Task=`d2a3ae38-34a7-4620-a9f3-91b8a49bffb8`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9841::DEBUG::2012-12-02 23:35:55,141::task::957::TaskManager.Task::(_decref) Task=`d2a3ae38-34a7-4620-a9f3-91b8a49bffb8`::ref 1 aborting False >Thread-9841::DEBUG::2012-12-02 23:35:55,146::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9841::DEBUG::2012-12-02 23:35:55,147::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9841::DEBUG::2012-12-02 23:35:55,351::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9841::WARNING::2012-12-02 23:35:55,353::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9841::DEBUG::2012-12-02 23:35:55,353::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9841::WARNING::2012-12-02 23:35:55,383::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9841::INFO::2012-12-02 23:35:55,383::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'alerts': []}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9841::DEBUG::2012-12-02 23:35:55,383::task::1151::TaskManager.Task::(prepare) Task=`d2a3ae38-34a7-4620-a9f3-91b8a49bffb8`::finished: {'info': {'spm_id': 2, 'master_uuid': '225bca28-f0a0-4899-b137-32acd04f57d4', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 2, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active', 'alerts': []}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-9841::DEBUG::2012-12-02 23:35:55,383::task::568::TaskManager.Task::(_updateState) Task=`d2a3ae38-34a7-4620-a9f3-91b8a49bffb8`::moving from state preparing -> state finished >Thread-9841::DEBUG::2012-12-02 23:35:55,384::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9841::DEBUG::2012-12-02 23:35:55,384::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9841::DEBUG::2012-12-02 23:35:55,384::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9841::DEBUG::2012-12-02 23:35:55,384::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9841::DEBUG::2012-12-02 23:35:55,384::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9841::DEBUG::2012-12-02 23:35:55,384::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9841::DEBUG::2012-12-02 23:35:55,385::task::957::TaskManager.Task::(_decref) Task=`d2a3ae38-34a7-4620-a9f3-91b8a49bffb8`::ref 0 aborting False >Thread-9904::DEBUG::2012-12-02 23:35:58,696::task::568::TaskManager.Task::(_updateState) Task=`f27bddb9-8d7a-4332-84e2-bf88bee0ec21`::moving from state init -> state preparing >Thread-9904::INFO::2012-12-02 23:35:58,696::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9904::INFO::2012-12-02 23:35:58,696::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-9904::DEBUG::2012-12-02 23:35:58,696::task::1151::TaskManager.Task::(prepare) Task=`f27bddb9-8d7a-4332-84e2-bf88bee0ec21`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-9904::DEBUG::2012-12-02 23:35:58,696::task::568::TaskManager.Task::(_updateState) Task=`f27bddb9-8d7a-4332-84e2-bf88bee0ec21`::moving from state preparing -> state finished >Thread-9904::DEBUG::2012-12-02 23:35:58,696::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9904::DEBUG::2012-12-02 23:35:58,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9904::DEBUG::2012-12-02 23:35:58,697::task::957::TaskManager.Task::(_decref) Task=`f27bddb9-8d7a-4332-84e2-bf88bee0ec21`::ref 0 aborting False >Thread-5152::DEBUG::2012-12-02 23:36:00,589::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 225bca28-f0a0-4899-b137-32acd04f57d4 >Thread-5152::DEBUG::2012-12-02 23:36:00,589::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5152::DEBUG::2012-12-02 23:36:00,590::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5152::DEBUG::2012-12-02 23:36:00,795::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5152::WARNING::2012-12-02 23:36:00,797::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5152::DEBUG::2012-12-02 23:36:00,797::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5152::DEBUG::2012-12-02 23:36:00,827::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5152::DEBUG::2012-12-02 23:36:00,827::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5152::DEBUG::2012-12-02 23:36:01,033::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5152::WARNING::2012-12-02 23:36:01,035::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5152::DEBUG::2012-12-02 23:36:01,035::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10031::DEBUG::2012-12-02 23:36:05,595::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10031::DEBUG::2012-12-02 23:36:05,595::task::568::TaskManager.Task::(_updateState) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::moving from state init -> state preparing >Thread-10031::INFO::2012-12-02 23:36:05,595::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10031::DEBUG::2012-12-02 23:36:05,596::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10031::DEBUG::2012-12-02 23:36:05,596::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10031::DEBUG::2012-12-02 23:36:05,803::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10031::WARNING::2012-12-02 23:36:05,805::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10031::DEBUG::2012-12-02 23:36:05,805::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10031::ERROR::2012-12-02 23:36:05,834::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10031::ERROR::2012-12-02 23:36:05,835::task::833::TaskManager.Task::(_setError) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-10031::DEBUG::2012-12-02 23:36:05,835::task::852::TaskManager.Task::(_run) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::Task._run: 549a2652-31f3-4d15-b681-78b8c11da9dd ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-10031::DEBUG::2012-12-02 23:36:05,835::task::1177::TaskManager.Task::(stop) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::stopping in state preparing (force False) >Thread-10031::DEBUG::2012-12-02 23:36:05,835::task::957::TaskManager.Task::(_decref) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::ref 1 aborting True >Thread-10031::INFO::2012-12-02 23:36:05,836::task::1134::TaskManager.Task::(prepare) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-10031::DEBUG::2012-12-02 23:36:05,836::task::1139::TaskManager.Task::(prepare) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::Prepare: aborted: Error validating master storage domain >Thread-10031::DEBUG::2012-12-02 23:36:05,836::task::957::TaskManager.Task::(_decref) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::ref 0 aborting True >Thread-10031::DEBUG::2012-12-02 23:36:05,836::task::892::TaskManager.Task::(_doAbort) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::Task._doAbort: force False >Thread-10031::DEBUG::2012-12-02 23:36:05,836::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10031::DEBUG::2012-12-02 23:36:05,836::task::568::TaskManager.Task::(_updateState) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::moving from state preparing -> state aborting >Thread-10031::DEBUG::2012-12-02 23:36:05,836::task::523::TaskManager.Task::(__state_aborting) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::_aborting: recover policy none >Thread-10031::DEBUG::2012-12-02 23:36:05,837::task::568::TaskManager.Task::(_updateState) Task=`549a2652-31f3-4d15-b681-78b8c11da9dd`::moving from state aborting -> state failed >Thread-10031::DEBUG::2012-12-02 23:36:05,837::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10031::DEBUG::2012-12-02 23:36:05,837::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10031::ERROR::2012-12-02 23:36:05,837::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-10093::DEBUG::2012-12-02 23:36:06,061::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10093::DEBUG::2012-12-02 23:36:06,061::task::568::TaskManager.Task::(_updateState) Task=`ffd0feb2-02d8-4be8-9b4d-9fc751245621`::moving from state init -> state preparing >Thread-10093::INFO::2012-12-02 23:36:06,061::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-10093::DEBUG::2012-12-02 23:36:06,061::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-10093::DEBUG::2012-12-02 23:36:06,061::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-10093::INFO::2012-12-02 23:36:06,062::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-10093::DEBUG::2012-12-02 23:36:06,062::task::1151::TaskManager.Task::(prepare) Task=`ffd0feb2-02d8-4be8-9b4d-9fc751245621`::finished: {'allTasksStatus': {}} >Thread-10093::DEBUG::2012-12-02 23:36:06,062::task::568::TaskManager.Task::(_updateState) Task=`ffd0feb2-02d8-4be8-9b4d-9fc751245621`::moving from state preparing -> state finished >Thread-10093::DEBUG::2012-12-02 23:36:06,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10093::DEBUG::2012-12-02 23:36:06,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10093::DEBUG::2012-12-02 23:36:06,062::task::957::TaskManager.Task::(_decref) Task=`ffd0feb2-02d8-4be8-9b4d-9fc751245621`::ref 0 aborting False >Thread-10094::DEBUG::2012-12-02 23:36:06,212::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10094::DEBUG::2012-12-02 23:36:06,213::task::568::TaskManager.Task::(_updateState) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::moving from state init -> state preparing >Thread-10094::INFO::2012-12-02 23:36:06,213::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10094::DEBUG::2012-12-02 23:36:06,213::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10094::DEBUG::2012-12-02 23:36:06,214::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10094::DEBUG::2012-12-02 23:36:06,418::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10094::WARNING::2012-12-02 23:36:06,420::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10094::DEBUG::2012-12-02 23:36:06,420::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10094::ERROR::2012-12-02 23:36:06,449::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10094::ERROR::2012-12-02 23:36:06,449::task::833::TaskManager.Task::(_setError) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-10094::DEBUG::2012-12-02 23:36:06,449::task::852::TaskManager.Task::(_run) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::Task._run: cd581c26-c7d3-433f-b76a-2a6d82f50ec8 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-10094::DEBUG::2012-12-02 23:36:06,449::task::1177::TaskManager.Task::(stop) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::stopping in state preparing (force False) >Thread-10094::DEBUG::2012-12-02 23:36:06,450::task::957::TaskManager.Task::(_decref) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::ref 1 aborting True >Thread-10094::INFO::2012-12-02 23:36:06,450::task::1134::TaskManager.Task::(prepare) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-10094::DEBUG::2012-12-02 23:36:06,450::task::1139::TaskManager.Task::(prepare) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::Prepare: aborted: Error validating master storage domain >Thread-10094::DEBUG::2012-12-02 23:36:06,450::task::957::TaskManager.Task::(_decref) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::ref 0 aborting True >Thread-10094::DEBUG::2012-12-02 23:36:06,450::task::892::TaskManager.Task::(_doAbort) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::Task._doAbort: force False >Thread-10094::DEBUG::2012-12-02 23:36:06,450::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10094::DEBUG::2012-12-02 23:36:06,450::task::568::TaskManager.Task::(_updateState) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::moving from state preparing -> state aborting >Thread-10094::DEBUG::2012-12-02 23:36:06,450::task::523::TaskManager.Task::(__state_aborting) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::_aborting: recover policy none >Thread-10094::DEBUG::2012-12-02 23:36:06,451::task::568::TaskManager.Task::(_updateState) Task=`cd581c26-c7d3-433f-b76a-2a6d82f50ec8`::moving from state aborting -> state failed >Thread-10094::DEBUG::2012-12-02 23:36:06,451::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10094::DEBUG::2012-12-02 23:36:06,451::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10094::ERROR::2012-12-02 23:36:06,451::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-10157::DEBUG::2012-12-02 23:36:09,706::task::568::TaskManager.Task::(_updateState) Task=`5140ff82-9efa-442b-971c-471d008707f2`::moving from state init -> state preparing >Thread-10157::INFO::2012-12-02 23:36:09,706::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10157::INFO::2012-12-02 23:36:09,706::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-10157::DEBUG::2012-12-02 23:36:09,706::task::1151::TaskManager.Task::(prepare) Task=`5140ff82-9efa-442b-971c-471d008707f2`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-10157::DEBUG::2012-12-02 23:36:09,706::task::568::TaskManager.Task::(_updateState) Task=`5140ff82-9efa-442b-971c-471d008707f2`::moving from state preparing -> state finished >Thread-10157::DEBUG::2012-12-02 23:36:09,707::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10157::DEBUG::2012-12-02 23:36:09,707::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10157::DEBUG::2012-12-02 23:36:09,707::task::957::TaskManager.Task::(_decref) Task=`5140ff82-9efa-442b-971c-471d008707f2`::ref 0 aborting False >Thread-10162::DEBUG::2012-12-02 23:36:16,613::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10162::DEBUG::2012-12-02 23:36:16,613::task::568::TaskManager.Task::(_updateState) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::moving from state init -> state preparing >Thread-10162::INFO::2012-12-02 23:36:16,613::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10162::DEBUG::2012-12-02 23:36:16,613::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10162::DEBUG::2012-12-02 23:36:16,614::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10162::DEBUG::2012-12-02 23:36:16,819::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10162::WARNING::2012-12-02 23:36:16,820::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10162::DEBUG::2012-12-02 23:36:16,821::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10162::ERROR::2012-12-02 23:36:16,850::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10162::ERROR::2012-12-02 23:36:16,851::task::833::TaskManager.Task::(_setError) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-10162::DEBUG::2012-12-02 23:36:16,851::task::852::TaskManager.Task::(_run) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::Task._run: b126d7e8-e868-48e9-90e8-4127b8a6fe63 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-10162::DEBUG::2012-12-02 23:36:16,851::task::1177::TaskManager.Task::(stop) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::stopping in state preparing (force False) >Thread-10162::DEBUG::2012-12-02 23:36:16,851::task::957::TaskManager.Task::(_decref) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::ref 1 aborting True >Thread-10162::INFO::2012-12-02 23:36:16,852::task::1134::TaskManager.Task::(prepare) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-10162::DEBUG::2012-12-02 23:36:16,852::task::1139::TaskManager.Task::(prepare) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::Prepare: aborted: Error validating master storage domain >Thread-10162::DEBUG::2012-12-02 23:36:16,852::task::957::TaskManager.Task::(_decref) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::ref 0 aborting True >Thread-10162::DEBUG::2012-12-02 23:36:16,852::task::892::TaskManager.Task::(_doAbort) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::Task._doAbort: force False >Thread-10162::DEBUG::2012-12-02 23:36:16,852::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10162::DEBUG::2012-12-02 23:36:16,852::task::568::TaskManager.Task::(_updateState) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::moving from state preparing -> state aborting >Thread-10162::DEBUG::2012-12-02 23:36:16,852::task::523::TaskManager.Task::(__state_aborting) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::_aborting: recover policy none >Thread-10162::DEBUG::2012-12-02 23:36:16,852::task::568::TaskManager.Task::(_updateState) Task=`b126d7e8-e868-48e9-90e8-4127b8a6fe63`::moving from state aborting -> state failed >Thread-10162::DEBUG::2012-12-02 23:36:16,853::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10162::DEBUG::2012-12-02 23:36:16,853::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10162::ERROR::2012-12-02 23:36:16,853::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-10225::DEBUG::2012-12-02 23:36:20,694::task::568::TaskManager.Task::(_updateState) Task=`e2f9bc46-41ae-4231-b576-eba9beae92a1`::moving from state init -> state preparing >Thread-10225::INFO::2012-12-02 23:36:20,694::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10225::INFO::2012-12-02 23:36:20,694::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-10225::DEBUG::2012-12-02 23:36:20,695::task::1151::TaskManager.Task::(prepare) Task=`e2f9bc46-41ae-4231-b576-eba9beae92a1`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-10225::DEBUG::2012-12-02 23:36:20,695::task::568::TaskManager.Task::(_updateState) Task=`e2f9bc46-41ae-4231-b576-eba9beae92a1`::moving from state preparing -> state finished >Thread-10225::DEBUG::2012-12-02 23:36:20,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10225::DEBUG::2012-12-02 23:36:20,695::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10225::DEBUG::2012-12-02 23:36:20,695::task::957::TaskManager.Task::(_decref) Task=`e2f9bc46-41ae-4231-b576-eba9beae92a1`::ref 0 aborting False >Thread-10229::DEBUG::2012-12-02 23:36:27,023::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10229::DEBUG::2012-12-02 23:36:27,023::task::568::TaskManager.Task::(_updateState) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::moving from state init -> state preparing >Thread-10229::INFO::2012-12-02 23:36:27,023::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10229::DEBUG::2012-12-02 23:36:27,024::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10229::DEBUG::2012-12-02 23:36:27,024::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10229::DEBUG::2012-12-02 23:36:27,227::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10229::WARNING::2012-12-02 23:36:27,228::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10229::DEBUG::2012-12-02 23:36:27,229::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10229::ERROR::2012-12-02 23:36:27,262::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10229::ERROR::2012-12-02 23:36:27,262::task::833::TaskManager.Task::(_setError) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-10229::DEBUG::2012-12-02 23:36:27,262::task::852::TaskManager.Task::(_run) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::Task._run: bc7b8479-420a-4ed9-b7d1-03f9fa16e836 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-10229::DEBUG::2012-12-02 23:36:27,262::task::1177::TaskManager.Task::(stop) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::stopping in state preparing (force False) >Thread-10229::DEBUG::2012-12-02 23:36:27,263::task::957::TaskManager.Task::(_decref) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::ref 1 aborting True >Thread-10229::INFO::2012-12-02 23:36:27,263::task::1134::TaskManager.Task::(prepare) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-10229::DEBUG::2012-12-02 23:36:27,263::task::1139::TaskManager.Task::(prepare) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::Prepare: aborted: Error validating master storage domain >Thread-10229::DEBUG::2012-12-02 23:36:27,263::task::957::TaskManager.Task::(_decref) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::ref 0 aborting True >Thread-10229::DEBUG::2012-12-02 23:36:27,263::task::892::TaskManager.Task::(_doAbort) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::Task._doAbort: force False >Thread-10229::DEBUG::2012-12-02 23:36:27,263::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10229::DEBUG::2012-12-02 23:36:27,263::task::568::TaskManager.Task::(_updateState) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::moving from state preparing -> state aborting >Thread-10229::DEBUG::2012-12-02 23:36:27,264::task::523::TaskManager.Task::(__state_aborting) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::_aborting: recover policy none >Thread-10229::DEBUG::2012-12-02 23:36:27,264::task::568::TaskManager.Task::(_updateState) Task=`bc7b8479-420a-4ed9-b7d1-03f9fa16e836`::moving from state aborting -> state failed >Thread-10229::DEBUG::2012-12-02 23:36:27,264::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10229::DEBUG::2012-12-02 23:36:27,264::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10229::ERROR::2012-12-02 23:36:27,264::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >Thread-10293::DEBUG::2012-12-02 23:36:31,624::task::568::TaskManager.Task::(_updateState) Task=`1656c7d2-265f-4aff-bad7-7bdad22609da`::moving from state init -> state preparing >Thread-10293::INFO::2012-12-02 23:36:31,624::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10293::INFO::2012-12-02 23:36:31,625::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-10293::DEBUG::2012-12-02 23:36:31,625::task::1151::TaskManager.Task::(prepare) Task=`1656c7d2-265f-4aff-bad7-7bdad22609da`::finished: {'225bca28-f0a0-4899-b137-32acd04f57d4': {'delay': '0', 'lastCheck': 1354484150.5892301, 'code': 200, 'valid': False}} >Thread-10293::DEBUG::2012-12-02 23:36:31,625::task::568::TaskManager.Task::(_updateState) Task=`1656c7d2-265f-4aff-bad7-7bdad22609da`::moving from state preparing -> state finished >Thread-10293::DEBUG::2012-12-02 23:36:31,625::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10293::DEBUG::2012-12-02 23:36:31,625::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10293::DEBUG::2012-12-02 23:36:31,625::task::957::TaskManager.Task::(_decref) Task=`1656c7d2-265f-4aff-bad7-7bdad22609da`::ref 0 aborting False >Thread-10297::DEBUG::2012-12-02 23:36:37,431::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10297::DEBUG::2012-12-02 23:36:37,431::task::568::TaskManager.Task::(_updateState) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::moving from state init -> state preparing >Thread-10297::INFO::2012-12-02 23:36:37,431::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10297::DEBUG::2012-12-02 23:36:37,431::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10297::DEBUG::2012-12-02 23:36:37,432::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10297::DEBUG::2012-12-02 23:36:37,638::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10297::WARNING::2012-12-02 23:36:37,640::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10297::DEBUG::2012-12-02 23:36:37,640::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10297::ERROR::2012-12-02 23:36:37,670::hsm::530::Storage.HSM::(getSpmStatus) MD read error: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 523, in getSpmStatus > status = {'spmStatus':pool.spmRole, 'spmLver': pool.getSpmLver(), > File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver > return self.getMetaParam(PMDK_LVER) > File "/usr/share/vdsm/storage/sp.py", line 1488, in getMetaParam > return self._metadata[key] > File "/usr/share/vdsm/storage/sp.py", line 1315, in _metadata > return self._getPoolMD(self.masterDomain) > File "/usr/share/vdsm/storage/sp.py", line 1311, in _getPoolMD > return DictValidator(domain._metadata._dict, SP_MD_FIELDS) > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10297::ERROR::2012-12-02 23:36:37,670::task::833::TaskManager.Task::(_setError) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 531, in getSpmStatus > raise se.StorageDomainMasterError("MD read error") >StorageDomainMasterError: Error validating master storage domain: ('MD read error',) >Thread-10297::DEBUG::2012-12-02 23:36:37,670::task::852::TaskManager.Task::(_run) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::Task._run: 5fa2e09c-7e64-407f-8816-4f85010c5a1f ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-10297::DEBUG::2012-12-02 23:36:37,671::task::1177::TaskManager.Task::(stop) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::stopping in state preparing (force False) >Thread-10297::DEBUG::2012-12-02 23:36:37,671::task::957::TaskManager.Task::(_decref) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::ref 1 aborting True >Thread-10297::INFO::2012-12-02 23:36:37,671::task::1134::TaskManager.Task::(prepare) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::aborting: Task is aborted: 'Error validating master storage domain' - code 355 >Thread-10297::DEBUG::2012-12-02 23:36:37,671::task::1139::TaskManager.Task::(prepare) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::Prepare: aborted: Error validating master storage domain >Thread-10297::DEBUG::2012-12-02 23:36:37,671::task::957::TaskManager.Task::(_decref) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::ref 0 aborting True >Thread-10297::DEBUG::2012-12-02 23:36:37,671::task::892::TaskManager.Task::(_doAbort) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::Task._doAbort: force False >Thread-10297::DEBUG::2012-12-02 23:36:37,671::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10297::DEBUG::2012-12-02 23:36:37,672::task::568::TaskManager.Task::(_updateState) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::moving from state preparing -> state aborting >Thread-10297::DEBUG::2012-12-02 23:36:37,672::task::523::TaskManager.Task::(__state_aborting) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::_aborting: recover policy none >Thread-10297::DEBUG::2012-12-02 23:36:37,672::task::568::TaskManager.Task::(_updateState) Task=`5fa2e09c-7e64-407f-8816-4f85010c5a1f`::moving from state aborting -> state failed >Thread-10297::DEBUG::2012-12-02 23:36:37,672::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10297::DEBUG::2012-12-02 23:36:37,672::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10297::ERROR::2012-12-02 23:36:37,672::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Error validating master storage domain: ('MD read error',)", 'code': 355}} >MainThread::INFO::2012-12-02 23:36:38,432::vmChannels::135::vds::(stop) VM channels listener was stopped. >MainThread::DEBUG::2012-12-02 23:36:38,432::task::568::TaskManager.Task::(_updateState) Task=`4c010ce3-b046-4d58-90be-2b035387ea2d`::moving from state init -> state preparing >MainThread::DEBUG::2012-12-02 23:36:38,432::clientIF::198::vds::(prepareForShutdown) cannot run prepareForShutdown concurrently >MainThread::INFO::2012-12-02 23:36:38,433::logUtils::37::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2012-12-02 23:36:38,433::storageServer::617::ConnectionMonitor::(_monitorConnections) Monitoring stopped >MainThread::DEBUG::2012-12-02 23:36:38,440::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >MainThread::DEBUG::2012-12-02 23:36:38,489::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks >MainThread::INFO::2012-12-02 23:36:38,490::logUtils::39::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None >MainThread::DEBUG::2012-12-02 23:36:38,490::task::1151::TaskManager.Task::(prepare) Task=`4c010ce3-b046-4d58-90be-2b035387ea2d`::finished: None >MainThread::DEBUG::2012-12-02 23:36:38,490::task::568::TaskManager.Task::(_updateState) Task=`4c010ce3-b046-4d58-90be-2b035387ea2d`::moving from state preparing -> state finished >MainThread::DEBUG::2012-12-02 23:36:38,490::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2012-12-02 23:36:38,490::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2012-12-02 23:36:38,490::task::957::TaskManager.Task::(_decref) Task=`4c010ce3-b046-4d58-90be-2b035387ea2d`::ref 0 aborting False >MainThread::INFO::2012-12-02 23:36:38,490::vdsm::76::vds::(run) VDSM main thread ended. Waiting for 13 other threads... >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <_MainThread(MainThread, started 139747386918656)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <Thread(libvirtEventLoop, started daemon 139747308062464)> >MainThread::INFO::2012-12-02 23:36:38,491::vmChannels::135::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <Listener(VM Channels Listener, started daemon 139746089105152)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-3, started daemon 139747179198208)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-2, started daemon 139747189688064)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <KsmMonitorThread(KsmMonitor, started daemon 139746516907776)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(3d8ad152-e945-43ea-bdef-fe11d53183fd, started daemon 139746930050816)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-10, started daemon 139746558867200)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-8, started daemon 139746919560960)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-4, started daemon 139746961520384)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-6, started daemon 139746940540672)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-9, started daemon 139746909071104)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-1, started daemon 139747200177920)> >MainThread::INFO::2012-12-02 23:36:38,491::vdsm::79::vds::(run) <WorkerThread(Thread-5, started daemon 139746951030528)> >MainThread::INFO::2012-12-02 23:36:38,552::vdsm::71::vds::(run) I am the actual vdsm 4.10-0.239 >MainThread::DEBUG::2012-12-02 23:36:38,682::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2012-12-02 23:36:38,683::threadPool::45::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::DEBUG::2012-12-02 23:36:38,692::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >MainThread::DEBUG::2012-12-02 23:36:38,713::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/kill -9 18658' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:38,723::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:36:38,723::supervdsm::91::SuperVdsmProxy::(_launchSupervdsm) Launching Super Vdsm >MainThread::DEBUG::2012-12-02 23:36:38,723::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /usr/bin/python /usr/share/vdsm/supervdsmServer.py f96bba70-6e2b-43fd-877a-dc626c443266 1890' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:38,839::supervdsmServer::306::SuperVdsm.Server::(main) Making sure I'm root >MainThread::DEBUG::2012-12-02 23:36:38,839::supervdsmServer::310::SuperVdsm.Server::(main) Parsing cmd args >MainThread::DEBUG::2012-12-02 23:36:38,839::supervdsmServer::313::SuperVdsm.Server::(main) Creating PID file >MainThread::DEBUG::2012-12-02 23:36:38,839::supervdsmServer::317::SuperVdsm.Server::(main) Cleaning old socket >MainThread::DEBUG::2012-12-02 23:36:38,839::supervdsmServer::321::SuperVdsm.Server::(main) Setting up keep alive thread >MainThread::DEBUG::2012-12-02 23:36:38,840::supervdsmServer::327::SuperVdsm.Server::(main) Creating remote object manager >MainThread::DEBUG::2012-12-02 23:36:38,840::supervdsmServer::338::SuperVdsm.Server::(main) Started serving super vdsm object >MainThread::DEBUG::2012-12-02 23:36:40,728::supervdsm::113::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2012-12-02 23:36:40,732::supervdsm::84::SuperVdsmProxy::(__init__) Connected to Super Vdsm >MainThread::DEBUG::2012-12-02 23:36:40,733::multipath::106::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2012-12-02 23:36:40,733::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:40,807::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:36:40,807::lvm::334::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2012-12-02 23:36:40,809::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:41,115::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:36:41,115::lvm::359::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >MainThread::DEBUG::2012-12-02 23:36:41,115::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2012-12-02 23:36:41,116::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:41,498::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:36:41,501::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >MainThread::DEBUG::2012-12-02 23:36:41,501::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:41,839::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >Thread-12::DEBUG::2012-12-02 23:36:41,856::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >MainThread::INFO::2012-12-02 23:36:41,857::dispatcher::95::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >Thread-12::DEBUG::2012-12-02 23:36:41,857::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-12::DEBUG::2012-12-02 23:36:41,857::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-12::DEBUG::2012-12-02 23:36:41,858::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-12::DEBUG::2012-12-02 23:36:41,858::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >MainThread::DEBUG::2012-12-02 23:36:41,864::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/pgrep -xf ksmd' (cwd None) >Thread-12::DEBUG::2012-12-02 23:36:41,886::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:36:41,886::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-12::DEBUG::2012-12-02 23:36:41,887::misc::1071::SamplingMethod::(__call__) Returning last result >MainThread::INFO::2012-12-02 23:36:41,888::vmChannels::139::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2012-12-02 23:36:41,890::vmChannels::127::vds::(run) Starting VM channels listener thread. >MainThread::WARNING::2012-12-02 23:36:41,894::clientIF::169::vds::(_prepareBindings) Unable to load the rest server module. Please make sure it is installed. >Thread-12::DEBUG::2012-12-02 23:36:42,521::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-12::DEBUG::2012-12-02 23:36:42,597::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-12::DEBUG::2012-12-02 23:36:42,598::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 23:36:42,598::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 23:36:42,598::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 23:36:42,598::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 23:36:42,599::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 23:36:42,599::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 23:36:42,599::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-12::DEBUG::2012-12-02 23:36:42,599::hsm::355::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >Thread-12::DEBUG::2012-12-02 23:36:42,606::hsm::387::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >Thread-12::DEBUG::2012-12-02 23:36:42,606::hsm::388::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_101', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0941', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0851', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sdds1', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sdds2', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_export1', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_data3', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252'] >Thread-12::DEBUG::2012-12-02 23:36:42,606::hsm::390::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >Thread-12::DEBUG::2012-12-02 23:36:42,607::hsm::433::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >Thread-14::DEBUG::2012-12-02 23:36:42,935::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-14::DEBUG::2012-12-02 23:36:43,025::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-14::DEBUG::2012-12-02 23:36:43,036::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-14::DEBUG::2012-12-02 23:36:43,036::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-14::DEBUG::2012-12-02 23:36:43,044::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-14::DEBUG::2012-12-02 23:36:43,045::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-14::DEBUG::2012-12-02 23:36:43,055::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 23:36:43,056::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-14::DEBUG::2012-12-02 23:36:43,066::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 23:36:43,067::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-14::DEBUG::2012-12-02 23:36:43,078::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 23:36:43,295::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '0.0.0.0'}} >Thread-16::DEBUG::2012-12-02 23:36:43,852::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-16::DEBUG::2012-12-02 23:36:43,852::task::568::TaskManager.Task::(_updateState) Task=`08d78b0e-6c7a-4519-86b8-91b7b9014d17`::moving from state init -> state preparing >Thread-16::INFO::2012-12-02 23:36:43,853::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-16::DEBUG::2012-12-02 23:36:43,858::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2012-12-02 23:36:43,858::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2012-12-02 23:36:43,858::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2012-12-02 23:36:43,858::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::DEBUG::2012-12-02 23:36:43,858::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-16::DEBUG::2012-12-02 23:36:43,858::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-16::INFO::2012-12-02 23:36:43,858::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-16::DEBUG::2012-12-02 23:36:43,858::task::1151::TaskManager.Task::(prepare) Task=`08d78b0e-6c7a-4519-86b8-91b7b9014d17`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-16::DEBUG::2012-12-02 23:36:43,858::task::568::TaskManager.Task::(_updateState) Task=`08d78b0e-6c7a-4519-86b8-91b7b9014d17`::moving from state preparing -> state finished >Thread-16::DEBUG::2012-12-02 23:36:43,859::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-16::DEBUG::2012-12-02 23:36:43,859::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-16::DEBUG::2012-12-02 23:36:43,859::task::957::TaskManager.Task::(_decref) Task=`08d78b0e-6c7a-4519-86b8-91b7b9014d17`::ref 0 aborting False >Thread-17::DEBUG::2012-12-02 23:36:44,033::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-17::DEBUG::2012-12-02 23:36:44,034::task::568::TaskManager.Task::(_updateState) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::moving from state init -> state preparing >Thread-17::INFO::2012-12-02 23:36:44,034::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-17::DEBUG::2012-12-02 23:36:44,034::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`10295e87-13c5-4c92-8917-d19fc7bbb3a3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-17::DEBUG::2012-12-02 23:36:44,034::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-17::DEBUG::2012-12-02 23:36:44,034::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-17::DEBUG::2012-12-02 23:36:44,035::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`10295e87-13c5-4c92-8917-d19fc7bbb3a3`::Granted request >Thread-17::INFO::2012-12-02 23:36:44,035::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-17::DEBUG::2012-12-02 23:36:44,035::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,036::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,036::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,036::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,036::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,036::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,036::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-17::DEBUG::2012-12-02 23:36:44,036::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-17::DEBUG::2012-12-02 23:36:44,036::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-17::DEBUG::2012-12-02 23:36:44,036::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-17::DEBUG::2012-12-02 23:36:44,036::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-17::DEBUG::2012-12-02 23:36:44,063::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-17::DEBUG::2012-12-02 23:36:44,063::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-17::DEBUG::2012-12-02 23:36:44,696::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-17::DEBUG::2012-12-02 23:36:44,764::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-17::DEBUG::2012-12-02 23:36:44,764::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,764::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,764::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,765::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,765::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,765::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,765::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-17::DEBUG::2012-12-02 23:36:44,765::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:44,767::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-17::DEBUG::2012-12-02 23:36:44,971::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-17::WARNING::2012-12-02 23:36:44,972::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-17::DEBUG::2012-12-02 23:36:44,972::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:36:45,094::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-17::DEBUG::2012-12-02 23:36:45,094::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-17::DEBUG::2012-12-02 23:36:45,094::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-17::DEBUG::2012-12-02 23:36:45,094::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-17::ERROR::2012-12-02 23:36:45,095::task::833::TaskManager.Task::(_setError) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-17::DEBUG::2012-12-02 23:36:45,097::task::852::TaskManager.Task::(_run) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::Task._run: e406047b-0af1-4da6-82fb-f1f2bdba37da ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-17::DEBUG::2012-12-02 23:36:45,097::task::1177::TaskManager.Task::(stop) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::stopping in state preparing (force False) >Thread-17::DEBUG::2012-12-02 23:36:45,097::task::957::TaskManager.Task::(_decref) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::ref 1 aborting True >Thread-17::INFO::2012-12-02 23:36:45,097::task::1134::TaskManager.Task::(prepare) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-17::DEBUG::2012-12-02 23:36:45,097::task::1139::TaskManager.Task::(prepare) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::Prepare: aborted: Cannot find master domain >Thread-17::DEBUG::2012-12-02 23:36:45,097::task::957::TaskManager.Task::(_decref) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::ref 0 aborting True >Thread-17::DEBUG::2012-12-02 23:36:45,097::task::892::TaskManager.Task::(_doAbort) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::Task._doAbort: force False >Thread-17::DEBUG::2012-12-02 23:36:45,098::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-17::DEBUG::2012-12-02 23:36:45,098::task::568::TaskManager.Task::(_updateState) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::moving from state preparing -> state aborting >Thread-17::DEBUG::2012-12-02 23:36:45,098::task::523::TaskManager.Task::(__state_aborting) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::_aborting: recover policy none >Thread-17::DEBUG::2012-12-02 23:36:45,098::task::568::TaskManager.Task::(_updateState) Task=`e406047b-0af1-4da6-82fb-f1f2bdba37da`::moving from state aborting -> state failed >Thread-17::DEBUG::2012-12-02 23:36:45,098::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-17::DEBUG::2012-12-02 23:36:45,098::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-17::ERROR::2012-12-02 23:36:45,098::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-80::DEBUG::2012-12-02 23:36:47,833::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-80::DEBUG::2012-12-02 23:36:47,833::task::568::TaskManager.Task::(_updateState) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::moving from state init -> state preparing >Thread-80::INFO::2012-12-02 23:36:47,834::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-80::ERROR::2012-12-02 23:36:47,834::task::833::TaskManager.Task::(_setError) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-80::DEBUG::2012-12-02 23:36:47,834::task::852::TaskManager.Task::(_run) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::Task._run: 2e466d45-ab79-4472-88ec-6b3af500e1ac ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-80::DEBUG::2012-12-02 23:36:47,834::task::1177::TaskManager.Task::(stop) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::stopping in state preparing (force False) >Thread-80::DEBUG::2012-12-02 23:36:47,834::task::957::TaskManager.Task::(_decref) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::ref 1 aborting True >Thread-80::INFO::2012-12-02 23:36:47,834::task::1134::TaskManager.Task::(prepare) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-80::DEBUG::2012-12-02 23:36:47,835::task::1139::TaskManager.Task::(prepare) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::Prepare: aborted: Unknown pool id, pool not connected >Thread-80::DEBUG::2012-12-02 23:36:47,835::task::957::TaskManager.Task::(_decref) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::ref 0 aborting True >Thread-80::DEBUG::2012-12-02 23:36:47,835::task::892::TaskManager.Task::(_doAbort) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::Task._doAbort: force False >Thread-80::DEBUG::2012-12-02 23:36:47,835::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-80::DEBUG::2012-12-02 23:36:47,835::task::568::TaskManager.Task::(_updateState) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::moving from state preparing -> state aborting >Thread-80::DEBUG::2012-12-02 23:36:47,835::task::523::TaskManager.Task::(__state_aborting) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::_aborting: recover policy none >Thread-80::DEBUG::2012-12-02 23:36:47,835::task::568::TaskManager.Task::(_updateState) Task=`2e466d45-ab79-4472-88ec-6b3af500e1ac`::moving from state aborting -> state failed >Thread-80::DEBUG::2012-12-02 23:36:47,835::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-80::DEBUG::2012-12-02 23:36:47,835::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-80::ERROR::2012-12-02 23:36:47,836::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-81::DEBUG::2012-12-02 23:36:48,021::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-81::DEBUG::2012-12-02 23:36:48,021::task::568::TaskManager.Task::(_updateState) Task=`b3719a69-5238-4448-b264-7d70218939d8`::moving from state init -> state preparing >Thread-81::INFO::2012-12-02 23:36:48,022::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-81::DEBUG::2012-12-02 23:36:48,022::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6b5ec9a1-cc4f-4ab5-93c7-cba274caaedd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-81::DEBUG::2012-12-02 23:36:48,022::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-81::DEBUG::2012-12-02 23:36:48,022::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-81::DEBUG::2012-12-02 23:36:48,022::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6b5ec9a1-cc4f-4ab5-93c7-cba274caaedd`::Granted request >Thread-81::INFO::2012-12-02 23:36:48,023::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-81::DEBUG::2012-12-02 23:36:48,023::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,023::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,023::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,023::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,024::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,024::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,024::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-81::DEBUG::2012-12-02 23:36:48,024::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-81::DEBUG::2012-12-02 23:36:48,024::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-81::DEBUG::2012-12-02 23:36:48,024::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-81::DEBUG::2012-12-02 23:36:48,024::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-81::DEBUG::2012-12-02 23:36:48,053::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-81::DEBUG::2012-12-02 23:36:48,053::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-81::DEBUG::2012-12-02 23:36:48,686::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-81::DEBUG::2012-12-02 23:36:48,752::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-81::DEBUG::2012-12-02 23:36:48,752::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,752::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,753::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,753::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,753::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,753::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,753::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-81::DEBUG::2012-12-02 23:36:48,754::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,755::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-81::DEBUG::2012-12-02 23:36:48,962::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-81::WARNING::2012-12-02 23:36:48,963::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-81::DEBUG::2012-12-02 23:36:48,964::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-81::DEBUG::2012-12-02 23:36:48,993::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-81::DEBUG::2012-12-02 23:36:48,994::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-81::DEBUG::2012-12-02 23:36:48,994::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-81::DEBUG::2012-12-02 23:36:48,994::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-81::ERROR::2012-12-02 23:36:48,994::task::833::TaskManager.Task::(_setError) Task=`b3719a69-5238-4448-b264-7d70218939d8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-81::DEBUG::2012-12-02 23:36:48,994::task::852::TaskManager.Task::(_run) Task=`b3719a69-5238-4448-b264-7d70218939d8`::Task._run: b3719a69-5238-4448-b264-7d70218939d8 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-81::DEBUG::2012-12-02 23:36:48,995::task::1177::TaskManager.Task::(stop) Task=`b3719a69-5238-4448-b264-7d70218939d8`::stopping in state preparing (force False) >Thread-81::DEBUG::2012-12-02 23:36:48,995::task::957::TaskManager.Task::(_decref) Task=`b3719a69-5238-4448-b264-7d70218939d8`::ref 1 aborting True >Thread-81::INFO::2012-12-02 23:36:48,995::task::1134::TaskManager.Task::(prepare) Task=`b3719a69-5238-4448-b264-7d70218939d8`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-81::DEBUG::2012-12-02 23:36:48,995::task::1139::TaskManager.Task::(prepare) Task=`b3719a69-5238-4448-b264-7d70218939d8`::Prepare: aborted: Cannot find master domain >Thread-81::DEBUG::2012-12-02 23:36:48,995::task::957::TaskManager.Task::(_decref) Task=`b3719a69-5238-4448-b264-7d70218939d8`::ref 0 aborting True >Thread-81::DEBUG::2012-12-02 23:36:48,995::task::892::TaskManager.Task::(_doAbort) Task=`b3719a69-5238-4448-b264-7d70218939d8`::Task._doAbort: force False >Thread-81::DEBUG::2012-12-02 23:36:48,995::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-81::DEBUG::2012-12-02 23:36:48,996::task::568::TaskManager.Task::(_updateState) Task=`b3719a69-5238-4448-b264-7d70218939d8`::moving from state preparing -> state aborting >Thread-81::DEBUG::2012-12-02 23:36:48,996::task::523::TaskManager.Task::(__state_aborting) Task=`b3719a69-5238-4448-b264-7d70218939d8`::_aborting: recover policy none >Thread-81::DEBUG::2012-12-02 23:36:48,996::task::568::TaskManager.Task::(_updateState) Task=`b3719a69-5238-4448-b264-7d70218939d8`::moving from state aborting -> state failed >Thread-81::DEBUG::2012-12-02 23:36:48,996::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-81::DEBUG::2012-12-02 23:36:48,996::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-81::ERROR::2012-12-02 23:36:48,996::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-143::DEBUG::2012-12-02 23:36:49,320::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-143::DEBUG::2012-12-02 23:36:49,320::task::568::TaskManager.Task::(_updateState) Task=`bb165c19-a752-4174-9767-155c4a1b3061`::moving from state init -> state preparing >Thread-143::INFO::2012-12-02 23:36:49,320::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-143::DEBUG::2012-12-02 23:36:49,320::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-143::DEBUG::2012-12-02 23:36:49,321::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-143::INFO::2012-12-02 23:36:49,321::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-143::DEBUG::2012-12-02 23:36:49,321::task::1151::TaskManager.Task::(prepare) Task=`bb165c19-a752-4174-9767-155c4a1b3061`::finished: {'allTasksStatus': {}} >Thread-143::DEBUG::2012-12-02 23:36:49,321::task::568::TaskManager.Task::(_updateState) Task=`bb165c19-a752-4174-9767-155c4a1b3061`::moving from state preparing -> state finished >Thread-143::DEBUG::2012-12-02 23:36:49,321::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-143::DEBUG::2012-12-02 23:36:49,321::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-143::DEBUG::2012-12-02 23:36:49,321::task::957::TaskManager.Task::(_decref) Task=`bb165c19-a752-4174-9767-155c4a1b3061`::ref 0 aborting False >Thread-144::DEBUG::2012-12-02 23:36:49,458::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-144::DEBUG::2012-12-02 23:36:49,458::task::568::TaskManager.Task::(_updateState) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::moving from state init -> state preparing >Thread-144::INFO::2012-12-02 23:36:49,458::logUtils::37::dispatcher::(wrapper) Run and protect: spmStop(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-144::DEBUG::2012-12-02 23:36:49,458::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f6e2feab-083f-4f6a-9367-62bdac9174e2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-144::DEBUG::2012-12-02 23:36:49,458::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-144::DEBUG::2012-12-02 23:36:49,459::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-144::DEBUG::2012-12-02 23:36:49,459::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f6e2feab-083f-4f6a-9367-62bdac9174e2`::Granted request >Thread-144::DEBUG::2012-12-02 23:36:49,459::task::794::TaskManager.Task::(resourceAcquired) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-144::DEBUG::2012-12-02 23:36:49,459::task::957::TaskManager.Task::(_decref) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::ref 1 aborting False >Thread-144::ERROR::2012-12-02 23:36:49,459::task::833::TaskManager.Task::(_setError) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 516, in spmStop > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-144::DEBUG::2012-12-02 23:36:49,460::task::852::TaskManager.Task::(_run) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::Task._run: 37602c67-f2a5-4e59-acfa-55e31083283b ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-144::DEBUG::2012-12-02 23:36:49,460::task::1177::TaskManager.Task::(stop) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::stopping in state preparing (force False) >Thread-144::DEBUG::2012-12-02 23:36:49,460::task::957::TaskManager.Task::(_decref) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::ref 1 aborting True >Thread-144::INFO::2012-12-02 23:36:49,460::task::1134::TaskManager.Task::(prepare) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-144::DEBUG::2012-12-02 23:36:49,460::task::1139::TaskManager.Task::(prepare) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::Prepare: aborted: Unknown pool id, pool not connected >Thread-144::DEBUG::2012-12-02 23:36:49,460::task::957::TaskManager.Task::(_decref) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::ref 0 aborting True >Thread-144::DEBUG::2012-12-02 23:36:49,460::task::892::TaskManager.Task::(_doAbort) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::Task._doAbort: force False >Thread-144::DEBUG::2012-12-02 23:36:49,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-144::DEBUG::2012-12-02 23:36:49,461::task::568::TaskManager.Task::(_updateState) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::moving from state preparing -> state aborting >Thread-144::DEBUG::2012-12-02 23:36:49,461::task::523::TaskManager.Task::(__state_aborting) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::_aborting: recover policy none >Thread-144::DEBUG::2012-12-02 23:36:49,461::task::568::TaskManager.Task::(_updateState) Task=`37602c67-f2a5-4e59-acfa-55e31083283b`::moving from state aborting -> state failed >Thread-144::DEBUG::2012-12-02 23:36:49,461::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-144::DEBUG::2012-12-02 23:36:49,461::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-144::DEBUG::2012-12-02 23:36:49,461::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-144::DEBUG::2012-12-02 23:36:49,461::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-144::DEBUG::2012-12-02 23:36:49,462::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-144::DEBUG::2012-12-02 23:36:49,462::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-144::ERROR::2012-12-02 23:36:49,462::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-148::DEBUG::2012-12-02 23:36:55,932::task::568::TaskManager.Task::(_updateState) Task=`f59d6109-8862-4333-8fd6-834c8ed52248`::moving from state init -> state preparing >Thread-148::INFO::2012-12-02 23:36:55,933::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-148::INFO::2012-12-02 23:36:55,933::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-148::DEBUG::2012-12-02 23:36:55,933::task::1151::TaskManager.Task::(prepare) Task=`f59d6109-8862-4333-8fd6-834c8ed52248`::finished: {} >Thread-148::DEBUG::2012-12-02 23:36:55,933::task::568::TaskManager.Task::(_updateState) Task=`f59d6109-8862-4333-8fd6-834c8ed52248`::moving from state preparing -> state finished >Thread-148::DEBUG::2012-12-02 23:36:55,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-148::DEBUG::2012-12-02 23:36:55,933::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-148::DEBUG::2012-12-02 23:36:55,933::task::957::TaskManager.Task::(_decref) Task=`f59d6109-8862-4333-8fd6-834c8ed52248`::ref 0 aborting False >Thread-151::DEBUG::2012-12-02 23:36:59,709::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-151::DEBUG::2012-12-02 23:36:59,709::task::568::TaskManager.Task::(_updateState) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::moving from state init -> state preparing >Thread-151::INFO::2012-12-02 23:36:59,710::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-151::ERROR::2012-12-02 23:36:59,710::task::833::TaskManager.Task::(_setError) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-151::DEBUG::2012-12-02 23:36:59,710::task::852::TaskManager.Task::(_run) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::Task._run: e8c52d8c-028d-4b2d-9ecf-3a583364cf35 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-151::DEBUG::2012-12-02 23:36:59,710::task::1177::TaskManager.Task::(stop) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::stopping in state preparing (force False) >Thread-151::DEBUG::2012-12-02 23:36:59,710::task::957::TaskManager.Task::(_decref) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::ref 1 aborting True >Thread-151::INFO::2012-12-02 23:36:59,710::task::1134::TaskManager.Task::(prepare) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-151::DEBUG::2012-12-02 23:36:59,711::task::1139::TaskManager.Task::(prepare) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::Prepare: aborted: Unknown pool id, pool not connected >Thread-151::DEBUG::2012-12-02 23:36:59,711::task::957::TaskManager.Task::(_decref) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::ref 0 aborting True >Thread-151::DEBUG::2012-12-02 23:36:59,711::task::892::TaskManager.Task::(_doAbort) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::Task._doAbort: force False >Thread-151::DEBUG::2012-12-02 23:36:59,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-151::DEBUG::2012-12-02 23:36:59,711::task::568::TaskManager.Task::(_updateState) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::moving from state preparing -> state aborting >Thread-151::DEBUG::2012-12-02 23:36:59,711::task::523::TaskManager.Task::(__state_aborting) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::_aborting: recover policy none >Thread-151::DEBUG::2012-12-02 23:36:59,711::task::568::TaskManager.Task::(_updateState) Task=`e8c52d8c-028d-4b2d-9ecf-3a583364cf35`::moving from state aborting -> state failed >Thread-151::DEBUG::2012-12-02 23:36:59,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-151::DEBUG::2012-12-02 23:36:59,712::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-151::ERROR::2012-12-02 23:36:59,712::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-152::DEBUG::2012-12-02 23:36:59,909::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-152::DEBUG::2012-12-02 23:36:59,909::task::568::TaskManager.Task::(_updateState) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::moving from state init -> state preparing >Thread-152::INFO::2012-12-02 23:36:59,909::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-152::DEBUG::2012-12-02 23:36:59,910::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`43592af7-b41c-4fe6-8134-04d1d57edbbe`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-152::DEBUG::2012-12-02 23:36:59,910::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-152::DEBUG::2012-12-02 23:36:59,910::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-152::DEBUG::2012-12-02 23:36:59,910::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`43592af7-b41c-4fe6-8134-04d1d57edbbe`::Granted request >Thread-152::INFO::2012-12-02 23:36:59,910::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-152::DEBUG::2012-12-02 23:36:59,911::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:36:59,911::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:36:59,911::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:36:59,911::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:36:59,911::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:36:59,912::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:36:59,912::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-152::DEBUG::2012-12-02 23:36:59,912::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-152::DEBUG::2012-12-02 23:36:59,912::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-152::DEBUG::2012-12-02 23:36:59,912::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-152::DEBUG::2012-12-02 23:36:59,912::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-152::DEBUG::2012-12-02 23:36:59,941::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-152::DEBUG::2012-12-02 23:36:59,941::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-152::DEBUG::2012-12-02 23:37:00,574::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-152::DEBUG::2012-12-02 23:37:00,650::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-152::DEBUG::2012-12-02 23:37:00,651::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,651::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,651::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,651::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,651::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,652::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,652::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-152::DEBUG::2012-12-02 23:37:00,652::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,653::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-152::DEBUG::2012-12-02 23:37:00,860::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-152::WARNING::2012-12-02 23:37:00,861::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-152::DEBUG::2012-12-02 23:37:00,862::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-152::DEBUG::2012-12-02 23:37:00,890::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-152::DEBUG::2012-12-02 23:37:00,890::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-152::DEBUG::2012-12-02 23:37:00,890::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-152::DEBUG::2012-12-02 23:37:00,890::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-152::ERROR::2012-12-02 23:37:00,890::task::833::TaskManager.Task::(_setError) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-152::DEBUG::2012-12-02 23:37:00,891::task::852::TaskManager.Task::(_run) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::Task._run: f3c5b5c8-2d5e-40ee-b8f4-55daf3350894 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-152::DEBUG::2012-12-02 23:37:00,891::task::1177::TaskManager.Task::(stop) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::stopping in state preparing (force False) >Thread-152::DEBUG::2012-12-02 23:37:00,891::task::957::TaskManager.Task::(_decref) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::ref 1 aborting True >Thread-152::INFO::2012-12-02 23:37:00,891::task::1134::TaskManager.Task::(prepare) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-152::DEBUG::2012-12-02 23:37:00,891::task::1139::TaskManager.Task::(prepare) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::Prepare: aborted: Cannot find master domain >Thread-152::DEBUG::2012-12-02 23:37:00,891::task::957::TaskManager.Task::(_decref) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::ref 0 aborting True >Thread-152::DEBUG::2012-12-02 23:37:00,892::task::892::TaskManager.Task::(_doAbort) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::Task._doAbort: force False >Thread-152::DEBUG::2012-12-02 23:37:00,892::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-152::DEBUG::2012-12-02 23:37:00,892::task::568::TaskManager.Task::(_updateState) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::moving from state preparing -> state aborting >Thread-152::DEBUG::2012-12-02 23:37:00,892::task::523::TaskManager.Task::(__state_aborting) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::_aborting: recover policy none >Thread-152::DEBUG::2012-12-02 23:37:00,892::task::568::TaskManager.Task::(_updateState) Task=`f3c5b5c8-2d5e-40ee-b8f4-55daf3350894`::moving from state aborting -> state failed >Thread-152::DEBUG::2012-12-02 23:37:00,892::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-152::DEBUG::2012-12-02 23:37:00,892::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-152::ERROR::2012-12-02 23:37:00,892::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-217::DEBUG::2012-12-02 23:37:06,895::task::568::TaskManager.Task::(_updateState) Task=`a6e79fd7-3bca-40bb-a214-664cb9941709`::moving from state init -> state preparing >Thread-217::INFO::2012-12-02 23:37:06,895::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-217::INFO::2012-12-02 23:37:06,895::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-217::DEBUG::2012-12-02 23:37:06,895::task::1151::TaskManager.Task::(prepare) Task=`a6e79fd7-3bca-40bb-a214-664cb9941709`::finished: {} >Thread-217::DEBUG::2012-12-02 23:37:06,896::task::568::TaskManager.Task::(_updateState) Task=`a6e79fd7-3bca-40bb-a214-664cb9941709`::moving from state preparing -> state finished >Thread-217::DEBUG::2012-12-02 23:37:06,896::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-217::DEBUG::2012-12-02 23:37:06,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-217::DEBUG::2012-12-02 23:37:06,896::task::957::TaskManager.Task::(_decref) Task=`a6e79fd7-3bca-40bb-a214-664cb9941709`::ref 0 aborting False >Thread-220::DEBUG::2012-12-02 23:37:11,248::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-220::DEBUG::2012-12-02 23:37:11,248::task::568::TaskManager.Task::(_updateState) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::moving from state init -> state preparing >Thread-220::INFO::2012-12-02 23:37:11,248::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-220::ERROR::2012-12-02 23:37:11,248::task::833::TaskManager.Task::(_setError) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-220::DEBUG::2012-12-02 23:37:11,248::task::852::TaskManager.Task::(_run) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::Task._run: e3a7c3e2-3d76-430b-a49e-f748b114fc66 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-220::DEBUG::2012-12-02 23:37:11,249::task::1177::TaskManager.Task::(stop) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::stopping in state preparing (force False) >Thread-220::DEBUG::2012-12-02 23:37:11,249::task::957::TaskManager.Task::(_decref) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::ref 1 aborting True >Thread-220::INFO::2012-12-02 23:37:11,249::task::1134::TaskManager.Task::(prepare) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-220::DEBUG::2012-12-02 23:37:11,249::task::1139::TaskManager.Task::(prepare) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::Prepare: aborted: Unknown pool id, pool not connected >Thread-220::DEBUG::2012-12-02 23:37:11,249::task::957::TaskManager.Task::(_decref) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::ref 0 aborting True >Thread-220::DEBUG::2012-12-02 23:37:11,249::task::892::TaskManager.Task::(_doAbort) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::Task._doAbort: force False >Thread-220::DEBUG::2012-12-02 23:37:11,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-220::DEBUG::2012-12-02 23:37:11,249::task::568::TaskManager.Task::(_updateState) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::moving from state preparing -> state aborting >Thread-220::DEBUG::2012-12-02 23:37:11,250::task::523::TaskManager.Task::(__state_aborting) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::_aborting: recover policy none >Thread-220::DEBUG::2012-12-02 23:37:11,250::task::568::TaskManager.Task::(_updateState) Task=`e3a7c3e2-3d76-430b-a49e-f748b114fc66`::moving from state aborting -> state failed >Thread-220::DEBUG::2012-12-02 23:37:11,250::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-220::DEBUG::2012-12-02 23:37:11,250::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-220::ERROR::2012-12-02 23:37:11,250::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-222::DEBUG::2012-12-02 23:37:11,398::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-222::DEBUG::2012-12-02 23:37:11,398::task::568::TaskManager.Task::(_updateState) Task=`03d8dcaa-b4fd-42e4-8529-5d5630f94f43`::moving from state init -> state preparing >Thread-222::INFO::2012-12-02 23:37:11,399::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-222::DEBUG::2012-12-02 23:37:11,399::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-222::DEBUG::2012-12-02 23:37:11,399::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-222::INFO::2012-12-02 23:37:11,399::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-222::DEBUG::2012-12-02 23:37:11,399::task::1151::TaskManager.Task::(prepare) Task=`03d8dcaa-b4fd-42e4-8529-5d5630f94f43`::finished: {'allTasksStatus': {}} >Thread-222::DEBUG::2012-12-02 23:37:11,399::task::568::TaskManager.Task::(_updateState) Task=`03d8dcaa-b4fd-42e4-8529-5d5630f94f43`::moving from state preparing -> state finished >Thread-222::DEBUG::2012-12-02 23:37:11,399::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-222::DEBUG::2012-12-02 23:37:11,399::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-222::DEBUG::2012-12-02 23:37:11,400::task::957::TaskManager.Task::(_decref) Task=`03d8dcaa-b4fd-42e4-8529-5d5630f94f43`::ref 0 aborting False >Thread-223::DEBUG::2012-12-02 23:37:11,544::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-223::DEBUG::2012-12-02 23:37:11,544::task::568::TaskManager.Task::(_updateState) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::moving from state init -> state preparing >Thread-223::INFO::2012-12-02 23:37:11,544::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-223::ERROR::2012-12-02 23:37:11,544::task::833::TaskManager.Task::(_setError) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-223::DEBUG::2012-12-02 23:37:11,545::task::852::TaskManager.Task::(_run) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::Task._run: bd45c9f8-a056-4873-bff7-211816025ee7 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-223::DEBUG::2012-12-02 23:37:11,545::task::1177::TaskManager.Task::(stop) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::stopping in state preparing (force False) >Thread-223::DEBUG::2012-12-02 23:37:11,545::task::957::TaskManager.Task::(_decref) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::ref 1 aborting True >Thread-223::INFO::2012-12-02 23:37:11,545::task::1134::TaskManager.Task::(prepare) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-223::DEBUG::2012-12-02 23:37:11,545::task::1139::TaskManager.Task::(prepare) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::Prepare: aborted: Unknown pool id, pool not connected >Thread-223::DEBUG::2012-12-02 23:37:11,545::task::957::TaskManager.Task::(_decref) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::ref 0 aborting True >Thread-223::DEBUG::2012-12-02 23:37:11,545::task::892::TaskManager.Task::(_doAbort) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::Task._doAbort: force False >Thread-223::DEBUG::2012-12-02 23:37:11,545::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-223::DEBUG::2012-12-02 23:37:11,546::task::568::TaskManager.Task::(_updateState) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::moving from state preparing -> state aborting >Thread-223::DEBUG::2012-12-02 23:37:11,546::task::523::TaskManager.Task::(__state_aborting) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::_aborting: recover policy none >Thread-223::DEBUG::2012-12-02 23:37:11,546::task::568::TaskManager.Task::(_updateState) Task=`bd45c9f8-a056-4873-bff7-211816025ee7`::moving from state aborting -> state failed >Thread-223::DEBUG::2012-12-02 23:37:11,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-223::DEBUG::2012-12-02 23:37:11,546::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-223::ERROR::2012-12-02 23:37:11,546::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-224::DEBUG::2012-12-02 23:37:11,766::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-224::DEBUG::2012-12-02 23:37:11,766::task::568::TaskManager.Task::(_updateState) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::moving from state init -> state preparing >Thread-224::INFO::2012-12-02 23:37:11,766::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-224::DEBUG::2012-12-02 23:37:11,767::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`11fbee6a-1c0c-41bb-a634-3d2cf4916a7a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-224::DEBUG::2012-12-02 23:37:11,767::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-224::DEBUG::2012-12-02 23:37:11,767::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-224::DEBUG::2012-12-02 23:37:11,767::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`11fbee6a-1c0c-41bb-a634-3d2cf4916a7a`::Granted request >Thread-224::INFO::2012-12-02 23:37:11,767::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-224::DEBUG::2012-12-02 23:37:11,768::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:11,768::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:11,768::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:11,768::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:11,768::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:11,769::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:11,769::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-224::DEBUG::2012-12-02 23:37:11,769::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-224::DEBUG::2012-12-02 23:37:11,769::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-224::DEBUG::2012-12-02 23:37:11,769::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-224::DEBUG::2012-12-02 23:37:11,769::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-224::DEBUG::2012-12-02 23:37:11,797::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-224::DEBUG::2012-12-02 23:37:11,797::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-224::DEBUG::2012-12-02 23:37:12,433::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-224::DEBUG::2012-12-02 23:37:12,502::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-224::DEBUG::2012-12-02 23:37:12,502::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,502::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,502::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,502::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,503::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,503::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,503::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-224::DEBUG::2012-12-02 23:37:12,503::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,505::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-224::DEBUG::2012-12-02 23:37:12,712::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-224::WARNING::2012-12-02 23:37:12,714::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-224::DEBUG::2012-12-02 23:37:12,714::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-224::DEBUG::2012-12-02 23:37:12,742::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-224::DEBUG::2012-12-02 23:37:12,742::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-224::DEBUG::2012-12-02 23:37:12,742::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-224::DEBUG::2012-12-02 23:37:12,743::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-224::ERROR::2012-12-02 23:37:12,743::task::833::TaskManager.Task::(_setError) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-224::DEBUG::2012-12-02 23:37:12,743::task::852::TaskManager.Task::(_run) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::Task._run: 25b79d1c-bab6-4498-aefd-2738c60f3667 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-224::DEBUG::2012-12-02 23:37:12,743::task::1177::TaskManager.Task::(stop) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::stopping in state preparing (force False) >Thread-224::DEBUG::2012-12-02 23:37:12,743::task::957::TaskManager.Task::(_decref) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::ref 1 aborting True >Thread-224::INFO::2012-12-02 23:37:12,744::task::1134::TaskManager.Task::(prepare) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-224::DEBUG::2012-12-02 23:37:12,744::task::1139::TaskManager.Task::(prepare) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::Prepare: aborted: Cannot find master domain >Thread-224::DEBUG::2012-12-02 23:37:12,744::task::957::TaskManager.Task::(_decref) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::ref 0 aborting True >Thread-224::DEBUG::2012-12-02 23:37:12,744::task::892::TaskManager.Task::(_doAbort) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::Task._doAbort: force False >Thread-224::DEBUG::2012-12-02 23:37:12,744::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-224::DEBUG::2012-12-02 23:37:12,744::task::568::TaskManager.Task::(_updateState) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::moving from state preparing -> state aborting >Thread-224::DEBUG::2012-12-02 23:37:12,744::task::523::TaskManager.Task::(__state_aborting) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::_aborting: recover policy none >Thread-224::DEBUG::2012-12-02 23:37:12,745::task::568::TaskManager.Task::(_updateState) Task=`25b79d1c-bab6-4498-aefd-2738c60f3667`::moving from state aborting -> state failed >Thread-224::DEBUG::2012-12-02 23:37:12,745::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-224::DEBUG::2012-12-02 23:37:12,745::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-224::ERROR::2012-12-02 23:37:12,745::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-288::DEBUG::2012-12-02 23:37:17,821::task::568::TaskManager.Task::(_updateState) Task=`d51023fb-8fe4-47ff-9438-ffb3a6234c79`::moving from state init -> state preparing >Thread-288::INFO::2012-12-02 23:37:17,821::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-288::INFO::2012-12-02 23:37:17,821::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-288::DEBUG::2012-12-02 23:37:17,821::task::1151::TaskManager.Task::(prepare) Task=`d51023fb-8fe4-47ff-9438-ffb3a6234c79`::finished: {} >Thread-288::DEBUG::2012-12-02 23:37:17,821::task::568::TaskManager.Task::(_updateState) Task=`d51023fb-8fe4-47ff-9438-ffb3a6234c79`::moving from state preparing -> state finished >Thread-288::DEBUG::2012-12-02 23:37:17,822::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-288::DEBUG::2012-12-02 23:37:17,822::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-288::DEBUG::2012-12-02 23:37:17,822::task::957::TaskManager.Task::(_decref) Task=`d51023fb-8fe4-47ff-9438-ffb3a6234c79`::ref 0 aborting False >Thread-292::DEBUG::2012-12-02 23:37:23,092::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-292::DEBUG::2012-12-02 23:37:23,092::task::568::TaskManager.Task::(_updateState) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::moving from state init -> state preparing >Thread-292::INFO::2012-12-02 23:37:23,092::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-292::ERROR::2012-12-02 23:37:23,092::task::833::TaskManager.Task::(_setError) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-292::DEBUG::2012-12-02 23:37:23,093::task::852::TaskManager.Task::(_run) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::Task._run: e545ee8e-1086-4e5d-b7bc-de4f430174b3 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-292::DEBUG::2012-12-02 23:37:23,093::task::1177::TaskManager.Task::(stop) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::stopping in state preparing (force False) >Thread-292::DEBUG::2012-12-02 23:37:23,093::task::957::TaskManager.Task::(_decref) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::ref 1 aborting True >Thread-292::INFO::2012-12-02 23:37:23,093::task::1134::TaskManager.Task::(prepare) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-292::DEBUG::2012-12-02 23:37:23,093::task::1139::TaskManager.Task::(prepare) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::Prepare: aborted: Unknown pool id, pool not connected >Thread-292::DEBUG::2012-12-02 23:37:23,093::task::957::TaskManager.Task::(_decref) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::ref 0 aborting True >Thread-292::DEBUG::2012-12-02 23:37:23,093::task::892::TaskManager.Task::(_doAbort) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::Task._doAbort: force False >Thread-292::DEBUG::2012-12-02 23:37:23,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-292::DEBUG::2012-12-02 23:37:23,094::task::568::TaskManager.Task::(_updateState) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::moving from state preparing -> state aborting >Thread-292::DEBUG::2012-12-02 23:37:23,094::task::523::TaskManager.Task::(__state_aborting) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::_aborting: recover policy none >Thread-292::DEBUG::2012-12-02 23:37:23,094::task::568::TaskManager.Task::(_updateState) Task=`e545ee8e-1086-4e5d-b7bc-de4f430174b3`::moving from state aborting -> state failed >Thread-292::DEBUG::2012-12-02 23:37:23,094::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-292::DEBUG::2012-12-02 23:37:23,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-292::ERROR::2012-12-02 23:37:23,094::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-293::DEBUG::2012-12-02 23:37:23,243::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-293::DEBUG::2012-12-02 23:37:23,244::task::568::TaskManager.Task::(_updateState) Task=`90e96e88-d489-4785-b258-e57975256569`::moving from state init -> state preparing >Thread-293::INFO::2012-12-02 23:37:23,244::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-293::DEBUG::2012-12-02 23:37:23,244::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-293::DEBUG::2012-12-02 23:37:23,244::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-293::INFO::2012-12-02 23:37:23,244::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-293::DEBUG::2012-12-02 23:37:23,245::task::1151::TaskManager.Task::(prepare) Task=`90e96e88-d489-4785-b258-e57975256569`::finished: {'allTasksStatus': {}} >Thread-293::DEBUG::2012-12-02 23:37:23,245::task::568::TaskManager.Task::(_updateState) Task=`90e96e88-d489-4785-b258-e57975256569`::moving from state preparing -> state finished >Thread-293::DEBUG::2012-12-02 23:37:23,245::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-293::DEBUG::2012-12-02 23:37:23,245::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-293::DEBUG::2012-12-02 23:37:23,245::task::957::TaskManager.Task::(_decref) Task=`90e96e88-d489-4785-b258-e57975256569`::ref 0 aborting False >Thread-294::DEBUG::2012-12-02 23:37:23,407::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-294::DEBUG::2012-12-02 23:37:23,407::task::568::TaskManager.Task::(_updateState) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::moving from state init -> state preparing >Thread-294::INFO::2012-12-02 23:37:23,408::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-294::ERROR::2012-12-02 23:37:23,408::task::833::TaskManager.Task::(_setError) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-294::DEBUG::2012-12-02 23:37:23,408::task::852::TaskManager.Task::(_run) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::Task._run: e44b0b68-134d-481d-a04c-e5d221ae2f65 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-294::DEBUG::2012-12-02 23:37:23,408::task::1177::TaskManager.Task::(stop) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::stopping in state preparing (force False) >Thread-294::DEBUG::2012-12-02 23:37:23,408::task::957::TaskManager.Task::(_decref) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::ref 1 aborting True >Thread-294::INFO::2012-12-02 23:37:23,408::task::1134::TaskManager.Task::(prepare) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-294::DEBUG::2012-12-02 23:37:23,409::task::1139::TaskManager.Task::(prepare) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::Prepare: aborted: Unknown pool id, pool not connected >Thread-294::DEBUG::2012-12-02 23:37:23,409::task::957::TaskManager.Task::(_decref) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::ref 0 aborting True >Thread-294::DEBUG::2012-12-02 23:37:23,409::task::892::TaskManager.Task::(_doAbort) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::Task._doAbort: force False >Thread-294::DEBUG::2012-12-02 23:37:23,409::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-294::DEBUG::2012-12-02 23:37:23,409::task::568::TaskManager.Task::(_updateState) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::moving from state preparing -> state aborting >Thread-294::DEBUG::2012-12-02 23:37:23,409::task::523::TaskManager.Task::(__state_aborting) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::_aborting: recover policy none >Thread-294::DEBUG::2012-12-02 23:37:23,409::task::568::TaskManager.Task::(_updateState) Task=`e44b0b68-134d-481d-a04c-e5d221ae2f65`::moving from state aborting -> state failed >Thread-294::DEBUG::2012-12-02 23:37:23,409::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-294::DEBUG::2012-12-02 23:37:23,410::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-294::ERROR::2012-12-02 23:37:23,410::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-295::DEBUG::2012-12-02 23:37:23,638::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-295::DEBUG::2012-12-02 23:37:23,638::task::568::TaskManager.Task::(_updateState) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::moving from state init -> state preparing >Thread-295::INFO::2012-12-02 23:37:23,638::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-295::DEBUG::2012-12-02 23:37:23,639::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`0ee9fb2a-b9da-4096-934f-4606f374e928`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-295::DEBUG::2012-12-02 23:37:23,639::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-295::DEBUG::2012-12-02 23:37:23,639::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-295::DEBUG::2012-12-02 23:37:23,639::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`0ee9fb2a-b9da-4096-934f-4606f374e928`::Granted request >Thread-295::INFO::2012-12-02 23:37:23,640::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-295::DEBUG::2012-12-02 23:37:23,640::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:23,640::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:23,640::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:23,640::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:23,641::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:23,641::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:23,641::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-295::DEBUG::2012-12-02 23:37:23,641::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-295::DEBUG::2012-12-02 23:37:23,641::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-295::DEBUG::2012-12-02 23:37:23,641::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-295::DEBUG::2012-12-02 23:37:23,641::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-295::DEBUG::2012-12-02 23:37:23,666::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-295::DEBUG::2012-12-02 23:37:23,666::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-295::DEBUG::2012-12-02 23:37:24,300::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-295::DEBUG::2012-12-02 23:37:24,366::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-295::DEBUG::2012-12-02 23:37:24,366::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,366::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,367::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,367::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,367::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,367::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,367::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-295::DEBUG::2012-12-02 23:37:24,367::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,369::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-295::DEBUG::2012-12-02 23:37:24,574::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-295::WARNING::2012-12-02 23:37:24,575::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-295::DEBUG::2012-12-02 23:37:24,575::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-295::DEBUG::2012-12-02 23:37:24,603::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-295::DEBUG::2012-12-02 23:37:24,604::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-295::DEBUG::2012-12-02 23:37:24,604::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-295::DEBUG::2012-12-02 23:37:24,604::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-295::ERROR::2012-12-02 23:37:24,604::task::833::TaskManager.Task::(_setError) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-295::DEBUG::2012-12-02 23:37:24,604::task::852::TaskManager.Task::(_run) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::Task._run: 0a937829-85fc-450b-a6c2-612cf1a5c731 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-295::DEBUG::2012-12-02 23:37:24,605::task::1177::TaskManager.Task::(stop) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::stopping in state preparing (force False) >Thread-295::DEBUG::2012-12-02 23:37:24,605::task::957::TaskManager.Task::(_decref) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::ref 1 aborting True >Thread-295::INFO::2012-12-02 23:37:24,605::task::1134::TaskManager.Task::(prepare) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-295::DEBUG::2012-12-02 23:37:24,605::task::1139::TaskManager.Task::(prepare) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::Prepare: aborted: Cannot find master domain >Thread-295::DEBUG::2012-12-02 23:37:24,605::task::957::TaskManager.Task::(_decref) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::ref 0 aborting True >Thread-295::DEBUG::2012-12-02 23:37:24,605::task::892::TaskManager.Task::(_doAbort) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::Task._doAbort: force False >Thread-295::DEBUG::2012-12-02 23:37:24,605::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-295::DEBUG::2012-12-02 23:37:24,606::task::568::TaskManager.Task::(_updateState) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::moving from state preparing -> state aborting >Thread-295::DEBUG::2012-12-02 23:37:24,606::task::523::TaskManager.Task::(__state_aborting) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::_aborting: recover policy none >Thread-295::DEBUG::2012-12-02 23:37:24,606::task::568::TaskManager.Task::(_updateState) Task=`0a937829-85fc-450b-a6c2-612cf1a5c731`::moving from state aborting -> state failed >Thread-295::DEBUG::2012-12-02 23:37:24,606::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-295::DEBUG::2012-12-02 23:37:24,606::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-295::ERROR::2012-12-02 23:37:24,606::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-359::DEBUG::2012-12-02 23:37:28,909::task::568::TaskManager.Task::(_updateState) Task=`2ca79dca-7d0e-4367-b57e-f0c96585323b`::moving from state init -> state preparing >Thread-359::INFO::2012-12-02 23:37:28,909::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-359::INFO::2012-12-02 23:37:28,910::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-359::DEBUG::2012-12-02 23:37:28,910::task::1151::TaskManager.Task::(prepare) Task=`2ca79dca-7d0e-4367-b57e-f0c96585323b`::finished: {} >Thread-359::DEBUG::2012-12-02 23:37:28,910::task::568::TaskManager.Task::(_updateState) Task=`2ca79dca-7d0e-4367-b57e-f0c96585323b`::moving from state preparing -> state finished >Thread-359::DEBUG::2012-12-02 23:37:28,910::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-359::DEBUG::2012-12-02 23:37:28,910::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-359::DEBUG::2012-12-02 23:37:28,910::task::957::TaskManager.Task::(_decref) Task=`2ca79dca-7d0e-4367-b57e-f0c96585323b`::ref 0 aborting False >Thread-363::DEBUG::2012-12-02 23:37:34,897::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-363::DEBUG::2012-12-02 23:37:34,898::task::568::TaskManager.Task::(_updateState) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::moving from state init -> state preparing >Thread-363::INFO::2012-12-02 23:37:34,898::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-363::ERROR::2012-12-02 23:37:34,898::task::833::TaskManager.Task::(_setError) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-363::DEBUG::2012-12-02 23:37:34,898::task::852::TaskManager.Task::(_run) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::Task._run: 93cb6be5-eb0f-4afb-a965-a1bd19241496 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-363::DEBUG::2012-12-02 23:37:34,898::task::1177::TaskManager.Task::(stop) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::stopping in state preparing (force False) >Thread-363::DEBUG::2012-12-02 23:37:34,898::task::957::TaskManager.Task::(_decref) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::ref 1 aborting True >Thread-363::INFO::2012-12-02 23:37:34,899::task::1134::TaskManager.Task::(prepare) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-363::DEBUG::2012-12-02 23:37:34,899::task::1139::TaskManager.Task::(prepare) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::Prepare: aborted: Unknown pool id, pool not connected >Thread-363::DEBUG::2012-12-02 23:37:34,899::task::957::TaskManager.Task::(_decref) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::ref 0 aborting True >Thread-363::DEBUG::2012-12-02 23:37:34,899::task::892::TaskManager.Task::(_doAbort) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::Task._doAbort: force False >Thread-363::DEBUG::2012-12-02 23:37:34,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-363::DEBUG::2012-12-02 23:37:34,899::task::568::TaskManager.Task::(_updateState) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::moving from state preparing -> state aborting >Thread-363::DEBUG::2012-12-02 23:37:34,899::task::523::TaskManager.Task::(__state_aborting) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::_aborting: recover policy none >Thread-363::DEBUG::2012-12-02 23:37:34,899::task::568::TaskManager.Task::(_updateState) Task=`93cb6be5-eb0f-4afb-a965-a1bd19241496`::moving from state aborting -> state failed >Thread-363::DEBUG::2012-12-02 23:37:34,900::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-363::DEBUG::2012-12-02 23:37:34,900::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-363::ERROR::2012-12-02 23:37:34,900::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-364::DEBUG::2012-12-02 23:37:35,037::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-364::DEBUG::2012-12-02 23:37:35,038::task::568::TaskManager.Task::(_updateState) Task=`512d9bd7-ae6f-44e7-8a2c-21457d216184`::moving from state init -> state preparing >Thread-364::INFO::2012-12-02 23:37:35,038::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-364::DEBUG::2012-12-02 23:37:35,038::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-364::DEBUG::2012-12-02 23:37:35,038::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-364::INFO::2012-12-02 23:37:35,038::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-364::DEBUG::2012-12-02 23:37:35,038::task::1151::TaskManager.Task::(prepare) Task=`512d9bd7-ae6f-44e7-8a2c-21457d216184`::finished: {'allTasksStatus': {}} >Thread-364::DEBUG::2012-12-02 23:37:35,038::task::568::TaskManager.Task::(_updateState) Task=`512d9bd7-ae6f-44e7-8a2c-21457d216184`::moving from state preparing -> state finished >Thread-364::DEBUG::2012-12-02 23:37:35,039::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-364::DEBUG::2012-12-02 23:37:35,039::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-364::DEBUG::2012-12-02 23:37:35,039::task::957::TaskManager.Task::(_decref) Task=`512d9bd7-ae6f-44e7-8a2c-21457d216184`::ref 0 aborting False >Thread-365::DEBUG::2012-12-02 23:37:35,190::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-365::DEBUG::2012-12-02 23:37:35,190::task::568::TaskManager.Task::(_updateState) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::moving from state init -> state preparing >Thread-365::INFO::2012-12-02 23:37:35,190::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-365::ERROR::2012-12-02 23:37:35,190::task::833::TaskManager.Task::(_setError) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-365::DEBUG::2012-12-02 23:37:35,191::task::852::TaskManager.Task::(_run) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::Task._run: cdba29dd-e4a3-421e-ad2f-21414483a301 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-365::DEBUG::2012-12-02 23:37:35,191::task::1177::TaskManager.Task::(stop) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::stopping in state preparing (force False) >Thread-365::DEBUG::2012-12-02 23:37:35,191::task::957::TaskManager.Task::(_decref) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::ref 1 aborting True >Thread-365::INFO::2012-12-02 23:37:35,191::task::1134::TaskManager.Task::(prepare) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-365::DEBUG::2012-12-02 23:37:35,191::task::1139::TaskManager.Task::(prepare) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::Prepare: aborted: Unknown pool id, pool not connected >Thread-365::DEBUG::2012-12-02 23:37:35,191::task::957::TaskManager.Task::(_decref) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::ref 0 aborting True >Thread-365::DEBUG::2012-12-02 23:37:35,191::task::892::TaskManager.Task::(_doAbort) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::Task._doAbort: force False >Thread-365::DEBUG::2012-12-02 23:37:35,192::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-365::DEBUG::2012-12-02 23:37:35,192::task::568::TaskManager.Task::(_updateState) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::moving from state preparing -> state aborting >Thread-365::DEBUG::2012-12-02 23:37:35,192::task::523::TaskManager.Task::(__state_aborting) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::_aborting: recover policy none >Thread-365::DEBUG::2012-12-02 23:37:35,192::task::568::TaskManager.Task::(_updateState) Task=`cdba29dd-e4a3-421e-ad2f-21414483a301`::moving from state aborting -> state failed >Thread-365::DEBUG::2012-12-02 23:37:35,192::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-365::DEBUG::2012-12-02 23:37:35,192::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-365::ERROR::2012-12-02 23:37:35,192::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-366::DEBUG::2012-12-02 23:37:35,384::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-366::DEBUG::2012-12-02 23:37:35,385::task::568::TaskManager.Task::(_updateState) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::moving from state init -> state preparing >Thread-366::INFO::2012-12-02 23:37:35,385::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-366::DEBUG::2012-12-02 23:37:35,385::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ff398898-d411-4090-95fe-dc32ae369ff3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-366::DEBUG::2012-12-02 23:37:35,385::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-366::DEBUG::2012-12-02 23:37:35,385::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-366::DEBUG::2012-12-02 23:37:35,385::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ff398898-d411-4090-95fe-dc32ae369ff3`::Granted request >Thread-366::INFO::2012-12-02 23:37:35,386::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-366::DEBUG::2012-12-02 23:37:35,386::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:35,386::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:35,386::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:35,386::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:35,386::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:35,386::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:35,387::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-366::DEBUG::2012-12-02 23:37:35,387::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-366::DEBUG::2012-12-02 23:37:35,387::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-366::DEBUG::2012-12-02 23:37:35,387::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-366::DEBUG::2012-12-02 23:37:35,387::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-366::DEBUG::2012-12-02 23:37:35,409::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-366::DEBUG::2012-12-02 23:37:35,409::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-366::DEBUG::2012-12-02 23:37:36,054::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-366::DEBUG::2012-12-02 23:37:36,117::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-366::DEBUG::2012-12-02 23:37:36,117::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,117::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,117::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,118::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,118::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,118::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,118::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-366::DEBUG::2012-12-02 23:37:36,118::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,120::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-366::DEBUG::2012-12-02 23:37:36,323::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-366::WARNING::2012-12-02 23:37:36,324::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-366::DEBUG::2012-12-02 23:37:36,324::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-366::DEBUG::2012-12-02 23:37:36,356::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-366::DEBUG::2012-12-02 23:37:36,356::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-366::DEBUG::2012-12-02 23:37:36,356::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-366::DEBUG::2012-12-02 23:37:36,356::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-366::ERROR::2012-12-02 23:37:36,357::task::833::TaskManager.Task::(_setError) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-366::DEBUG::2012-12-02 23:37:36,357::task::852::TaskManager.Task::(_run) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::Task._run: f72f9a92-6ff5-4578-a19b-869ef9dc0ca6 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-366::DEBUG::2012-12-02 23:37:36,357::task::1177::TaskManager.Task::(stop) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::stopping in state preparing (force False) >Thread-366::DEBUG::2012-12-02 23:37:36,357::task::957::TaskManager.Task::(_decref) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::ref 1 aborting True >Thread-366::INFO::2012-12-02 23:37:36,357::task::1134::TaskManager.Task::(prepare) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-366::DEBUG::2012-12-02 23:37:36,358::task::1139::TaskManager.Task::(prepare) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::Prepare: aborted: Cannot find master domain >Thread-366::DEBUG::2012-12-02 23:37:36,358::task::957::TaskManager.Task::(_decref) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::ref 0 aborting True >Thread-366::DEBUG::2012-12-02 23:37:36,358::task::892::TaskManager.Task::(_doAbort) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::Task._doAbort: force False >Thread-366::DEBUG::2012-12-02 23:37:36,358::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-366::DEBUG::2012-12-02 23:37:36,358::task::568::TaskManager.Task::(_updateState) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::moving from state preparing -> state aborting >Thread-366::DEBUG::2012-12-02 23:37:36,358::task::523::TaskManager.Task::(__state_aborting) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::_aborting: recover policy none >Thread-366::DEBUG::2012-12-02 23:37:36,358::task::568::TaskManager.Task::(_updateState) Task=`f72f9a92-6ff5-4578-a19b-869ef9dc0ca6`::moving from state aborting -> state failed >Thread-366::DEBUG::2012-12-02 23:37:36,358::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-366::DEBUG::2012-12-02 23:37:36,358::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-366::ERROR::2012-12-02 23:37:36,359::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-430::DEBUG::2012-12-02 23:37:39,852::task::568::TaskManager.Task::(_updateState) Task=`abf2868d-06b0-4747-aea5-2e5115fd3161`::moving from state init -> state preparing >Thread-430::INFO::2012-12-02 23:37:39,852::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-430::INFO::2012-12-02 23:37:39,852::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-430::DEBUG::2012-12-02 23:37:39,852::task::1151::TaskManager.Task::(prepare) Task=`abf2868d-06b0-4747-aea5-2e5115fd3161`::finished: {} >Thread-430::DEBUG::2012-12-02 23:37:39,852::task::568::TaskManager.Task::(_updateState) Task=`abf2868d-06b0-4747-aea5-2e5115fd3161`::moving from state preparing -> state finished >Thread-430::DEBUG::2012-12-02 23:37:39,853::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-430::DEBUG::2012-12-02 23:37:39,853::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-430::DEBUG::2012-12-02 23:37:39,853::task::957::TaskManager.Task::(_decref) Task=`abf2868d-06b0-4747-aea5-2e5115fd3161`::ref 0 aborting False >Thread-435::DEBUG::2012-12-02 23:37:46,731::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-435::DEBUG::2012-12-02 23:37:46,731::task::568::TaskManager.Task::(_updateState) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::moving from state init -> state preparing >Thread-435::INFO::2012-12-02 23:37:46,731::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-435::ERROR::2012-12-02 23:37:46,732::task::833::TaskManager.Task::(_setError) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-435::DEBUG::2012-12-02 23:37:46,732::task::852::TaskManager.Task::(_run) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::Task._run: 4da7d082-2b81-45d8-8ee3-13525bf6e658 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-435::DEBUG::2012-12-02 23:37:46,732::task::1177::TaskManager.Task::(stop) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::stopping in state preparing (force False) >Thread-435::DEBUG::2012-12-02 23:37:46,732::task::957::TaskManager.Task::(_decref) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::ref 1 aborting True >Thread-435::INFO::2012-12-02 23:37:46,732::task::1134::TaskManager.Task::(prepare) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-435::DEBUG::2012-12-02 23:37:46,732::task::1139::TaskManager.Task::(prepare) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::Prepare: aborted: Unknown pool id, pool not connected >Thread-435::DEBUG::2012-12-02 23:37:46,733::task::957::TaskManager.Task::(_decref) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::ref 0 aborting True >Thread-435::DEBUG::2012-12-02 23:37:46,733::task::892::TaskManager.Task::(_doAbort) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::Task._doAbort: force False >Thread-435::DEBUG::2012-12-02 23:37:46,733::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-435::DEBUG::2012-12-02 23:37:46,733::task::568::TaskManager.Task::(_updateState) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::moving from state preparing -> state aborting >Thread-435::DEBUG::2012-12-02 23:37:46,733::task::523::TaskManager.Task::(__state_aborting) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::_aborting: recover policy none >Thread-435::DEBUG::2012-12-02 23:37:46,733::task::568::TaskManager.Task::(_updateState) Task=`4da7d082-2b81-45d8-8ee3-13525bf6e658`::moving from state aborting -> state failed >Thread-435::DEBUG::2012-12-02 23:37:46,733::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-435::DEBUG::2012-12-02 23:37:46,733::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-435::ERROR::2012-12-02 23:37:46,733::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-436::DEBUG::2012-12-02 23:37:46,875::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-436::DEBUG::2012-12-02 23:37:46,875::task::568::TaskManager.Task::(_updateState) Task=`a942ac66-7005-4d98-93d6-aaa486cda372`::moving from state init -> state preparing >Thread-436::INFO::2012-12-02 23:37:46,875::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-436::DEBUG::2012-12-02 23:37:46,876::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-436::DEBUG::2012-12-02 23:37:46,876::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-436::INFO::2012-12-02 23:37:46,876::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-436::DEBUG::2012-12-02 23:37:46,876::task::1151::TaskManager.Task::(prepare) Task=`a942ac66-7005-4d98-93d6-aaa486cda372`::finished: {'allTasksStatus': {}} >Thread-436::DEBUG::2012-12-02 23:37:46,876::task::568::TaskManager.Task::(_updateState) Task=`a942ac66-7005-4d98-93d6-aaa486cda372`::moving from state preparing -> state finished >Thread-436::DEBUG::2012-12-02 23:37:46,876::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-436::DEBUG::2012-12-02 23:37:46,876::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-436::DEBUG::2012-12-02 23:37:46,876::task::957::TaskManager.Task::(_decref) Task=`a942ac66-7005-4d98-93d6-aaa486cda372`::ref 0 aborting False >Thread-437::DEBUG::2012-12-02 23:37:47,025::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-437::DEBUG::2012-12-02 23:37:47,026::task::568::TaskManager.Task::(_updateState) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::moving from state init -> state preparing >Thread-437::INFO::2012-12-02 23:37:47,026::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-437::ERROR::2012-12-02 23:37:47,026::task::833::TaskManager.Task::(_setError) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-437::DEBUG::2012-12-02 23:37:47,026::task::852::TaskManager.Task::(_run) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::Task._run: 8d2a2591-bc74-46e0-a7e8-60c90f394b29 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-437::DEBUG::2012-12-02 23:37:47,026::task::1177::TaskManager.Task::(stop) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::stopping in state preparing (force False) >Thread-437::DEBUG::2012-12-02 23:37:47,026::task::957::TaskManager.Task::(_decref) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::ref 1 aborting True >Thread-437::INFO::2012-12-02 23:37:47,027::task::1134::TaskManager.Task::(prepare) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-437::DEBUG::2012-12-02 23:37:47,027::task::1139::TaskManager.Task::(prepare) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::Prepare: aborted: Unknown pool id, pool not connected >Thread-437::DEBUG::2012-12-02 23:37:47,027::task::957::TaskManager.Task::(_decref) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::ref 0 aborting True >Thread-437::DEBUG::2012-12-02 23:37:47,027::task::892::TaskManager.Task::(_doAbort) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::Task._doAbort: force False >Thread-437::DEBUG::2012-12-02 23:37:47,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-437::DEBUG::2012-12-02 23:37:47,027::task::568::TaskManager.Task::(_updateState) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::moving from state preparing -> state aborting >Thread-437::DEBUG::2012-12-02 23:37:47,027::task::523::TaskManager.Task::(__state_aborting) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::_aborting: recover policy none >Thread-437::DEBUG::2012-12-02 23:37:47,027::task::568::TaskManager.Task::(_updateState) Task=`8d2a2591-bc74-46e0-a7e8-60c90f394b29`::moving from state aborting -> state failed >Thread-437::DEBUG::2012-12-02 23:37:47,027::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-437::DEBUG::2012-12-02 23:37:47,028::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-437::ERROR::2012-12-02 23:37:47,028::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-438::DEBUG::2012-12-02 23:37:47,218::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-438::DEBUG::2012-12-02 23:37:47,218::task::568::TaskManager.Task::(_updateState) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::moving from state init -> state preparing >Thread-438::INFO::2012-12-02 23:37:47,219::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-438::DEBUG::2012-12-02 23:37:47,219::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e2b74926-2544-4805-9f31-bc5d9feb4dc4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-438::DEBUG::2012-12-02 23:37:47,219::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-438::DEBUG::2012-12-02 23:37:47,219::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-438::DEBUG::2012-12-02 23:37:47,219::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e2b74926-2544-4805-9f31-bc5d9feb4dc4`::Granted request >Thread-438::INFO::2012-12-02 23:37:47,220::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-438::DEBUG::2012-12-02 23:37:47,220::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,220::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,220::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,220::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,221::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,221::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,221::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-438::DEBUG::2012-12-02 23:37:47,221::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-438::DEBUG::2012-12-02 23:37:47,221::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-438::DEBUG::2012-12-02 23:37:47,221::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-438::DEBUG::2012-12-02 23:37:47,221::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-438::DEBUG::2012-12-02 23:37:47,246::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-438::DEBUG::2012-12-02 23:37:47,246::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-438::DEBUG::2012-12-02 23:37:47,904::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-438::DEBUG::2012-12-02 23:37:47,967::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-438::DEBUG::2012-12-02 23:37:47,968::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,968::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,968::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,968::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,968::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,968::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,969::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-438::DEBUG::2012-12-02 23:37:47,969::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:47,970::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-438::DEBUG::2012-12-02 23:37:48,175::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-438::WARNING::2012-12-02 23:37:48,176::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-438::DEBUG::2012-12-02 23:37:48,176::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-438::DEBUG::2012-12-02 23:37:48,207::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-438::DEBUG::2012-12-02 23:37:48,208::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-438::DEBUG::2012-12-02 23:37:48,208::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-438::DEBUG::2012-12-02 23:37:48,208::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-438::ERROR::2012-12-02 23:37:48,208::task::833::TaskManager.Task::(_setError) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-438::DEBUG::2012-12-02 23:37:48,208::task::852::TaskManager.Task::(_run) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::Task._run: 13d6ff2f-1b94-4939-a0a5-80d4d6631b8f ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-438::DEBUG::2012-12-02 23:37:48,209::task::1177::TaskManager.Task::(stop) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::stopping in state preparing (force False) >Thread-438::DEBUG::2012-12-02 23:37:48,209::task::957::TaskManager.Task::(_decref) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::ref 1 aborting True >Thread-438::INFO::2012-12-02 23:37:48,209::task::1134::TaskManager.Task::(prepare) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-438::DEBUG::2012-12-02 23:37:48,209::task::1139::TaskManager.Task::(prepare) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::Prepare: aborted: Cannot find master domain >Thread-438::DEBUG::2012-12-02 23:37:48,209::task::957::TaskManager.Task::(_decref) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::ref 0 aborting True >Thread-438::DEBUG::2012-12-02 23:37:48,209::task::892::TaskManager.Task::(_doAbort) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::Task._doAbort: force False >Thread-438::DEBUG::2012-12-02 23:37:48,209::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-438::DEBUG::2012-12-02 23:37:48,210::task::568::TaskManager.Task::(_updateState) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::moving from state preparing -> state aborting >Thread-438::DEBUG::2012-12-02 23:37:48,210::task::523::TaskManager.Task::(__state_aborting) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::_aborting: recover policy none >Thread-438::DEBUG::2012-12-02 23:37:48,210::task::568::TaskManager.Task::(_updateState) Task=`13d6ff2f-1b94-4939-a0a5-80d4d6631b8f`::moving from state aborting -> state failed >Thread-438::DEBUG::2012-12-02 23:37:48,210::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-438::DEBUG::2012-12-02 23:37:48,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-438::ERROR::2012-12-02 23:37:48,210::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-501::DEBUG::2012-12-02 23:37:50,816::task::568::TaskManager.Task::(_updateState) Task=`8fe7d822-9a52-4577-bfa8-9003f5708f69`::moving from state init -> state preparing >Thread-501::INFO::2012-12-02 23:37:50,816::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-501::INFO::2012-12-02 23:37:50,816::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-501::DEBUG::2012-12-02 23:37:50,817::task::1151::TaskManager.Task::(prepare) Task=`8fe7d822-9a52-4577-bfa8-9003f5708f69`::finished: {} >Thread-501::DEBUG::2012-12-02 23:37:50,817::task::568::TaskManager.Task::(_updateState) Task=`8fe7d822-9a52-4577-bfa8-9003f5708f69`::moving from state preparing -> state finished >Thread-501::DEBUG::2012-12-02 23:37:50,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-501::DEBUG::2012-12-02 23:37:50,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-501::DEBUG::2012-12-02 23:37:50,817::task::957::TaskManager.Task::(_decref) Task=`8fe7d822-9a52-4577-bfa8-9003f5708f69`::ref 0 aborting False >Thread-506::DEBUG::2012-12-02 23:37:58,569::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-506::DEBUG::2012-12-02 23:37:58,569::task::568::TaskManager.Task::(_updateState) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::moving from state init -> state preparing >Thread-506::INFO::2012-12-02 23:37:58,569::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-506::ERROR::2012-12-02 23:37:58,569::task::833::TaskManager.Task::(_setError) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-506::DEBUG::2012-12-02 23:37:58,570::task::852::TaskManager.Task::(_run) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::Task._run: 1ed67e53-fc13-47cf-90a8-0d6f551925c9 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-506::DEBUG::2012-12-02 23:37:58,570::task::1177::TaskManager.Task::(stop) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::stopping in state preparing (force False) >Thread-506::DEBUG::2012-12-02 23:37:58,570::task::957::TaskManager.Task::(_decref) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::ref 1 aborting True >Thread-506::INFO::2012-12-02 23:37:58,570::task::1134::TaskManager.Task::(prepare) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-506::DEBUG::2012-12-02 23:37:58,570::task::1139::TaskManager.Task::(prepare) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::Prepare: aborted: Unknown pool id, pool not connected >Thread-506::DEBUG::2012-12-02 23:37:58,570::task::957::TaskManager.Task::(_decref) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::ref 0 aborting True >Thread-506::DEBUG::2012-12-02 23:37:58,570::task::892::TaskManager.Task::(_doAbort) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::Task._doAbort: force False >Thread-506::DEBUG::2012-12-02 23:37:58,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-506::DEBUG::2012-12-02 23:37:58,571::task::568::TaskManager.Task::(_updateState) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::moving from state preparing -> state aborting >Thread-506::DEBUG::2012-12-02 23:37:58,571::task::523::TaskManager.Task::(__state_aborting) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::_aborting: recover policy none >Thread-506::DEBUG::2012-12-02 23:37:58,571::task::568::TaskManager.Task::(_updateState) Task=`1ed67e53-fc13-47cf-90a8-0d6f551925c9`::moving from state aborting -> state failed >Thread-506::DEBUG::2012-12-02 23:37:58,571::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-506::DEBUG::2012-12-02 23:37:58,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-506::ERROR::2012-12-02 23:37:58,571::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-507::DEBUG::2012-12-02 23:37:58,718::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-507::DEBUG::2012-12-02 23:37:58,718::task::568::TaskManager.Task::(_updateState) Task=`d9324d39-a70c-4b4f-a893-95e71fbb6c2f`::moving from state init -> state preparing >Thread-507::INFO::2012-12-02 23:37:58,718::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-507::DEBUG::2012-12-02 23:37:58,718::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-507::DEBUG::2012-12-02 23:37:58,718::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-507::INFO::2012-12-02 23:37:58,718::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-507::DEBUG::2012-12-02 23:37:58,719::task::1151::TaskManager.Task::(prepare) Task=`d9324d39-a70c-4b4f-a893-95e71fbb6c2f`::finished: {'allTasksStatus': {}} >Thread-507::DEBUG::2012-12-02 23:37:58,719::task::568::TaskManager.Task::(_updateState) Task=`d9324d39-a70c-4b4f-a893-95e71fbb6c2f`::moving from state preparing -> state finished >Thread-507::DEBUG::2012-12-02 23:37:58,719::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-507::DEBUG::2012-12-02 23:37:58,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-507::DEBUG::2012-12-02 23:37:58,719::task::957::TaskManager.Task::(_decref) Task=`d9324d39-a70c-4b4f-a893-95e71fbb6c2f`::ref 0 aborting False >Thread-508::DEBUG::2012-12-02 23:37:58,867::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-508::DEBUG::2012-12-02 23:37:58,867::task::568::TaskManager.Task::(_updateState) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::moving from state init -> state preparing >Thread-508::INFO::2012-12-02 23:37:58,868::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-508::ERROR::2012-12-02 23:37:58,868::task::833::TaskManager.Task::(_setError) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-508::DEBUG::2012-12-02 23:37:58,868::task::852::TaskManager.Task::(_run) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::Task._run: 660b1823-d0b1-4182-9bff-d97197ac6812 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-508::DEBUG::2012-12-02 23:37:58,868::task::1177::TaskManager.Task::(stop) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::stopping in state preparing (force False) >Thread-508::DEBUG::2012-12-02 23:37:58,868::task::957::TaskManager.Task::(_decref) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::ref 1 aborting True >Thread-508::INFO::2012-12-02 23:37:58,868::task::1134::TaskManager.Task::(prepare) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-508::DEBUG::2012-12-02 23:37:58,868::task::1139::TaskManager.Task::(prepare) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::Prepare: aborted: Unknown pool id, pool not connected >Thread-508::DEBUG::2012-12-02 23:37:58,869::task::957::TaskManager.Task::(_decref) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::ref 0 aborting True >Thread-508::DEBUG::2012-12-02 23:37:58,869::task::892::TaskManager.Task::(_doAbort) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::Task._doAbort: force False >Thread-508::DEBUG::2012-12-02 23:37:58,869::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-508::DEBUG::2012-12-02 23:37:58,869::task::568::TaskManager.Task::(_updateState) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::moving from state preparing -> state aborting >Thread-508::DEBUG::2012-12-02 23:37:58,869::task::523::TaskManager.Task::(__state_aborting) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::_aborting: recover policy none >Thread-508::DEBUG::2012-12-02 23:37:58,869::task::568::TaskManager.Task::(_updateState) Task=`660b1823-d0b1-4182-9bff-d97197ac6812`::moving from state aborting -> state failed >Thread-508::DEBUG::2012-12-02 23:37:58,869::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-508::DEBUG::2012-12-02 23:37:58,869::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-508::ERROR::2012-12-02 23:37:58,870::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-509::DEBUG::2012-12-02 23:37:59,065::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-509::DEBUG::2012-12-02 23:37:59,065::task::568::TaskManager.Task::(_updateState) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::moving from state init -> state preparing >Thread-509::INFO::2012-12-02 23:37:59,066::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-509::DEBUG::2012-12-02 23:37:59,066::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2ee74491-628d-486c-8e2d-372ec9bddfc8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-509::DEBUG::2012-12-02 23:37:59,066::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-509::DEBUG::2012-12-02 23:37:59,066::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-509::DEBUG::2012-12-02 23:37:59,066::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2ee74491-628d-486c-8e2d-372ec9bddfc8`::Granted request >Thread-509::INFO::2012-12-02 23:37:59,067::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-509::DEBUG::2012-12-02 23:37:59,067::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,067::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,067::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,067::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,068::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,068::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,068::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-509::DEBUG::2012-12-02 23:37:59,068::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-509::DEBUG::2012-12-02 23:37:59,068::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-509::DEBUG::2012-12-02 23:37:59,068::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-509::DEBUG::2012-12-02 23:37:59,068::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-509::DEBUG::2012-12-02 23:37:59,092::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-509::DEBUG::2012-12-02 23:37:59,092::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-509::DEBUG::2012-12-02 23:37:59,725::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-509::DEBUG::2012-12-02 23:37:59,817::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-509::DEBUG::2012-12-02 23:37:59,817::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,820::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,820::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,820::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,820::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,820::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,820::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-509::DEBUG::2012-12-02 23:37:59,821::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-509::DEBUG::2012-12-02 23:37:59,822::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-509::DEBUG::2012-12-02 23:38:00,029::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-509::WARNING::2012-12-02 23:38:00,030::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-509::DEBUG::2012-12-02 23:38:00,030::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-509::DEBUG::2012-12-02 23:38:00,060::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-509::DEBUG::2012-12-02 23:38:00,060::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-509::DEBUG::2012-12-02 23:38:00,060::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-509::DEBUG::2012-12-02 23:38:00,061::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-509::ERROR::2012-12-02 23:38:00,061::task::833::TaskManager.Task::(_setError) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-509::DEBUG::2012-12-02 23:38:00,061::task::852::TaskManager.Task::(_run) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::Task._run: aef26106-4fb0-486f-80ec-fa91d4b58638 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-509::DEBUG::2012-12-02 23:38:00,061::task::1177::TaskManager.Task::(stop) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::stopping in state preparing (force False) >Thread-509::DEBUG::2012-12-02 23:38:00,061::task::957::TaskManager.Task::(_decref) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::ref 1 aborting True >Thread-509::INFO::2012-12-02 23:38:00,062::task::1134::TaskManager.Task::(prepare) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-509::DEBUG::2012-12-02 23:38:00,062::task::1139::TaskManager.Task::(prepare) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::Prepare: aborted: Cannot find master domain >Thread-509::DEBUG::2012-12-02 23:38:00,062::task::957::TaskManager.Task::(_decref) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::ref 0 aborting True >Thread-509::DEBUG::2012-12-02 23:38:00,062::task::892::TaskManager.Task::(_doAbort) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::Task._doAbort: force False >Thread-509::DEBUG::2012-12-02 23:38:00,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-509::DEBUG::2012-12-02 23:38:00,062::task::568::TaskManager.Task::(_updateState) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::moving from state preparing -> state aborting >Thread-509::DEBUG::2012-12-02 23:38:00,062::task::523::TaskManager.Task::(__state_aborting) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::_aborting: recover policy none >Thread-509::DEBUG::2012-12-02 23:38:00,063::task::568::TaskManager.Task::(_updateState) Task=`aef26106-4fb0-486f-80ec-fa91d4b58638`::moving from state aborting -> state failed >Thread-509::DEBUG::2012-12-02 23:38:00,063::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-509::DEBUG::2012-12-02 23:38:00,063::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-509::ERROR::2012-12-02 23:38:00,063::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-572::DEBUG::2012-12-02 23:38:01,727::task::568::TaskManager.Task::(_updateState) Task=`2c781845-46de-49c6-ba4e-8a80a3252412`::moving from state init -> state preparing >Thread-572::INFO::2012-12-02 23:38:01,727::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-572::INFO::2012-12-02 23:38:01,727::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-572::DEBUG::2012-12-02 23:38:01,728::task::1151::TaskManager.Task::(prepare) Task=`2c781845-46de-49c6-ba4e-8a80a3252412`::finished: {} >Thread-572::DEBUG::2012-12-02 23:38:01,728::task::568::TaskManager.Task::(_updateState) Task=`2c781845-46de-49c6-ba4e-8a80a3252412`::moving from state preparing -> state finished >Thread-572::DEBUG::2012-12-02 23:38:01,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-572::DEBUG::2012-12-02 23:38:01,728::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-572::DEBUG::2012-12-02 23:38:01,728::task::957::TaskManager.Task::(_decref) Task=`2c781845-46de-49c6-ba4e-8a80a3252412`::ref 0 aborting False >Thread-577::DEBUG::2012-12-02 23:38:10,420::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-577::DEBUG::2012-12-02 23:38:10,420::task::568::TaskManager.Task::(_updateState) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::moving from state init -> state preparing >Thread-577::INFO::2012-12-02 23:38:10,421::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-577::ERROR::2012-12-02 23:38:10,421::task::833::TaskManager.Task::(_setError) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-577::DEBUG::2012-12-02 23:38:10,421::task::852::TaskManager.Task::(_run) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::Task._run: a9374c52-9b4b-44df-8a99-561d6c422510 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-577::DEBUG::2012-12-02 23:38:10,421::task::1177::TaskManager.Task::(stop) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::stopping in state preparing (force False) >Thread-577::DEBUG::2012-12-02 23:38:10,421::task::957::TaskManager.Task::(_decref) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::ref 1 aborting True >Thread-577::INFO::2012-12-02 23:38:10,421::task::1134::TaskManager.Task::(prepare) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-577::DEBUG::2012-12-02 23:38:10,422::task::1139::TaskManager.Task::(prepare) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::Prepare: aborted: Unknown pool id, pool not connected >Thread-577::DEBUG::2012-12-02 23:38:10,422::task::957::TaskManager.Task::(_decref) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::ref 0 aborting True >Thread-577::DEBUG::2012-12-02 23:38:10,422::task::892::TaskManager.Task::(_doAbort) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::Task._doAbort: force False >Thread-577::DEBUG::2012-12-02 23:38:10,422::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-577::DEBUG::2012-12-02 23:38:10,422::task::568::TaskManager.Task::(_updateState) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::moving from state preparing -> state aborting >Thread-577::DEBUG::2012-12-02 23:38:10,422::task::523::TaskManager.Task::(__state_aborting) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::_aborting: recover policy none >Thread-577::DEBUG::2012-12-02 23:38:10,422::task::568::TaskManager.Task::(_updateState) Task=`a9374c52-9b4b-44df-8a99-561d6c422510`::moving from state aborting -> state failed >Thread-577::DEBUG::2012-12-02 23:38:10,422::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-577::DEBUG::2012-12-02 23:38:10,423::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-577::ERROR::2012-12-02 23:38:10,423::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-579::DEBUG::2012-12-02 23:38:10,560::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-579::DEBUG::2012-12-02 23:38:10,561::task::568::TaskManager.Task::(_updateState) Task=`76ba527a-ba99-4f88-8cf7-050616512b0b`::moving from state init -> state preparing >Thread-579::INFO::2012-12-02 23:38:10,561::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-579::DEBUG::2012-12-02 23:38:10,561::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-579::DEBUG::2012-12-02 23:38:10,561::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-579::INFO::2012-12-02 23:38:10,561::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-579::DEBUG::2012-12-02 23:38:10,562::task::1151::TaskManager.Task::(prepare) Task=`76ba527a-ba99-4f88-8cf7-050616512b0b`::finished: {'allTasksStatus': {}} >Thread-579::DEBUG::2012-12-02 23:38:10,562::task::568::TaskManager.Task::(_updateState) Task=`76ba527a-ba99-4f88-8cf7-050616512b0b`::moving from state preparing -> state finished >Thread-579::DEBUG::2012-12-02 23:38:10,562::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-579::DEBUG::2012-12-02 23:38:10,562::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-579::DEBUG::2012-12-02 23:38:10,562::task::957::TaskManager.Task::(_decref) Task=`76ba527a-ba99-4f88-8cf7-050616512b0b`::ref 0 aborting False >Thread-580::DEBUG::2012-12-02 23:38:10,709::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-580::DEBUG::2012-12-02 23:38:10,709::task::568::TaskManager.Task::(_updateState) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::moving from state init -> state preparing >Thread-580::INFO::2012-12-02 23:38:10,710::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-580::ERROR::2012-12-02 23:38:10,710::task::833::TaskManager.Task::(_setError) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-580::DEBUG::2012-12-02 23:38:10,710::task::852::TaskManager.Task::(_run) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::Task._run: dfc67471-935b-47a2-b644-e22e94ba9573 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-580::DEBUG::2012-12-02 23:38:10,710::task::1177::TaskManager.Task::(stop) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::stopping in state preparing (force False) >Thread-580::DEBUG::2012-12-02 23:38:10,710::task::957::TaskManager.Task::(_decref) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::ref 1 aborting True >Thread-580::INFO::2012-12-02 23:38:10,710::task::1134::TaskManager.Task::(prepare) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-580::DEBUG::2012-12-02 23:38:10,710::task::1139::TaskManager.Task::(prepare) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::Prepare: aborted: Unknown pool id, pool not connected >Thread-580::DEBUG::2012-12-02 23:38:10,711::task::957::TaskManager.Task::(_decref) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::ref 0 aborting True >Thread-580::DEBUG::2012-12-02 23:38:10,711::task::892::TaskManager.Task::(_doAbort) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::Task._doAbort: force False >Thread-580::DEBUG::2012-12-02 23:38:10,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-580::DEBUG::2012-12-02 23:38:10,711::task::568::TaskManager.Task::(_updateState) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::moving from state preparing -> state aborting >Thread-580::DEBUG::2012-12-02 23:38:10,711::task::523::TaskManager.Task::(__state_aborting) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::_aborting: recover policy none >Thread-580::DEBUG::2012-12-02 23:38:10,711::task::568::TaskManager.Task::(_updateState) Task=`dfc67471-935b-47a2-b644-e22e94ba9573`::moving from state aborting -> state failed >Thread-580::DEBUG::2012-12-02 23:38:10,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-580::DEBUG::2012-12-02 23:38:10,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-580::ERROR::2012-12-02 23:38:10,712::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-581::DEBUG::2012-12-02 23:38:10,901::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-581::DEBUG::2012-12-02 23:38:10,901::task::568::TaskManager.Task::(_updateState) Task=`103020d1-ad75-400e-907d-5841d66af664`::moving from state init -> state preparing >Thread-581::INFO::2012-12-02 23:38:10,901::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-581::DEBUG::2012-12-02 23:38:10,901::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8bab1146-f206-432d-9366-bb679430c565`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-581::DEBUG::2012-12-02 23:38:10,902::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-581::DEBUG::2012-12-02 23:38:10,902::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-581::DEBUG::2012-12-02 23:38:10,902::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8bab1146-f206-432d-9366-bb679430c565`::Granted request >Thread-581::INFO::2012-12-02 23:38:10,902::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-581::DEBUG::2012-12-02 23:38:10,903::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:10,903::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:10,903::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:10,903::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:10,903::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:10,903::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:10,903::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-581::DEBUG::2012-12-02 23:38:10,904::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-581::DEBUG::2012-12-02 23:38:10,904::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-581::DEBUG::2012-12-02 23:38:10,904::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-581::DEBUG::2012-12-02 23:38:10,904::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-581::DEBUG::2012-12-02 23:38:10,928::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-581::DEBUG::2012-12-02 23:38:10,928::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-581::DEBUG::2012-12-02 23:38:11,561::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-581::DEBUG::2012-12-02 23:38:11,624::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-581::DEBUG::2012-12-02 23:38:11,624::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,625::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,625::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,625::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,625::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,625::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,625::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-581::DEBUG::2012-12-02 23:38:11,626::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,627::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-581::DEBUG::2012-12-02 23:38:11,832::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-581::WARNING::2012-12-02 23:38:11,833::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-581::DEBUG::2012-12-02 23:38:11,834::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-581::DEBUG::2012-12-02 23:38:11,863::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-581::DEBUG::2012-12-02 23:38:11,863::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-581::DEBUG::2012-12-02 23:38:11,863::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-581::DEBUG::2012-12-02 23:38:11,863::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-581::ERROR::2012-12-02 23:38:11,863::task::833::TaskManager.Task::(_setError) Task=`103020d1-ad75-400e-907d-5841d66af664`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-581::DEBUG::2012-12-02 23:38:11,864::task::852::TaskManager.Task::(_run) Task=`103020d1-ad75-400e-907d-5841d66af664`::Task._run: 103020d1-ad75-400e-907d-5841d66af664 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-581::DEBUG::2012-12-02 23:38:11,864::task::1177::TaskManager.Task::(stop) Task=`103020d1-ad75-400e-907d-5841d66af664`::stopping in state preparing (force False) >Thread-581::DEBUG::2012-12-02 23:38:11,864::task::957::TaskManager.Task::(_decref) Task=`103020d1-ad75-400e-907d-5841d66af664`::ref 1 aborting True >Thread-581::INFO::2012-12-02 23:38:11,864::task::1134::TaskManager.Task::(prepare) Task=`103020d1-ad75-400e-907d-5841d66af664`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-581::DEBUG::2012-12-02 23:38:11,864::task::1139::TaskManager.Task::(prepare) Task=`103020d1-ad75-400e-907d-5841d66af664`::Prepare: aborted: Cannot find master domain >Thread-581::DEBUG::2012-12-02 23:38:11,864::task::957::TaskManager.Task::(_decref) Task=`103020d1-ad75-400e-907d-5841d66af664`::ref 0 aborting True >Thread-581::DEBUG::2012-12-02 23:38:11,864::task::892::TaskManager.Task::(_doAbort) Task=`103020d1-ad75-400e-907d-5841d66af664`::Task._doAbort: force False >Thread-581::DEBUG::2012-12-02 23:38:11,865::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-581::DEBUG::2012-12-02 23:38:11,865::task::568::TaskManager.Task::(_updateState) Task=`103020d1-ad75-400e-907d-5841d66af664`::moving from state preparing -> state aborting >Thread-581::DEBUG::2012-12-02 23:38:11,865::task::523::TaskManager.Task::(__state_aborting) Task=`103020d1-ad75-400e-907d-5841d66af664`::_aborting: recover policy none >Thread-581::DEBUG::2012-12-02 23:38:11,865::task::568::TaskManager.Task::(_updateState) Task=`103020d1-ad75-400e-907d-5841d66af664`::moving from state aborting -> state failed >Thread-581::DEBUG::2012-12-02 23:38:11,865::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-581::DEBUG::2012-12-02 23:38:11,865::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-581::ERROR::2012-12-02 23:38:11,865::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-643::DEBUG::2012-12-02 23:38:12,635::task::568::TaskManager.Task::(_updateState) Task=`455f7fa4-2c40-466c-99e9-e1dd6d326eb7`::moving from state init -> state preparing >Thread-643::INFO::2012-12-02 23:38:12,635::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-643::INFO::2012-12-02 23:38:12,636::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-643::DEBUG::2012-12-02 23:38:12,636::task::1151::TaskManager.Task::(prepare) Task=`455f7fa4-2c40-466c-99e9-e1dd6d326eb7`::finished: {} >Thread-643::DEBUG::2012-12-02 23:38:12,636::task::568::TaskManager.Task::(_updateState) Task=`455f7fa4-2c40-466c-99e9-e1dd6d326eb7`::moving from state preparing -> state finished >Thread-643::DEBUG::2012-12-02 23:38:12,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-643::DEBUG::2012-12-02 23:38:12,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-643::DEBUG::2012-12-02 23:38:12,636::task::957::TaskManager.Task::(_decref) Task=`455f7fa4-2c40-466c-99e9-e1dd6d326eb7`::ref 0 aborting False >Thread-649::DEBUG::2012-12-02 23:38:22,175::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-649::DEBUG::2012-12-02 23:38:22,175::task::568::TaskManager.Task::(_updateState) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::moving from state init -> state preparing >Thread-649::INFO::2012-12-02 23:38:22,175::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-649::ERROR::2012-12-02 23:38:22,176::task::833::TaskManager.Task::(_setError) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-649::DEBUG::2012-12-02 23:38:22,176::task::852::TaskManager.Task::(_run) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::Task._run: 4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-649::DEBUG::2012-12-02 23:38:22,176::task::1177::TaskManager.Task::(stop) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::stopping in state preparing (force False) >Thread-649::DEBUG::2012-12-02 23:38:22,176::task::957::TaskManager.Task::(_decref) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::ref 1 aborting True >Thread-649::INFO::2012-12-02 23:38:22,176::task::1134::TaskManager.Task::(prepare) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-649::DEBUG::2012-12-02 23:38:22,176::task::1139::TaskManager.Task::(prepare) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::Prepare: aborted: Unknown pool id, pool not connected >Thread-649::DEBUG::2012-12-02 23:38:22,177::task::957::TaskManager.Task::(_decref) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::ref 0 aborting True >Thread-649::DEBUG::2012-12-02 23:38:22,177::task::892::TaskManager.Task::(_doAbort) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::Task._doAbort: force False >Thread-649::DEBUG::2012-12-02 23:38:22,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-649::DEBUG::2012-12-02 23:38:22,177::task::568::TaskManager.Task::(_updateState) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::moving from state preparing -> state aborting >Thread-649::DEBUG::2012-12-02 23:38:22,177::task::523::TaskManager.Task::(__state_aborting) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::_aborting: recover policy none >Thread-649::DEBUG::2012-12-02 23:38:22,177::task::568::TaskManager.Task::(_updateState) Task=`4d0fd26c-cde3-4b7d-86d7-4a69cea6f87f`::moving from state aborting -> state failed >Thread-649::DEBUG::2012-12-02 23:38:22,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-649::DEBUG::2012-12-02 23:38:22,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-649::ERROR::2012-12-02 23:38:22,177::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-650::DEBUG::2012-12-02 23:38:22,323::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-650::DEBUG::2012-12-02 23:38:22,323::task::568::TaskManager.Task::(_updateState) Task=`c076862c-5735-4fe9-9845-51912e597a82`::moving from state init -> state preparing >Thread-650::INFO::2012-12-02 23:38:22,323::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-650::DEBUG::2012-12-02 23:38:22,324::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-650::DEBUG::2012-12-02 23:38:22,324::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-650::INFO::2012-12-02 23:38:22,324::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-650::DEBUG::2012-12-02 23:38:22,324::task::1151::TaskManager.Task::(prepare) Task=`c076862c-5735-4fe9-9845-51912e597a82`::finished: {'allTasksStatus': {}} >Thread-650::DEBUG::2012-12-02 23:38:22,324::task::568::TaskManager.Task::(_updateState) Task=`c076862c-5735-4fe9-9845-51912e597a82`::moving from state preparing -> state finished >Thread-650::DEBUG::2012-12-02 23:38:22,324::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-650::DEBUG::2012-12-02 23:38:22,324::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-650::DEBUG::2012-12-02 23:38:22,324::task::957::TaskManager.Task::(_decref) Task=`c076862c-5735-4fe9-9845-51912e597a82`::ref 0 aborting False >Thread-651::DEBUG::2012-12-02 23:38:22,470::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-651::DEBUG::2012-12-02 23:38:22,470::task::568::TaskManager.Task::(_updateState) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::moving from state init -> state preparing >Thread-651::INFO::2012-12-02 23:38:22,471::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-651::ERROR::2012-12-02 23:38:22,471::task::833::TaskManager.Task::(_setError) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-651::DEBUG::2012-12-02 23:38:22,471::task::852::TaskManager.Task::(_run) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::Task._run: edd84228-3f0a-4871-9de8-9b3e06adc87e ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-651::DEBUG::2012-12-02 23:38:22,471::task::1177::TaskManager.Task::(stop) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::stopping in state preparing (force False) >Thread-651::DEBUG::2012-12-02 23:38:22,471::task::957::TaskManager.Task::(_decref) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::ref 1 aborting True >Thread-651::INFO::2012-12-02 23:38:22,471::task::1134::TaskManager.Task::(prepare) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-651::DEBUG::2012-12-02 23:38:22,471::task::1139::TaskManager.Task::(prepare) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::Prepare: aborted: Unknown pool id, pool not connected >Thread-651::DEBUG::2012-12-02 23:38:22,472::task::957::TaskManager.Task::(_decref) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::ref 0 aborting True >Thread-651::DEBUG::2012-12-02 23:38:22,472::task::892::TaskManager.Task::(_doAbort) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::Task._doAbort: force False >Thread-651::DEBUG::2012-12-02 23:38:22,472::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-651::DEBUG::2012-12-02 23:38:22,472::task::568::TaskManager.Task::(_updateState) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::moving from state preparing -> state aborting >Thread-651::DEBUG::2012-12-02 23:38:22,472::task::523::TaskManager.Task::(__state_aborting) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::_aborting: recover policy none >Thread-651::DEBUG::2012-12-02 23:38:22,472::task::568::TaskManager.Task::(_updateState) Task=`edd84228-3f0a-4871-9de8-9b3e06adc87e`::moving from state aborting -> state failed >Thread-651::DEBUG::2012-12-02 23:38:22,472::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-651::DEBUG::2012-12-02 23:38:22,472::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-651::ERROR::2012-12-02 23:38:22,473::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-652::DEBUG::2012-12-02 23:38:22,664::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-652::DEBUG::2012-12-02 23:38:22,665::task::568::TaskManager.Task::(_updateState) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::moving from state init -> state preparing >Thread-652::INFO::2012-12-02 23:38:22,665::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-652::DEBUG::2012-12-02 23:38:22,665::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c58dd0fe-2cec-4f9e-8f5c-dd335cde94b4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-652::DEBUG::2012-12-02 23:38:22,665::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-652::DEBUG::2012-12-02 23:38:22,665::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-652::DEBUG::2012-12-02 23:38:22,665::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c58dd0fe-2cec-4f9e-8f5c-dd335cde94b4`::Granted request >Thread-652::INFO::2012-12-02 23:38:22,666::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-652::DEBUG::2012-12-02 23:38:22,666::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:22,666::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:22,666::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:22,666::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:22,666::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:22,666::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:22,667::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-652::DEBUG::2012-12-02 23:38:22,667::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-652::DEBUG::2012-12-02 23:38:22,667::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-652::DEBUG::2012-12-02 23:38:22,667::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-652::DEBUG::2012-12-02 23:38:22,667::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-652::DEBUG::2012-12-02 23:38:22,694::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-652::DEBUG::2012-12-02 23:38:22,694::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-652::DEBUG::2012-12-02 23:38:23,327::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-652::DEBUG::2012-12-02 23:38:23,392::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-652::DEBUG::2012-12-02 23:38:23,392::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,392::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,393::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,393::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,393::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,393::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,393::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-652::DEBUG::2012-12-02 23:38:23,393::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,395::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-653::DEBUG::2012-12-02 23:38:23,552::task::568::TaskManager.Task::(_updateState) Task=`9f9a3d58-6e0b-4222-b3f4-ee0f44c975e2`::moving from state init -> state preparing >Thread-653::INFO::2012-12-02 23:38:23,553::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-653::INFO::2012-12-02 23:38:23,553::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-653::DEBUG::2012-12-02 23:38:23,553::task::1151::TaskManager.Task::(prepare) Task=`9f9a3d58-6e0b-4222-b3f4-ee0f44c975e2`::finished: {} >Thread-653::DEBUG::2012-12-02 23:38:23,553::task::568::TaskManager.Task::(_updateState) Task=`9f9a3d58-6e0b-4222-b3f4-ee0f44c975e2`::moving from state preparing -> state finished >Thread-653::DEBUG::2012-12-02 23:38:23,553::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-653::DEBUG::2012-12-02 23:38:23,553::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-653::DEBUG::2012-12-02 23:38:23,553::task::957::TaskManager.Task::(_decref) Task=`9f9a3d58-6e0b-4222-b3f4-ee0f44c975e2`::ref 0 aborting False >Thread-652::DEBUG::2012-12-02 23:38:23,602::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-652::WARNING::2012-12-02 23:38:23,603::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-652::DEBUG::2012-12-02 23:38:23,603::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-652::DEBUG::2012-12-02 23:38:23,634::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-652::DEBUG::2012-12-02 23:38:23,634::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-652::DEBUG::2012-12-02 23:38:23,634::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-652::DEBUG::2012-12-02 23:38:23,634::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-652::ERROR::2012-12-02 23:38:23,634::task::833::TaskManager.Task::(_setError) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-652::DEBUG::2012-12-02 23:38:23,635::task::852::TaskManager.Task::(_run) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::Task._run: f757f936-70f5-4218-a195-6b3710ac89bd ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-652::DEBUG::2012-12-02 23:38:23,635::task::1177::TaskManager.Task::(stop) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::stopping in state preparing (force False) >Thread-652::DEBUG::2012-12-02 23:38:23,635::task::957::TaskManager.Task::(_decref) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::ref 1 aborting True >Thread-652::INFO::2012-12-02 23:38:23,635::task::1134::TaskManager.Task::(prepare) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-652::DEBUG::2012-12-02 23:38:23,635::task::1139::TaskManager.Task::(prepare) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::Prepare: aborted: Cannot find master domain >Thread-652::DEBUG::2012-12-02 23:38:23,635::task::957::TaskManager.Task::(_decref) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::ref 0 aborting True >Thread-652::DEBUG::2012-12-02 23:38:23,636::task::892::TaskManager.Task::(_doAbort) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::Task._doAbort: force False >Thread-652::DEBUG::2012-12-02 23:38:23,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-652::DEBUG::2012-12-02 23:38:23,636::task::568::TaskManager.Task::(_updateState) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::moving from state preparing -> state aborting >Thread-652::DEBUG::2012-12-02 23:38:23,636::task::523::TaskManager.Task::(__state_aborting) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::_aborting: recover policy none >Thread-652::DEBUG::2012-12-02 23:38:23,636::task::568::TaskManager.Task::(_updateState) Task=`f757f936-70f5-4218-a195-6b3710ac89bd`::moving from state aborting -> state failed >Thread-652::DEBUG::2012-12-02 23:38:23,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-652::DEBUG::2012-12-02 23:38:23,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-652::ERROR::2012-12-02 23:38:23,636::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-720::DEBUG::2012-12-02 23:38:33,943::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-720::DEBUG::2012-12-02 23:38:33,943::task::568::TaskManager.Task::(_updateState) Task=`12250acc-736a-491b-be16-7acf5de3695d`::moving from state init -> state preparing >Thread-720::INFO::2012-12-02 23:38:33,943::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-720::ERROR::2012-12-02 23:38:33,944::task::833::TaskManager.Task::(_setError) Task=`12250acc-736a-491b-be16-7acf5de3695d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-720::DEBUG::2012-12-02 23:38:33,944::task::852::TaskManager.Task::(_run) Task=`12250acc-736a-491b-be16-7acf5de3695d`::Task._run: 12250acc-736a-491b-be16-7acf5de3695d ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-720::DEBUG::2012-12-02 23:38:33,944::task::1177::TaskManager.Task::(stop) Task=`12250acc-736a-491b-be16-7acf5de3695d`::stopping in state preparing (force False) >Thread-720::DEBUG::2012-12-02 23:38:33,944::task::957::TaskManager.Task::(_decref) Task=`12250acc-736a-491b-be16-7acf5de3695d`::ref 1 aborting True >Thread-720::INFO::2012-12-02 23:38:33,944::task::1134::TaskManager.Task::(prepare) Task=`12250acc-736a-491b-be16-7acf5de3695d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-720::DEBUG::2012-12-02 23:38:33,944::task::1139::TaskManager.Task::(prepare) Task=`12250acc-736a-491b-be16-7acf5de3695d`::Prepare: aborted: Unknown pool id, pool not connected >Thread-720::DEBUG::2012-12-02 23:38:33,944::task::957::TaskManager.Task::(_decref) Task=`12250acc-736a-491b-be16-7acf5de3695d`::ref 0 aborting True >Thread-720::DEBUG::2012-12-02 23:38:33,945::task::892::TaskManager.Task::(_doAbort) Task=`12250acc-736a-491b-be16-7acf5de3695d`::Task._doAbort: force False >Thread-720::DEBUG::2012-12-02 23:38:33,945::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-720::DEBUG::2012-12-02 23:38:33,945::task::568::TaskManager.Task::(_updateState) Task=`12250acc-736a-491b-be16-7acf5de3695d`::moving from state preparing -> state aborting >Thread-720::DEBUG::2012-12-02 23:38:33,945::task::523::TaskManager.Task::(__state_aborting) Task=`12250acc-736a-491b-be16-7acf5de3695d`::_aborting: recover policy none >Thread-720::DEBUG::2012-12-02 23:38:33,945::task::568::TaskManager.Task::(_updateState) Task=`12250acc-736a-491b-be16-7acf5de3695d`::moving from state aborting -> state failed >Thread-720::DEBUG::2012-12-02 23:38:33,945::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-720::DEBUG::2012-12-02 23:38:33,945::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-720::ERROR::2012-12-02 23:38:33,945::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-721::DEBUG::2012-12-02 23:38:34,088::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-721::DEBUG::2012-12-02 23:38:34,088::task::568::TaskManager.Task::(_updateState) Task=`f6e76e2d-45bd-436b-b6f6-cb90225f9db6`::moving from state init -> state preparing >Thread-721::INFO::2012-12-02 23:38:34,088::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-721::DEBUG::2012-12-02 23:38:34,088::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-721::DEBUG::2012-12-02 23:38:34,088::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-721::INFO::2012-12-02 23:38:34,089::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-721::DEBUG::2012-12-02 23:38:34,089::task::1151::TaskManager.Task::(prepare) Task=`f6e76e2d-45bd-436b-b6f6-cb90225f9db6`::finished: {'allTasksStatus': {}} >Thread-721::DEBUG::2012-12-02 23:38:34,089::task::568::TaskManager.Task::(_updateState) Task=`f6e76e2d-45bd-436b-b6f6-cb90225f9db6`::moving from state preparing -> state finished >Thread-721::DEBUG::2012-12-02 23:38:34,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-721::DEBUG::2012-12-02 23:38:34,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-721::DEBUG::2012-12-02 23:38:34,089::task::957::TaskManager.Task::(_decref) Task=`f6e76e2d-45bd-436b-b6f6-cb90225f9db6`::ref 0 aborting False >Thread-722::DEBUG::2012-12-02 23:38:34,234::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-722::DEBUG::2012-12-02 23:38:34,234::task::568::TaskManager.Task::(_updateState) Task=`550596ea-228b-4595-8149-f32529a40b83`::moving from state init -> state preparing >Thread-722::INFO::2012-12-02 23:38:34,234::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-722::ERROR::2012-12-02 23:38:34,234::task::833::TaskManager.Task::(_setError) Task=`550596ea-228b-4595-8149-f32529a40b83`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-722::DEBUG::2012-12-02 23:38:34,235::task::852::TaskManager.Task::(_run) Task=`550596ea-228b-4595-8149-f32529a40b83`::Task._run: 550596ea-228b-4595-8149-f32529a40b83 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-722::DEBUG::2012-12-02 23:38:34,235::task::1177::TaskManager.Task::(stop) Task=`550596ea-228b-4595-8149-f32529a40b83`::stopping in state preparing (force False) >Thread-722::DEBUG::2012-12-02 23:38:34,235::task::957::TaskManager.Task::(_decref) Task=`550596ea-228b-4595-8149-f32529a40b83`::ref 1 aborting True >Thread-722::INFO::2012-12-02 23:38:34,235::task::1134::TaskManager.Task::(prepare) Task=`550596ea-228b-4595-8149-f32529a40b83`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-722::DEBUG::2012-12-02 23:38:34,235::task::1139::TaskManager.Task::(prepare) Task=`550596ea-228b-4595-8149-f32529a40b83`::Prepare: aborted: Unknown pool id, pool not connected >Thread-722::DEBUG::2012-12-02 23:38:34,235::task::957::TaskManager.Task::(_decref) Task=`550596ea-228b-4595-8149-f32529a40b83`::ref 0 aborting True >Thread-722::DEBUG::2012-12-02 23:38:34,235::task::892::TaskManager.Task::(_doAbort) Task=`550596ea-228b-4595-8149-f32529a40b83`::Task._doAbort: force False >Thread-722::DEBUG::2012-12-02 23:38:34,235::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-722::DEBUG::2012-12-02 23:38:34,236::task::568::TaskManager.Task::(_updateState) Task=`550596ea-228b-4595-8149-f32529a40b83`::moving from state preparing -> state aborting >Thread-722::DEBUG::2012-12-02 23:38:34,236::task::523::TaskManager.Task::(__state_aborting) Task=`550596ea-228b-4595-8149-f32529a40b83`::_aborting: recover policy none >Thread-722::DEBUG::2012-12-02 23:38:34,236::task::568::TaskManager.Task::(_updateState) Task=`550596ea-228b-4595-8149-f32529a40b83`::moving from state aborting -> state failed >Thread-722::DEBUG::2012-12-02 23:38:34,236::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-722::DEBUG::2012-12-02 23:38:34,236::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-722::ERROR::2012-12-02 23:38:34,236::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-723::DEBUG::2012-12-02 23:38:34,440::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-723::DEBUG::2012-12-02 23:38:34,440::task::568::TaskManager.Task::(_updateState) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::moving from state init -> state preparing >Thread-723::INFO::2012-12-02 23:38:34,440::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-723::DEBUG::2012-12-02 23:38:34,441::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9899187c-91b7-4f0f-a423-9b0c70ae421c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-723::DEBUG::2012-12-02 23:38:34,441::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-723::DEBUG::2012-12-02 23:38:34,441::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-723::DEBUG::2012-12-02 23:38:34,441::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9899187c-91b7-4f0f-a423-9b0c70ae421c`::Granted request >Thread-723::INFO::2012-12-02 23:38:34,442::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-723::DEBUG::2012-12-02 23:38:34,442::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:34,442::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:34,442::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:34,442::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:34,443::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:34,443::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:34,443::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-723::DEBUG::2012-12-02 23:38:34,443::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-723::DEBUG::2012-12-02 23:38:34,443::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-723::DEBUG::2012-12-02 23:38:34,443::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-723::DEBUG::2012-12-02 23:38:34,443::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-724::DEBUG::2012-12-02 23:38:34,460::task::568::TaskManager.Task::(_updateState) Task=`426660e7-b3f4-4dd6-bbea-d17829baff92`::moving from state init -> state preparing >Thread-724::INFO::2012-12-02 23:38:34,461::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-724::INFO::2012-12-02 23:38:34,461::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-724::DEBUG::2012-12-02 23:38:34,461::task::1151::TaskManager.Task::(prepare) Task=`426660e7-b3f4-4dd6-bbea-d17829baff92`::finished: {} >Thread-724::DEBUG::2012-12-02 23:38:34,461::task::568::TaskManager.Task::(_updateState) Task=`426660e7-b3f4-4dd6-bbea-d17829baff92`::moving from state preparing -> state finished >Thread-724::DEBUG::2012-12-02 23:38:34,461::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-724::DEBUG::2012-12-02 23:38:34,461::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-724::DEBUG::2012-12-02 23:38:34,462::task::957::TaskManager.Task::(_decref) Task=`426660e7-b3f4-4dd6-bbea-d17829baff92`::ref 0 aborting False >Thread-723::DEBUG::2012-12-02 23:38:34,468::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-723::DEBUG::2012-12-02 23:38:34,468::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-723::DEBUG::2012-12-02 23:38:35,101::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-723::DEBUG::2012-12-02 23:38:35,148::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-723::DEBUG::2012-12-02 23:38:35,148::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,149::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,149::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,149::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,149::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,149::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,150::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-723::DEBUG::2012-12-02 23:38:35,150::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,151::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-723::DEBUG::2012-12-02 23:38:35,354::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-723::WARNING::2012-12-02 23:38:35,355::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-723::DEBUG::2012-12-02 23:38:35,355::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-723::DEBUG::2012-12-02 23:38:35,385::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-723::DEBUG::2012-12-02 23:38:35,385::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-723::DEBUG::2012-12-02 23:38:35,385::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-723::DEBUG::2012-12-02 23:38:35,385::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-723::ERROR::2012-12-02 23:38:35,385::task::833::TaskManager.Task::(_setError) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-723::DEBUG::2012-12-02 23:38:35,386::task::852::TaskManager.Task::(_run) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::Task._run: f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-723::DEBUG::2012-12-02 23:38:35,386::task::1177::TaskManager.Task::(stop) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::stopping in state preparing (force False) >Thread-723::DEBUG::2012-12-02 23:38:35,386::task::957::TaskManager.Task::(_decref) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::ref 1 aborting True >Thread-723::INFO::2012-12-02 23:38:35,386::task::1134::TaskManager.Task::(prepare) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-723::DEBUG::2012-12-02 23:38:35,386::task::1139::TaskManager.Task::(prepare) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::Prepare: aborted: Cannot find master domain >Thread-723::DEBUG::2012-12-02 23:38:35,387::task::957::TaskManager.Task::(_decref) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::ref 0 aborting True >Thread-723::DEBUG::2012-12-02 23:38:35,387::task::892::TaskManager.Task::(_doAbort) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::Task._doAbort: force False >Thread-723::DEBUG::2012-12-02 23:38:35,387::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-723::DEBUG::2012-12-02 23:38:35,387::task::568::TaskManager.Task::(_updateState) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::moving from state preparing -> state aborting >Thread-723::DEBUG::2012-12-02 23:38:35,387::task::523::TaskManager.Task::(__state_aborting) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::_aborting: recover policy none >Thread-723::DEBUG::2012-12-02 23:38:35,387::task::568::TaskManager.Task::(_updateState) Task=`f5176cd8-7c6c-4e46-a1c7-60a2f002cb2f`::moving from state aborting -> state failed >Thread-723::DEBUG::2012-12-02 23:38:35,387::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-723::DEBUG::2012-12-02 23:38:35,387::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-723::ERROR::2012-12-02 23:38:35,388::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-791::DEBUG::2012-12-02 23:38:45,373::task::568::TaskManager.Task::(_updateState) Task=`cb85fd21-6e30-408a-a06d-34622aa68974`::moving from state init -> state preparing >Thread-791::INFO::2012-12-02 23:38:45,373::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-791::INFO::2012-12-02 23:38:45,373::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-791::DEBUG::2012-12-02 23:38:45,373::task::1151::TaskManager.Task::(prepare) Task=`cb85fd21-6e30-408a-a06d-34622aa68974`::finished: {} >Thread-791::DEBUG::2012-12-02 23:38:45,373::task::568::TaskManager.Task::(_updateState) Task=`cb85fd21-6e30-408a-a06d-34622aa68974`::moving from state preparing -> state finished >Thread-791::DEBUG::2012-12-02 23:38:45,373::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-791::DEBUG::2012-12-02 23:38:45,374::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-791::DEBUG::2012-12-02 23:38:45,374::task::957::TaskManager.Task::(_decref) Task=`cb85fd21-6e30-408a-a06d-34622aa68974`::ref 0 aborting False >Thread-793::DEBUG::2012-12-02 23:38:45,713::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-793::DEBUG::2012-12-02 23:38:45,714::task::568::TaskManager.Task::(_updateState) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::moving from state init -> state preparing >Thread-793::INFO::2012-12-02 23:38:45,714::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-793::ERROR::2012-12-02 23:38:45,714::task::833::TaskManager.Task::(_setError) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-793::DEBUG::2012-12-02 23:38:45,714::task::852::TaskManager.Task::(_run) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::Task._run: 0e4e6eb2-f335-4f1d-aabb-fe06c51f961c ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-793::DEBUG::2012-12-02 23:38:45,714::task::1177::TaskManager.Task::(stop) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::stopping in state preparing (force False) >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::957::TaskManager.Task::(_decref) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::ref 1 aborting True >Thread-793::INFO::2012-12-02 23:38:45,715::task::1134::TaskManager.Task::(prepare) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::1139::TaskManager.Task::(prepare) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::Prepare: aborted: Unknown pool id, pool not connected >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::957::TaskManager.Task::(_decref) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::ref 0 aborting True >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::892::TaskManager.Task::(_doAbort) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::Task._doAbort: force False >Thread-793::DEBUG::2012-12-02 23:38:45,715::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::568::TaskManager.Task::(_updateState) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::moving from state preparing -> state aborting >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::523::TaskManager.Task::(__state_aborting) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::_aborting: recover policy none >Thread-793::DEBUG::2012-12-02 23:38:45,715::task::568::TaskManager.Task::(_updateState) Task=`0e4e6eb2-f335-4f1d-aabb-fe06c51f961c`::moving from state aborting -> state failed >Thread-793::DEBUG::2012-12-02 23:38:45,716::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-793::DEBUG::2012-12-02 23:38:45,716::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-793::ERROR::2012-12-02 23:38:45,716::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-794::DEBUG::2012-12-02 23:38:45,862::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-794::DEBUG::2012-12-02 23:38:45,863::task::568::TaskManager.Task::(_updateState) Task=`984301a7-702d-4d61-b6fc-7c5a00a8c5b6`::moving from state init -> state preparing >Thread-794::INFO::2012-12-02 23:38:45,863::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-794::DEBUG::2012-12-02 23:38:45,863::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-794::DEBUG::2012-12-02 23:38:45,863::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-794::INFO::2012-12-02 23:38:45,863::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-794::DEBUG::2012-12-02 23:38:45,863::task::1151::TaskManager.Task::(prepare) Task=`984301a7-702d-4d61-b6fc-7c5a00a8c5b6`::finished: {'allTasksStatus': {}} >Thread-794::DEBUG::2012-12-02 23:38:45,863::task::568::TaskManager.Task::(_updateState) Task=`984301a7-702d-4d61-b6fc-7c5a00a8c5b6`::moving from state preparing -> state finished >Thread-794::DEBUG::2012-12-02 23:38:45,864::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-794::DEBUG::2012-12-02 23:38:45,864::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-794::DEBUG::2012-12-02 23:38:45,864::task::957::TaskManager.Task::(_decref) Task=`984301a7-702d-4d61-b6fc-7c5a00a8c5b6`::ref 0 aborting False >Thread-795::DEBUG::2012-12-02 23:38:46,023::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-795::DEBUG::2012-12-02 23:38:46,024::task::568::TaskManager.Task::(_updateState) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::moving from state init -> state preparing >Thread-795::INFO::2012-12-02 23:38:46,024::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-795::ERROR::2012-12-02 23:38:46,024::task::833::TaskManager.Task::(_setError) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-795::DEBUG::2012-12-02 23:38:46,024::task::852::TaskManager.Task::(_run) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::Task._run: 76c1800f-23fa-4c1a-af08-9a7302760496 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-795::DEBUG::2012-12-02 23:38:46,024::task::1177::TaskManager.Task::(stop) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::stopping in state preparing (force False) >Thread-795::DEBUG::2012-12-02 23:38:46,024::task::957::TaskManager.Task::(_decref) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::ref 1 aborting True >Thread-795::INFO::2012-12-02 23:38:46,025::task::1134::TaskManager.Task::(prepare) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-795::DEBUG::2012-12-02 23:38:46,025::task::1139::TaskManager.Task::(prepare) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::Prepare: aborted: Unknown pool id, pool not connected >Thread-795::DEBUG::2012-12-02 23:38:46,025::task::957::TaskManager.Task::(_decref) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::ref 0 aborting True >Thread-795::DEBUG::2012-12-02 23:38:46,025::task::892::TaskManager.Task::(_doAbort) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::Task._doAbort: force False >Thread-795::DEBUG::2012-12-02 23:38:46,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-795::DEBUG::2012-12-02 23:38:46,025::task::568::TaskManager.Task::(_updateState) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::moving from state preparing -> state aborting >Thread-795::DEBUG::2012-12-02 23:38:46,025::task::523::TaskManager.Task::(__state_aborting) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::_aborting: recover policy none >Thread-795::DEBUG::2012-12-02 23:38:46,025::task::568::TaskManager.Task::(_updateState) Task=`76c1800f-23fa-4c1a-af08-9a7302760496`::moving from state aborting -> state failed >Thread-795::DEBUG::2012-12-02 23:38:46,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-795::DEBUG::2012-12-02 23:38:46,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-795::ERROR::2012-12-02 23:38:46,025::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-796::DEBUG::2012-12-02 23:38:46,208::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-796::DEBUG::2012-12-02 23:38:46,208::task::568::TaskManager.Task::(_updateState) Task=`8223f691-7504-4733-bc1d-201aa1224348`::moving from state init -> state preparing >Thread-796::INFO::2012-12-02 23:38:46,209::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-796::DEBUG::2012-12-02 23:38:46,209::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7e661b39-08ea-4da1-8cfd-2979918be035`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-796::DEBUG::2012-12-02 23:38:46,209::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-796::DEBUG::2012-12-02 23:38:46,209::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-796::DEBUG::2012-12-02 23:38:46,209::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7e661b39-08ea-4da1-8cfd-2979918be035`::Granted request >Thread-796::INFO::2012-12-02 23:38:46,210::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-796::DEBUG::2012-12-02 23:38:46,210::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,210::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,210::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,211::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,211::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,211::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,211::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-796::DEBUG::2012-12-02 23:38:46,211::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-796::DEBUG::2012-12-02 23:38:46,211::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-796::DEBUG::2012-12-02 23:38:46,211::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-796::DEBUG::2012-12-02 23:38:46,212::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-796::DEBUG::2012-12-02 23:38:46,235::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-796::DEBUG::2012-12-02 23:38:46,235::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-796::DEBUG::2012-12-02 23:38:46,870::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-796::DEBUG::2012-12-02 23:38:46,940::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-796::DEBUG::2012-12-02 23:38:46,941::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,941::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,941::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,941::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,941::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,942::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,942::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-796::DEBUG::2012-12-02 23:38:46,942::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:46,943::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-796::DEBUG::2012-12-02 23:38:47,150::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-796::WARNING::2012-12-02 23:38:47,151::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-796::DEBUG::2012-12-02 23:38:47,152::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-796::DEBUG::2012-12-02 23:38:47,185::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-796::DEBUG::2012-12-02 23:38:47,185::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-796::DEBUG::2012-12-02 23:38:47,185::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-796::DEBUG::2012-12-02 23:38:47,185::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-796::ERROR::2012-12-02 23:38:47,186::task::833::TaskManager.Task::(_setError) Task=`8223f691-7504-4733-bc1d-201aa1224348`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-796::DEBUG::2012-12-02 23:38:47,186::task::852::TaskManager.Task::(_run) Task=`8223f691-7504-4733-bc1d-201aa1224348`::Task._run: 8223f691-7504-4733-bc1d-201aa1224348 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-796::DEBUG::2012-12-02 23:38:47,186::task::1177::TaskManager.Task::(stop) Task=`8223f691-7504-4733-bc1d-201aa1224348`::stopping in state preparing (force False) >Thread-796::DEBUG::2012-12-02 23:38:47,186::task::957::TaskManager.Task::(_decref) Task=`8223f691-7504-4733-bc1d-201aa1224348`::ref 1 aborting True >Thread-796::INFO::2012-12-02 23:38:47,187::task::1134::TaskManager.Task::(prepare) Task=`8223f691-7504-4733-bc1d-201aa1224348`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-796::DEBUG::2012-12-02 23:38:47,187::task::1139::TaskManager.Task::(prepare) Task=`8223f691-7504-4733-bc1d-201aa1224348`::Prepare: aborted: Cannot find master domain >Thread-796::DEBUG::2012-12-02 23:38:47,187::task::957::TaskManager.Task::(_decref) Task=`8223f691-7504-4733-bc1d-201aa1224348`::ref 0 aborting True >Thread-796::DEBUG::2012-12-02 23:38:47,187::task::892::TaskManager.Task::(_doAbort) Task=`8223f691-7504-4733-bc1d-201aa1224348`::Task._doAbort: force False >Thread-796::DEBUG::2012-12-02 23:38:47,187::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-796::DEBUG::2012-12-02 23:38:47,187::task::568::TaskManager.Task::(_updateState) Task=`8223f691-7504-4733-bc1d-201aa1224348`::moving from state preparing -> state aborting >Thread-796::DEBUG::2012-12-02 23:38:47,187::task::523::TaskManager.Task::(__state_aborting) Task=`8223f691-7504-4733-bc1d-201aa1224348`::_aborting: recover policy none >Thread-796::DEBUG::2012-12-02 23:38:47,187::task::568::TaskManager.Task::(_updateState) Task=`8223f691-7504-4733-bc1d-201aa1224348`::moving from state aborting -> state failed >Thread-796::DEBUG::2012-12-02 23:38:47,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-796::DEBUG::2012-12-02 23:38:47,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-796::ERROR::2012-12-02 23:38:47,188::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-862::DEBUG::2012-12-02 23:38:56,293::task::568::TaskManager.Task::(_updateState) Task=`51f013b6-bd44-46ee-886a-5d69f0250b98`::moving from state init -> state preparing >Thread-862::INFO::2012-12-02 23:38:56,293::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-862::INFO::2012-12-02 23:38:56,293::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-862::DEBUG::2012-12-02 23:38:56,294::task::1151::TaskManager.Task::(prepare) Task=`51f013b6-bd44-46ee-886a-5d69f0250b98`::finished: {} >Thread-862::DEBUG::2012-12-02 23:38:56,294::task::568::TaskManager.Task::(_updateState) Task=`51f013b6-bd44-46ee-886a-5d69f0250b98`::moving from state preparing -> state finished >Thread-862::DEBUG::2012-12-02 23:38:56,294::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-862::DEBUG::2012-12-02 23:38:56,294::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-862::DEBUG::2012-12-02 23:38:56,294::task::957::TaskManager.Task::(_decref) Task=`51f013b6-bd44-46ee-886a-5d69f0250b98`::ref 0 aborting False >Thread-864::DEBUG::2012-12-02 23:38:57,549::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-864::DEBUG::2012-12-02 23:38:57,550::task::568::TaskManager.Task::(_updateState) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::moving from state init -> state preparing >Thread-864::INFO::2012-12-02 23:38:57,550::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-864::ERROR::2012-12-02 23:38:57,550::task::833::TaskManager.Task::(_setError) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-864::DEBUG::2012-12-02 23:38:57,550::task::852::TaskManager.Task::(_run) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::Task._run: 8326e573-ead7-4001-af8a-326cebc1a1a0 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-864::DEBUG::2012-12-02 23:38:57,551::task::1177::TaskManager.Task::(stop) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::stopping in state preparing (force False) >Thread-864::DEBUG::2012-12-02 23:38:57,551::task::957::TaskManager.Task::(_decref) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::ref 1 aborting True >Thread-864::INFO::2012-12-02 23:38:57,551::task::1134::TaskManager.Task::(prepare) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-864::DEBUG::2012-12-02 23:38:57,551::task::1139::TaskManager.Task::(prepare) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::Prepare: aborted: Unknown pool id, pool not connected >Thread-864::DEBUG::2012-12-02 23:38:57,551::task::957::TaskManager.Task::(_decref) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::ref 0 aborting True >Thread-864::DEBUG::2012-12-02 23:38:57,551::task::892::TaskManager.Task::(_doAbort) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::Task._doAbort: force False >Thread-864::DEBUG::2012-12-02 23:38:57,551::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-864::DEBUG::2012-12-02 23:38:57,551::task::568::TaskManager.Task::(_updateState) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::moving from state preparing -> state aborting >Thread-864::DEBUG::2012-12-02 23:38:57,552::task::523::TaskManager.Task::(__state_aborting) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::_aborting: recover policy none >Thread-864::DEBUG::2012-12-02 23:38:57,552::task::568::TaskManager.Task::(_updateState) Task=`8326e573-ead7-4001-af8a-326cebc1a1a0`::moving from state aborting -> state failed >Thread-864::DEBUG::2012-12-02 23:38:57,552::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-864::DEBUG::2012-12-02 23:38:57,552::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-864::ERROR::2012-12-02 23:38:57,552::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-865::DEBUG::2012-12-02 23:38:57,698::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-865::DEBUG::2012-12-02 23:38:57,698::task::568::TaskManager.Task::(_updateState) Task=`b87812ad-ca48-455a-896e-3cbe73e4c6e8`::moving from state init -> state preparing >Thread-865::INFO::2012-12-02 23:38:57,699::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-865::DEBUG::2012-12-02 23:38:57,699::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-865::DEBUG::2012-12-02 23:38:57,699::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-865::INFO::2012-12-02 23:38:57,699::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-865::DEBUG::2012-12-02 23:38:57,699::task::1151::TaskManager.Task::(prepare) Task=`b87812ad-ca48-455a-896e-3cbe73e4c6e8`::finished: {'allTasksStatus': {}} >Thread-865::DEBUG::2012-12-02 23:38:57,699::task::568::TaskManager.Task::(_updateState) Task=`b87812ad-ca48-455a-896e-3cbe73e4c6e8`::moving from state preparing -> state finished >Thread-865::DEBUG::2012-12-02 23:38:57,699::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-865::DEBUG::2012-12-02 23:38:57,700::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-865::DEBUG::2012-12-02 23:38:57,700::task::957::TaskManager.Task::(_decref) Task=`b87812ad-ca48-455a-896e-3cbe73e4c6e8`::ref 0 aborting False >Thread-866::DEBUG::2012-12-02 23:38:57,846::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-866::DEBUG::2012-12-02 23:38:57,846::task::568::TaskManager.Task::(_updateState) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::moving from state init -> state preparing >Thread-866::INFO::2012-12-02 23:38:57,846::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-866::ERROR::2012-12-02 23:38:57,846::task::833::TaskManager.Task::(_setError) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-866::DEBUG::2012-12-02 23:38:57,847::task::852::TaskManager.Task::(_run) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::Task._run: d96a0c53-43c1-47c3-b9b3-22fa4ef9d047 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-866::DEBUG::2012-12-02 23:38:57,847::task::1177::TaskManager.Task::(stop) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::stopping in state preparing (force False) >Thread-866::DEBUG::2012-12-02 23:38:57,847::task::957::TaskManager.Task::(_decref) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::ref 1 aborting True >Thread-866::INFO::2012-12-02 23:38:57,847::task::1134::TaskManager.Task::(prepare) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-866::DEBUG::2012-12-02 23:38:57,847::task::1139::TaskManager.Task::(prepare) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::Prepare: aborted: Unknown pool id, pool not connected >Thread-866::DEBUG::2012-12-02 23:38:57,847::task::957::TaskManager.Task::(_decref) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::ref 0 aborting True >Thread-866::DEBUG::2012-12-02 23:38:57,848::task::892::TaskManager.Task::(_doAbort) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::Task._doAbort: force False >Thread-866::DEBUG::2012-12-02 23:38:57,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-866::DEBUG::2012-12-02 23:38:57,848::task::568::TaskManager.Task::(_updateState) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::moving from state preparing -> state aborting >Thread-866::DEBUG::2012-12-02 23:38:57,848::task::523::TaskManager.Task::(__state_aborting) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::_aborting: recover policy none >Thread-866::DEBUG::2012-12-02 23:38:57,848::task::568::TaskManager.Task::(_updateState) Task=`d96a0c53-43c1-47c3-b9b3-22fa4ef9d047`::moving from state aborting -> state failed >Thread-866::DEBUG::2012-12-02 23:38:57,848::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-866::DEBUG::2012-12-02 23:38:57,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-866::ERROR::2012-12-02 23:38:57,848::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-867::DEBUG::2012-12-02 23:38:58,028::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-867::DEBUG::2012-12-02 23:38:58,028::task::568::TaskManager.Task::(_updateState) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::moving from state init -> state preparing >Thread-867::INFO::2012-12-02 23:38:58,028::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-867::DEBUG::2012-12-02 23:38:58,029::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`78fb1e18-2697-4359-844d-bc17d0aa2c5f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-867::DEBUG::2012-12-02 23:38:58,029::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-867::DEBUG::2012-12-02 23:38:58,029::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-867::DEBUG::2012-12-02 23:38:58,029::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`78fb1e18-2697-4359-844d-bc17d0aa2c5f`::Granted request >Thread-867::INFO::2012-12-02 23:38:58,030::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-867::DEBUG::2012-12-02 23:38:58,030::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,030::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,030::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,030::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,031::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,031::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,031::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-867::DEBUG::2012-12-02 23:38:58,031::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-867::DEBUG::2012-12-02 23:38:58,031::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-867::DEBUG::2012-12-02 23:38:58,031::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-867::DEBUG::2012-12-02 23:38:58,031::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-867::DEBUG::2012-12-02 23:38:58,055::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-867::DEBUG::2012-12-02 23:38:58,055::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-867::DEBUG::2012-12-02 23:38:58,687::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-867::DEBUG::2012-12-02 23:38:58,756::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-867::DEBUG::2012-12-02 23:38:58,757::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,757::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,757::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,757::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,757::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,758::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,758::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-867::DEBUG::2012-12-02 23:38:58,758::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:58,759::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-867::DEBUG::2012-12-02 23:38:58,964::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-867::WARNING::2012-12-02 23:38:58,965::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-867::DEBUG::2012-12-02 23:38:58,966::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-867::DEBUG::2012-12-02 23:38:59,014::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-867::DEBUG::2012-12-02 23:38:59,014::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-867::DEBUG::2012-12-02 23:38:59,015::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-867::DEBUG::2012-12-02 23:38:59,015::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-867::ERROR::2012-12-02 23:38:59,015::task::833::TaskManager.Task::(_setError) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-867::DEBUG::2012-12-02 23:38:59,015::task::852::TaskManager.Task::(_run) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::Task._run: dacab569-2b13-492a-a9bf-9ed190c397e2 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-867::DEBUG::2012-12-02 23:38:59,015::task::1177::TaskManager.Task::(stop) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::stopping in state preparing (force False) >Thread-867::DEBUG::2012-12-02 23:38:59,015::task::957::TaskManager.Task::(_decref) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::ref 1 aborting True >Thread-867::INFO::2012-12-02 23:38:59,016::task::1134::TaskManager.Task::(prepare) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-867::DEBUG::2012-12-02 23:38:59,016::task::1139::TaskManager.Task::(prepare) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::Prepare: aborted: Cannot find master domain >Thread-867::DEBUG::2012-12-02 23:38:59,016::task::957::TaskManager.Task::(_decref) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::ref 0 aborting True >Thread-867::DEBUG::2012-12-02 23:38:59,016::task::892::TaskManager.Task::(_doAbort) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::Task._doAbort: force False >Thread-867::DEBUG::2012-12-02 23:38:59,016::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-867::DEBUG::2012-12-02 23:38:59,016::task::568::TaskManager.Task::(_updateState) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::moving from state preparing -> state aborting >Thread-867::DEBUG::2012-12-02 23:38:59,016::task::523::TaskManager.Task::(__state_aborting) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::_aborting: recover policy none >Thread-867::DEBUG::2012-12-02 23:38:59,016::task::568::TaskManager.Task::(_updateState) Task=`dacab569-2b13-492a-a9bf-9ed190c397e2`::moving from state aborting -> state failed >Thread-867::DEBUG::2012-12-02 23:38:59,016::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-867::DEBUG::2012-12-02 23:38:59,016::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-867::ERROR::2012-12-02 23:38:59,016::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-933::DEBUG::2012-12-02 23:39:07,252::task::568::TaskManager.Task::(_updateState) Task=`149233ca-2e46-420b-8fdf-12bb9b578bba`::moving from state init -> state preparing >Thread-933::INFO::2012-12-02 23:39:07,252::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-933::INFO::2012-12-02 23:39:07,252::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-933::DEBUG::2012-12-02 23:39:07,252::task::1151::TaskManager.Task::(prepare) Task=`149233ca-2e46-420b-8fdf-12bb9b578bba`::finished: {} >Thread-933::DEBUG::2012-12-02 23:39:07,252::task::568::TaskManager.Task::(_updateState) Task=`149233ca-2e46-420b-8fdf-12bb9b578bba`::moving from state preparing -> state finished >Thread-933::DEBUG::2012-12-02 23:39:07,252::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-933::DEBUG::2012-12-02 23:39:07,252::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-933::DEBUG::2012-12-02 23:39:07,253::task::957::TaskManager.Task::(_decref) Task=`149233ca-2e46-420b-8fdf-12bb9b578bba`::ref 0 aborting False >Thread-935::DEBUG::2012-12-02 23:39:09,360::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-935::DEBUG::2012-12-02 23:39:09,360::task::568::TaskManager.Task::(_updateState) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::moving from state init -> state preparing >Thread-935::INFO::2012-12-02 23:39:09,360::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-935::ERROR::2012-12-02 23:39:09,360::task::833::TaskManager.Task::(_setError) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-935::DEBUG::2012-12-02 23:39:09,361::task::852::TaskManager.Task::(_run) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::Task._run: 8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-935::DEBUG::2012-12-02 23:39:09,361::task::1177::TaskManager.Task::(stop) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::stopping in state preparing (force False) >Thread-935::DEBUG::2012-12-02 23:39:09,361::task::957::TaskManager.Task::(_decref) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::ref 1 aborting True >Thread-935::INFO::2012-12-02 23:39:09,361::task::1134::TaskManager.Task::(prepare) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-935::DEBUG::2012-12-02 23:39:09,361::task::1139::TaskManager.Task::(prepare) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::Prepare: aborted: Unknown pool id, pool not connected >Thread-935::DEBUG::2012-12-02 23:39:09,361::task::957::TaskManager.Task::(_decref) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::ref 0 aborting True >Thread-935::DEBUG::2012-12-02 23:39:09,361::task::892::TaskManager.Task::(_doAbort) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::Task._doAbort: force False >Thread-935::DEBUG::2012-12-02 23:39:09,362::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-935::DEBUG::2012-12-02 23:39:09,362::task::568::TaskManager.Task::(_updateState) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::moving from state preparing -> state aborting >Thread-935::DEBUG::2012-12-02 23:39:09,362::task::523::TaskManager.Task::(__state_aborting) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::_aborting: recover policy none >Thread-935::DEBUG::2012-12-02 23:39:09,362::task::568::TaskManager.Task::(_updateState) Task=`8d7882dc-ab62-49fd-a7ac-d42bbe6f3de2`::moving from state aborting -> state failed >Thread-935::DEBUG::2012-12-02 23:39:09,362::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-935::DEBUG::2012-12-02 23:39:09,362::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-935::ERROR::2012-12-02 23:39:09,362::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-936::DEBUG::2012-12-02 23:39:09,508::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-936::DEBUG::2012-12-02 23:39:09,508::task::568::TaskManager.Task::(_updateState) Task=`4c46198e-ab86-48cc-9ee6-80c59b775e9e`::moving from state init -> state preparing >Thread-936::INFO::2012-12-02 23:39:09,508::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-936::DEBUG::2012-12-02 23:39:09,508::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-936::DEBUG::2012-12-02 23:39:09,508::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-936::INFO::2012-12-02 23:39:09,509::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-936::DEBUG::2012-12-02 23:39:09,509::task::1151::TaskManager.Task::(prepare) Task=`4c46198e-ab86-48cc-9ee6-80c59b775e9e`::finished: {'allTasksStatus': {}} >Thread-936::DEBUG::2012-12-02 23:39:09,509::task::568::TaskManager.Task::(_updateState) Task=`4c46198e-ab86-48cc-9ee6-80c59b775e9e`::moving from state preparing -> state finished >Thread-936::DEBUG::2012-12-02 23:39:09,509::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-936::DEBUG::2012-12-02 23:39:09,509::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-936::DEBUG::2012-12-02 23:39:09,509::task::957::TaskManager.Task::(_decref) Task=`4c46198e-ab86-48cc-9ee6-80c59b775e9e`::ref 0 aborting False >Thread-938::DEBUG::2012-12-02 23:39:09,661::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-938::DEBUG::2012-12-02 23:39:09,661::task::568::TaskManager.Task::(_updateState) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::moving from state init -> state preparing >Thread-938::INFO::2012-12-02 23:39:09,661::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-938::ERROR::2012-12-02 23:39:09,662::task::833::TaskManager.Task::(_setError) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-938::DEBUG::2012-12-02 23:39:09,662::task::852::TaskManager.Task::(_run) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::Task._run: 71c51f54-402f-44dc-8d29-3b2634b49500 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-938::DEBUG::2012-12-02 23:39:09,662::task::1177::TaskManager.Task::(stop) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::stopping in state preparing (force False) >Thread-938::DEBUG::2012-12-02 23:39:09,662::task::957::TaskManager.Task::(_decref) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::ref 1 aborting True >Thread-938::INFO::2012-12-02 23:39:09,662::task::1134::TaskManager.Task::(prepare) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-938::DEBUG::2012-12-02 23:39:09,662::task::1139::TaskManager.Task::(prepare) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::Prepare: aborted: Unknown pool id, pool not connected >Thread-938::DEBUG::2012-12-02 23:39:09,663::task::957::TaskManager.Task::(_decref) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::ref 0 aborting True >Thread-938::DEBUG::2012-12-02 23:39:09,663::task::892::TaskManager.Task::(_doAbort) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::Task._doAbort: force False >Thread-938::DEBUG::2012-12-02 23:39:09,663::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-938::DEBUG::2012-12-02 23:39:09,663::task::568::TaskManager.Task::(_updateState) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::moving from state preparing -> state aborting >Thread-938::DEBUG::2012-12-02 23:39:09,663::task::523::TaskManager.Task::(__state_aborting) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::_aborting: recover policy none >Thread-938::DEBUG::2012-12-02 23:39:09,663::task::568::TaskManager.Task::(_updateState) Task=`71c51f54-402f-44dc-8d29-3b2634b49500`::moving from state aborting -> state failed >Thread-938::DEBUG::2012-12-02 23:39:09,663::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-938::DEBUG::2012-12-02 23:39:09,663::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-938::ERROR::2012-12-02 23:39:09,663::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-939::DEBUG::2012-12-02 23:39:09,848::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-939::DEBUG::2012-12-02 23:39:09,849::task::568::TaskManager.Task::(_updateState) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::moving from state init -> state preparing >Thread-939::INFO::2012-12-02 23:39:09,849::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-939::DEBUG::2012-12-02 23:39:09,849::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`be60e675-06df-4353-91a8-f3774b4ce3ce`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-939::DEBUG::2012-12-02 23:39:09,849::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-939::DEBUG::2012-12-02 23:39:09,849::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-939::DEBUG::2012-12-02 23:39:09,849::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`be60e675-06df-4353-91a8-f3774b4ce3ce`::Granted request >Thread-939::INFO::2012-12-02 23:39:09,850::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-939::DEBUG::2012-12-02 23:39:09,850::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:09,850::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:09,850::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:09,851::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:09,851::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:09,851::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:09,851::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-939::DEBUG::2012-12-02 23:39:09,851::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-939::DEBUG::2012-12-02 23:39:09,851::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-939::DEBUG::2012-12-02 23:39:09,851::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-939::DEBUG::2012-12-02 23:39:09,852::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-939::DEBUG::2012-12-02 23:39:09,875::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-939::DEBUG::2012-12-02 23:39:09,875::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-939::DEBUG::2012-12-02 23:39:10,508::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-939::DEBUG::2012-12-02 23:39:10,556::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-939::DEBUG::2012-12-02 23:39:10,556::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,556::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,556::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,556::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,557::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,557::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,557::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-939::DEBUG::2012-12-02 23:39:10,557::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,559::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-939::DEBUG::2012-12-02 23:39:10,763::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-939::WARNING::2012-12-02 23:39:10,765::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-939::DEBUG::2012-12-02 23:39:10,765::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-939::DEBUG::2012-12-02 23:39:10,795::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-939::DEBUG::2012-12-02 23:39:10,796::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-939::DEBUG::2012-12-02 23:39:10,796::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-939::DEBUG::2012-12-02 23:39:10,796::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-939::ERROR::2012-12-02 23:39:10,796::task::833::TaskManager.Task::(_setError) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-939::DEBUG::2012-12-02 23:39:10,796::task::852::TaskManager.Task::(_run) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::Task._run: e8d1e738-49a8-4174-997f-5315ec87bbdc ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-939::DEBUG::2012-12-02 23:39:10,797::task::1177::TaskManager.Task::(stop) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::stopping in state preparing (force False) >Thread-939::DEBUG::2012-12-02 23:39:10,797::task::957::TaskManager.Task::(_decref) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::ref 1 aborting True >Thread-939::INFO::2012-12-02 23:39:10,797::task::1134::TaskManager.Task::(prepare) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-939::DEBUG::2012-12-02 23:39:10,797::task::1139::TaskManager.Task::(prepare) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::Prepare: aborted: Cannot find master domain >Thread-939::DEBUG::2012-12-02 23:39:10,797::task::957::TaskManager.Task::(_decref) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::ref 0 aborting True >Thread-939::DEBUG::2012-12-02 23:39:10,797::task::892::TaskManager.Task::(_doAbort) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::Task._doAbort: force False >Thread-939::DEBUG::2012-12-02 23:39:10,797::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-939::DEBUG::2012-12-02 23:39:10,798::task::568::TaskManager.Task::(_updateState) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::moving from state preparing -> state aborting >Thread-939::DEBUG::2012-12-02 23:39:10,798::task::523::TaskManager.Task::(__state_aborting) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::_aborting: recover policy none >Thread-939::DEBUG::2012-12-02 23:39:10,798::task::568::TaskManager.Task::(_updateState) Task=`e8d1e738-49a8-4174-997f-5315ec87bbdc`::moving from state aborting -> state failed >Thread-939::DEBUG::2012-12-02 23:39:10,798::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-939::DEBUG::2012-12-02 23:39:10,798::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-939::ERROR::2012-12-02 23:39:10,798::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1004::DEBUG::2012-12-02 23:39:18,185::task::568::TaskManager.Task::(_updateState) Task=`2e6c32bf-af7c-4d34-a452-2ba92e90e464`::moving from state init -> state preparing >Thread-1004::INFO::2012-12-02 23:39:18,185::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1004::INFO::2012-12-02 23:39:18,186::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1004::DEBUG::2012-12-02 23:39:18,186::task::1151::TaskManager.Task::(prepare) Task=`2e6c32bf-af7c-4d34-a452-2ba92e90e464`::finished: {} >Thread-1004::DEBUG::2012-12-02 23:39:18,186::task::568::TaskManager.Task::(_updateState) Task=`2e6c32bf-af7c-4d34-a452-2ba92e90e464`::moving from state preparing -> state finished >Thread-1004::DEBUG::2012-12-02 23:39:18,186::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1004::DEBUG::2012-12-02 23:39:18,186::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1004::DEBUG::2012-12-02 23:39:18,186::task::957::TaskManager.Task::(_decref) Task=`2e6c32bf-af7c-4d34-a452-2ba92e90e464`::ref 0 aborting False >Thread-1007::DEBUG::2012-12-02 23:39:21,103::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1007::DEBUG::2012-12-02 23:39:21,104::task::568::TaskManager.Task::(_updateState) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::moving from state init -> state preparing >Thread-1007::INFO::2012-12-02 23:39:21,104::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1007::ERROR::2012-12-02 23:39:21,104::task::833::TaskManager.Task::(_setError) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1007::DEBUG::2012-12-02 23:39:21,104::task::852::TaskManager.Task::(_run) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::Task._run: 8ffbee92-44a7-4b1f-98fe-a58dd53a270b ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1007::DEBUG::2012-12-02 23:39:21,104::task::1177::TaskManager.Task::(stop) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::stopping in state preparing (force False) >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::957::TaskManager.Task::(_decref) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::ref 1 aborting True >Thread-1007::INFO::2012-12-02 23:39:21,105::task::1134::TaskManager.Task::(prepare) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::1139::TaskManager.Task::(prepare) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::957::TaskManager.Task::(_decref) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::ref 0 aborting True >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::892::TaskManager.Task::(_doAbort) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::Task._doAbort: force False >Thread-1007::DEBUG::2012-12-02 23:39:21,105::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::568::TaskManager.Task::(_updateState) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::moving from state preparing -> state aborting >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::523::TaskManager.Task::(__state_aborting) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::_aborting: recover policy none >Thread-1007::DEBUG::2012-12-02 23:39:21,105::task::568::TaskManager.Task::(_updateState) Task=`8ffbee92-44a7-4b1f-98fe-a58dd53a270b`::moving from state aborting -> state failed >Thread-1007::DEBUG::2012-12-02 23:39:21,105::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1007::DEBUG::2012-12-02 23:39:21,106::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1007::ERROR::2012-12-02 23:39:21,106::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1008::DEBUG::2012-12-02 23:39:21,248::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1008::DEBUG::2012-12-02 23:39:21,248::task::568::TaskManager.Task::(_updateState) Task=`a873abc0-de1a-4764-b6a7-f7abfaca7830`::moving from state init -> state preparing >Thread-1008::INFO::2012-12-02 23:39:21,248::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1008::DEBUG::2012-12-02 23:39:21,248::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1008::DEBUG::2012-12-02 23:39:21,248::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1008::INFO::2012-12-02 23:39:21,248::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1008::DEBUG::2012-12-02 23:39:21,248::task::1151::TaskManager.Task::(prepare) Task=`a873abc0-de1a-4764-b6a7-f7abfaca7830`::finished: {'allTasksStatus': {}} >Thread-1008::DEBUG::2012-12-02 23:39:21,248::task::568::TaskManager.Task::(_updateState) Task=`a873abc0-de1a-4764-b6a7-f7abfaca7830`::moving from state preparing -> state finished >Thread-1008::DEBUG::2012-12-02 23:39:21,249::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1008::DEBUG::2012-12-02 23:39:21,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1008::DEBUG::2012-12-02 23:39:21,249::task::957::TaskManager.Task::(_decref) Task=`a873abc0-de1a-4764-b6a7-f7abfaca7830`::ref 0 aborting False >Thread-1009::DEBUG::2012-12-02 23:39:21,396::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1009::DEBUG::2012-12-02 23:39:21,396::task::568::TaskManager.Task::(_updateState) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::moving from state init -> state preparing >Thread-1009::INFO::2012-12-02 23:39:21,396::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1009::ERROR::2012-12-02 23:39:21,396::task::833::TaskManager.Task::(_setError) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1009::DEBUG::2012-12-02 23:39:21,397::task::852::TaskManager.Task::(_run) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::Task._run: faa08b8f-507b-4331-9d8f-066276346d62 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1009::DEBUG::2012-12-02 23:39:21,397::task::1177::TaskManager.Task::(stop) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::stopping in state preparing (force False) >Thread-1009::DEBUG::2012-12-02 23:39:21,397::task::957::TaskManager.Task::(_decref) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::ref 1 aborting True >Thread-1009::INFO::2012-12-02 23:39:21,397::task::1134::TaskManager.Task::(prepare) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1009::DEBUG::2012-12-02 23:39:21,397::task::1139::TaskManager.Task::(prepare) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1009::DEBUG::2012-12-02 23:39:21,397::task::957::TaskManager.Task::(_decref) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::ref 0 aborting True >Thread-1009::DEBUG::2012-12-02 23:39:21,397::task::892::TaskManager.Task::(_doAbort) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::Task._doAbort: force False >Thread-1009::DEBUG::2012-12-02 23:39:21,397::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1009::DEBUG::2012-12-02 23:39:21,398::task::568::TaskManager.Task::(_updateState) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::moving from state preparing -> state aborting >Thread-1009::DEBUG::2012-12-02 23:39:21,398::task::523::TaskManager.Task::(__state_aborting) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::_aborting: recover policy none >Thread-1009::DEBUG::2012-12-02 23:39:21,398::task::568::TaskManager.Task::(_updateState) Task=`faa08b8f-507b-4331-9d8f-066276346d62`::moving from state aborting -> state failed >Thread-1009::DEBUG::2012-12-02 23:39:21,398::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1009::DEBUG::2012-12-02 23:39:21,398::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1009::ERROR::2012-12-02 23:39:21,398::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1010::DEBUG::2012-12-02 23:39:21,587::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1010::DEBUG::2012-12-02 23:39:21,587::task::568::TaskManager.Task::(_updateState) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::moving from state init -> state preparing >Thread-1010::INFO::2012-12-02 23:39:21,587::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1010::DEBUG::2012-12-02 23:39:21,588::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`28ce8aeb-582d-4c2f-a303-a0d9964563ab`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1010::DEBUG::2012-12-02 23:39:21,588::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1010::DEBUG::2012-12-02 23:39:21,588::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1010::DEBUG::2012-12-02 23:39:21,588::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`28ce8aeb-582d-4c2f-a303-a0d9964563ab`::Granted request >Thread-1010::INFO::2012-12-02 23:39:21,588::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1010::DEBUG::2012-12-02 23:39:21,589::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:21,589::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:21,589::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:21,589::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:21,589::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:21,589::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:21,590::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1010::DEBUG::2012-12-02 23:39:21,590::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1010::DEBUG::2012-12-02 23:39:21,590::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1010::DEBUG::2012-12-02 23:39:21,590::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1010::DEBUG::2012-12-02 23:39:21,590::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1010::DEBUG::2012-12-02 23:39:21,613::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1010::DEBUG::2012-12-02 23:39:21,613::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1010::DEBUG::2012-12-02 23:39:22,248::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1010::DEBUG::2012-12-02 23:39:22,314::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1010::DEBUG::2012-12-02 23:39:22,314::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,314::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,315::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,315::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,315::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,315::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,315::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1010::DEBUG::2012-12-02 23:39:22,315::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,317::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1010::DEBUG::2012-12-02 23:39:22,522::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1010::WARNING::2012-12-02 23:39:22,524::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1010::DEBUG::2012-12-02 23:39:22,524::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1010::DEBUG::2012-12-02 23:39:22,553::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1010::DEBUG::2012-12-02 23:39:22,553::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1010::DEBUG::2012-12-02 23:39:22,553::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1010::DEBUG::2012-12-02 23:39:22,553::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1010::ERROR::2012-12-02 23:39:22,554::task::833::TaskManager.Task::(_setError) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1010::DEBUG::2012-12-02 23:39:22,554::task::852::TaskManager.Task::(_run) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::Task._run: dc27d75d-1376-4225-9c41-390c0387d1bb ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1010::DEBUG::2012-12-02 23:39:22,554::task::1177::TaskManager.Task::(stop) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::stopping in state preparing (force False) >Thread-1010::DEBUG::2012-12-02 23:39:22,554::task::957::TaskManager.Task::(_decref) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::ref 1 aborting True >Thread-1010::INFO::2012-12-02 23:39:22,555::task::1134::TaskManager.Task::(prepare) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1010::DEBUG::2012-12-02 23:39:22,555::task::1139::TaskManager.Task::(prepare) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::Prepare: aborted: Cannot find master domain >Thread-1010::DEBUG::2012-12-02 23:39:22,555::task::957::TaskManager.Task::(_decref) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::ref 0 aborting True >Thread-1010::DEBUG::2012-12-02 23:39:22,555::task::892::TaskManager.Task::(_doAbort) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::Task._doAbort: force False >Thread-1010::DEBUG::2012-12-02 23:39:22,555::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1010::DEBUG::2012-12-02 23:39:22,555::task::568::TaskManager.Task::(_updateState) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::moving from state preparing -> state aborting >Thread-1010::DEBUG::2012-12-02 23:39:22,555::task::523::TaskManager.Task::(__state_aborting) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::_aborting: recover policy none >Thread-1010::DEBUG::2012-12-02 23:39:22,555::task::568::TaskManager.Task::(_updateState) Task=`dc27d75d-1376-4225-9c41-390c0387d1bb`::moving from state aborting -> state failed >Thread-1010::DEBUG::2012-12-02 23:39:22,556::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1010::DEBUG::2012-12-02 23:39:22,556::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1010::ERROR::2012-12-02 23:39:22,556::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1075::DEBUG::2012-12-02 23:39:29,131::task::568::TaskManager.Task::(_updateState) Task=`94bd12bf-383c-41eb-8220-694a7fae3fb2`::moving from state init -> state preparing >Thread-1075::INFO::2012-12-02 23:39:29,132::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1075::INFO::2012-12-02 23:39:29,132::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1075::DEBUG::2012-12-02 23:39:29,132::task::1151::TaskManager.Task::(prepare) Task=`94bd12bf-383c-41eb-8220-694a7fae3fb2`::finished: {} >Thread-1075::DEBUG::2012-12-02 23:39:29,132::task::568::TaskManager.Task::(_updateState) Task=`94bd12bf-383c-41eb-8220-694a7fae3fb2`::moving from state preparing -> state finished >Thread-1075::DEBUG::2012-12-02 23:39:29,132::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1075::DEBUG::2012-12-02 23:39:29,132::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1075::DEBUG::2012-12-02 23:39:29,132::task::957::TaskManager.Task::(_decref) Task=`94bd12bf-383c-41eb-8220-694a7fae3fb2`::ref 0 aborting False >Thread-1078::DEBUG::2012-12-02 23:39:32,876::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1078::DEBUG::2012-12-02 23:39:32,877::task::568::TaskManager.Task::(_updateState) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::moving from state init -> state preparing >Thread-1078::INFO::2012-12-02 23:39:32,877::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1078::ERROR::2012-12-02 23:39:32,877::task::833::TaskManager.Task::(_setError) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1078::DEBUG::2012-12-02 23:39:32,877::task::852::TaskManager.Task::(_run) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::Task._run: a3333d60-d799-4349-93f6-a8ebe6d3b738 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1078::DEBUG::2012-12-02 23:39:32,877::task::1177::TaskManager.Task::(stop) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::stopping in state preparing (force False) >Thread-1078::DEBUG::2012-12-02 23:39:32,878::task::957::TaskManager.Task::(_decref) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::ref 1 aborting True >Thread-1078::INFO::2012-12-02 23:39:32,878::task::1134::TaskManager.Task::(prepare) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1078::DEBUG::2012-12-02 23:39:32,878::task::1139::TaskManager.Task::(prepare) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1078::DEBUG::2012-12-02 23:39:32,878::task::957::TaskManager.Task::(_decref) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::ref 0 aborting True >Thread-1078::DEBUG::2012-12-02 23:39:32,878::task::892::TaskManager.Task::(_doAbort) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::Task._doAbort: force False >Thread-1078::DEBUG::2012-12-02 23:39:32,878::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1078::DEBUG::2012-12-02 23:39:32,878::task::568::TaskManager.Task::(_updateState) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::moving from state preparing -> state aborting >Thread-1078::DEBUG::2012-12-02 23:39:32,878::task::523::TaskManager.Task::(__state_aborting) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::_aborting: recover policy none >Thread-1078::DEBUG::2012-12-02 23:39:32,879::task::568::TaskManager.Task::(_updateState) Task=`a3333d60-d799-4349-93f6-a8ebe6d3b738`::moving from state aborting -> state failed >Thread-1078::DEBUG::2012-12-02 23:39:32,879::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1078::DEBUG::2012-12-02 23:39:32,879::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1078::ERROR::2012-12-02 23:39:32,879::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1079::DEBUG::2012-12-02 23:39:33,024::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1079::DEBUG::2012-12-02 23:39:33,024::task::568::TaskManager.Task::(_updateState) Task=`db87b8ae-e3f3-4abc-b1a1-cec240050abe`::moving from state init -> state preparing >Thread-1079::INFO::2012-12-02 23:39:33,025::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1079::DEBUG::2012-12-02 23:39:33,025::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1079::DEBUG::2012-12-02 23:39:33,025::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1079::INFO::2012-12-02 23:39:33,025::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1079::DEBUG::2012-12-02 23:39:33,025::task::1151::TaskManager.Task::(prepare) Task=`db87b8ae-e3f3-4abc-b1a1-cec240050abe`::finished: {'allTasksStatus': {}} >Thread-1079::DEBUG::2012-12-02 23:39:33,025::task::568::TaskManager.Task::(_updateState) Task=`db87b8ae-e3f3-4abc-b1a1-cec240050abe`::moving from state preparing -> state finished >Thread-1079::DEBUG::2012-12-02 23:39:33,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1079::DEBUG::2012-12-02 23:39:33,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1079::DEBUG::2012-12-02 23:39:33,025::task::957::TaskManager.Task::(_decref) Task=`db87b8ae-e3f3-4abc-b1a1-cec240050abe`::ref 0 aborting False >Thread-1080::DEBUG::2012-12-02 23:39:33,164::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1080::DEBUG::2012-12-02 23:39:33,165::task::568::TaskManager.Task::(_updateState) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::moving from state init -> state preparing >Thread-1080::INFO::2012-12-02 23:39:33,165::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1080::ERROR::2012-12-02 23:39:33,165::task::833::TaskManager.Task::(_setError) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1080::DEBUG::2012-12-02 23:39:33,165::task::852::TaskManager.Task::(_run) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::Task._run: 543831f9-ece5-4e1f-a3fc-52cadaff2ff5 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1080::DEBUG::2012-12-02 23:39:33,165::task::1177::TaskManager.Task::(stop) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::stopping in state preparing (force False) >Thread-1080::DEBUG::2012-12-02 23:39:33,165::task::957::TaskManager.Task::(_decref) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::ref 1 aborting True >Thread-1080::INFO::2012-12-02 23:39:33,166::task::1134::TaskManager.Task::(prepare) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1080::DEBUG::2012-12-02 23:39:33,166::task::1139::TaskManager.Task::(prepare) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1080::DEBUG::2012-12-02 23:39:33,166::task::957::TaskManager.Task::(_decref) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::ref 0 aborting True >Thread-1080::DEBUG::2012-12-02 23:39:33,166::task::892::TaskManager.Task::(_doAbort) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::Task._doAbort: force False >Thread-1080::DEBUG::2012-12-02 23:39:33,166::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1080::DEBUG::2012-12-02 23:39:33,166::task::568::TaskManager.Task::(_updateState) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::moving from state preparing -> state aborting >Thread-1080::DEBUG::2012-12-02 23:39:33,166::task::523::TaskManager.Task::(__state_aborting) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::_aborting: recover policy none >Thread-1080::DEBUG::2012-12-02 23:39:33,166::task::568::TaskManager.Task::(_updateState) Task=`543831f9-ece5-4e1f-a3fc-52cadaff2ff5`::moving from state aborting -> state failed >Thread-1080::DEBUG::2012-12-02 23:39:33,166::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1080::DEBUG::2012-12-02 23:39:33,166::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1080::ERROR::2012-12-02 23:39:33,166::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1081::DEBUG::2012-12-02 23:39:33,349::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1081::DEBUG::2012-12-02 23:39:33,350::task::568::TaskManager.Task::(_updateState) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::moving from state init -> state preparing >Thread-1081::INFO::2012-12-02 23:39:33,350::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1081::DEBUG::2012-12-02 23:39:33,350::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`db8f164a-30f3-461e-a626-202677ed0575`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1081::DEBUG::2012-12-02 23:39:33,350::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1081::DEBUG::2012-12-02 23:39:33,351::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1081::DEBUG::2012-12-02 23:39:33,351::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`db8f164a-30f3-461e-a626-202677ed0575`::Granted request >Thread-1081::INFO::2012-12-02 23:39:33,351::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1081::DEBUG::2012-12-02 23:39:33,351::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:33,352::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:33,352::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:33,352::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:33,352::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:33,352::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:33,352::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1081::DEBUG::2012-12-02 23:39:33,352::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1081::DEBUG::2012-12-02 23:39:33,353::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1081::DEBUG::2012-12-02 23:39:33,353::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1081::DEBUG::2012-12-02 23:39:33,353::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1081::DEBUG::2012-12-02 23:39:33,375::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1081::DEBUG::2012-12-02 23:39:33,375::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1081::DEBUG::2012-12-02 23:39:34,009::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1081::DEBUG::2012-12-02 23:39:34,081::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1081::DEBUG::2012-12-02 23:39:34,081::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,082::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,082::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,082::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,082::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,082::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,082::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1081::DEBUG::2012-12-02 23:39:34,083::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,084::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1081::DEBUG::2012-12-02 23:39:34,290::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1081::WARNING::2012-12-02 23:39:34,292::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1081::DEBUG::2012-12-02 23:39:34,292::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1081::DEBUG::2012-12-02 23:39:34,321::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1081::DEBUG::2012-12-02 23:39:34,322::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1081::DEBUG::2012-12-02 23:39:34,322::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1081::DEBUG::2012-12-02 23:39:34,322::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1081::ERROR::2012-12-02 23:39:34,322::task::833::TaskManager.Task::(_setError) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1081::DEBUG::2012-12-02 23:39:34,322::task::852::TaskManager.Task::(_run) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::Task._run: 40d20aa5-8b9a-445a-9c75-307560792c29 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1081::DEBUG::2012-12-02 23:39:34,323::task::1177::TaskManager.Task::(stop) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::stopping in state preparing (force False) >Thread-1081::DEBUG::2012-12-02 23:39:34,323::task::957::TaskManager.Task::(_decref) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::ref 1 aborting True >Thread-1081::INFO::2012-12-02 23:39:34,323::task::1134::TaskManager.Task::(prepare) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1081::DEBUG::2012-12-02 23:39:34,323::task::1139::TaskManager.Task::(prepare) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::Prepare: aborted: Cannot find master domain >Thread-1081::DEBUG::2012-12-02 23:39:34,323::task::957::TaskManager.Task::(_decref) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::ref 0 aborting True >Thread-1081::DEBUG::2012-12-02 23:39:34,323::task::892::TaskManager.Task::(_doAbort) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::Task._doAbort: force False >Thread-1081::DEBUG::2012-12-02 23:39:34,323::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1081::DEBUG::2012-12-02 23:39:34,323::task::568::TaskManager.Task::(_updateState) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::moving from state preparing -> state aborting >Thread-1081::DEBUG::2012-12-02 23:39:34,324::task::523::TaskManager.Task::(__state_aborting) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::_aborting: recover policy none >Thread-1081::DEBUG::2012-12-02 23:39:34,324::task::568::TaskManager.Task::(_updateState) Task=`40d20aa5-8b9a-445a-9c75-307560792c29`::moving from state aborting -> state failed >Thread-1081::DEBUG::2012-12-02 23:39:34,324::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1081::DEBUG::2012-12-02 23:39:34,324::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1081::ERROR::2012-12-02 23:39:34,324::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1146::DEBUG::2012-12-02 23:39:40,089::task::568::TaskManager.Task::(_updateState) Task=`8e8737b9-507a-4682-b3c1-23a06f78369f`::moving from state init -> state preparing >Thread-1146::INFO::2012-12-02 23:39:40,089::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1146::INFO::2012-12-02 23:39:40,089::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1146::DEBUG::2012-12-02 23:39:40,089::task::1151::TaskManager.Task::(prepare) Task=`8e8737b9-507a-4682-b3c1-23a06f78369f`::finished: {} >Thread-1146::DEBUG::2012-12-02 23:39:40,089::task::568::TaskManager.Task::(_updateState) Task=`8e8737b9-507a-4682-b3c1-23a06f78369f`::moving from state preparing -> state finished >Thread-1146::DEBUG::2012-12-02 23:39:40,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1146::DEBUG::2012-12-02 23:39:40,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1146::DEBUG::2012-12-02 23:39:40,089::task::957::TaskManager.Task::(_decref) Task=`8e8737b9-507a-4682-b3c1-23a06f78369f`::ref 0 aborting False >Thread-1150::DEBUG::2012-12-02 23:39:44,653::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1150::DEBUG::2012-12-02 23:39:44,653::task::568::TaskManager.Task::(_updateState) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::moving from state init -> state preparing >Thread-1150::INFO::2012-12-02 23:39:44,653::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1150::ERROR::2012-12-02 23:39:44,653::task::833::TaskManager.Task::(_setError) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1150::DEBUG::2012-12-02 23:39:44,654::task::852::TaskManager.Task::(_run) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::Task._run: 58c0cb6f-400d-49d0-8da4-df8e4e75e9b2 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1150::DEBUG::2012-12-02 23:39:44,654::task::1177::TaskManager.Task::(stop) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::stopping in state preparing (force False) >Thread-1150::DEBUG::2012-12-02 23:39:44,654::task::957::TaskManager.Task::(_decref) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::ref 1 aborting True >Thread-1150::INFO::2012-12-02 23:39:44,654::task::1134::TaskManager.Task::(prepare) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1150::DEBUG::2012-12-02 23:39:44,654::task::1139::TaskManager.Task::(prepare) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1150::DEBUG::2012-12-02 23:39:44,654::task::957::TaskManager.Task::(_decref) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::ref 0 aborting True >Thread-1150::DEBUG::2012-12-02 23:39:44,654::task::892::TaskManager.Task::(_doAbort) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::Task._doAbort: force False >Thread-1150::DEBUG::2012-12-02 23:39:44,655::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1150::DEBUG::2012-12-02 23:39:44,655::task::568::TaskManager.Task::(_updateState) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::moving from state preparing -> state aborting >Thread-1150::DEBUG::2012-12-02 23:39:44,655::task::523::TaskManager.Task::(__state_aborting) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::_aborting: recover policy none >Thread-1150::DEBUG::2012-12-02 23:39:44,655::task::568::TaskManager.Task::(_updateState) Task=`58c0cb6f-400d-49d0-8da4-df8e4e75e9b2`::moving from state aborting -> state failed >Thread-1150::DEBUG::2012-12-02 23:39:44,655::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1150::DEBUG::2012-12-02 23:39:44,655::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1150::ERROR::2012-12-02 23:39:44,655::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1151::DEBUG::2012-12-02 23:39:44,794::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1151::DEBUG::2012-12-02 23:39:44,794::task::568::TaskManager.Task::(_updateState) Task=`72d0605d-c530-4875-a06e-3d7a3ad61800`::moving from state init -> state preparing >Thread-1151::INFO::2012-12-02 23:39:44,795::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1151::DEBUG::2012-12-02 23:39:44,795::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1151::DEBUG::2012-12-02 23:39:44,795::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1151::INFO::2012-12-02 23:39:44,795::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1151::DEBUG::2012-12-02 23:39:44,795::task::1151::TaskManager.Task::(prepare) Task=`72d0605d-c530-4875-a06e-3d7a3ad61800`::finished: {'allTasksStatus': {}} >Thread-1151::DEBUG::2012-12-02 23:39:44,795::task::568::TaskManager.Task::(_updateState) Task=`72d0605d-c530-4875-a06e-3d7a3ad61800`::moving from state preparing -> state finished >Thread-1151::DEBUG::2012-12-02 23:39:44,795::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1151::DEBUG::2012-12-02 23:39:44,795::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1151::DEBUG::2012-12-02 23:39:44,795::task::957::TaskManager.Task::(_decref) Task=`72d0605d-c530-4875-a06e-3d7a3ad61800`::ref 0 aborting False >Thread-1152::DEBUG::2012-12-02 23:39:44,941::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1152::DEBUG::2012-12-02 23:39:44,941::task::568::TaskManager.Task::(_updateState) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::moving from state init -> state preparing >Thread-1152::INFO::2012-12-02 23:39:44,941::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1152::ERROR::2012-12-02 23:39:44,941::task::833::TaskManager.Task::(_setError) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1152::DEBUG::2012-12-02 23:39:44,941::task::852::TaskManager.Task::(_run) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::Task._run: ae6903d6-6c29-4798-bf80-ae799737bd2a ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1152::DEBUG::2012-12-02 23:39:44,942::task::1177::TaskManager.Task::(stop) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::stopping in state preparing (force False) >Thread-1152::DEBUG::2012-12-02 23:39:44,942::task::957::TaskManager.Task::(_decref) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::ref 1 aborting True >Thread-1152::INFO::2012-12-02 23:39:44,942::task::1134::TaskManager.Task::(prepare) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1152::DEBUG::2012-12-02 23:39:44,942::task::1139::TaskManager.Task::(prepare) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1152::DEBUG::2012-12-02 23:39:44,942::task::957::TaskManager.Task::(_decref) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::ref 0 aborting True >Thread-1152::DEBUG::2012-12-02 23:39:44,942::task::892::TaskManager.Task::(_doAbort) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::Task._doAbort: force False >Thread-1152::DEBUG::2012-12-02 23:39:44,942::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1152::DEBUG::2012-12-02 23:39:44,942::task::568::TaskManager.Task::(_updateState) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::moving from state preparing -> state aborting >Thread-1152::DEBUG::2012-12-02 23:39:44,943::task::523::TaskManager.Task::(__state_aborting) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::_aborting: recover policy none >Thread-1152::DEBUG::2012-12-02 23:39:44,943::task::568::TaskManager.Task::(_updateState) Task=`ae6903d6-6c29-4798-bf80-ae799737bd2a`::moving from state aborting -> state failed >Thread-1152::DEBUG::2012-12-02 23:39:44,943::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1152::DEBUG::2012-12-02 23:39:44,943::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1152::ERROR::2012-12-02 23:39:44,943::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1153::DEBUG::2012-12-02 23:39:45,135::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1153::DEBUG::2012-12-02 23:39:45,135::task::568::TaskManager.Task::(_updateState) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::moving from state init -> state preparing >Thread-1153::INFO::2012-12-02 23:39:45,136::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1153::DEBUG::2012-12-02 23:39:45,136::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1668c547-e65e-44bc-b2ec-704d13340d0e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1153::DEBUG::2012-12-02 23:39:45,136::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1153::DEBUG::2012-12-02 23:39:45,136::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1153::DEBUG::2012-12-02 23:39:45,136::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1668c547-e65e-44bc-b2ec-704d13340d0e`::Granted request >Thread-1153::INFO::2012-12-02 23:39:45,137::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1153::DEBUG::2012-12-02 23:39:45,137::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,137::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,137::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,138::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,138::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,138::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,138::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1153::DEBUG::2012-12-02 23:39:45,138::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1153::DEBUG::2012-12-02 23:39:45,138::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1153::DEBUG::2012-12-02 23:39:45,138::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1153::DEBUG::2012-12-02 23:39:45,139::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1153::DEBUG::2012-12-02 23:39:45,165::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1153::DEBUG::2012-12-02 23:39:45,165::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1153::DEBUG::2012-12-02 23:39:45,801::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1153::DEBUG::2012-12-02 23:39:45,883::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1153::DEBUG::2012-12-02 23:39:45,884::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,884::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,884::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,884::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,884::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,884::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,885::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1153::DEBUG::2012-12-02 23:39:45,885::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:45,886::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1153::DEBUG::2012-12-02 23:39:46,089::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1153::WARNING::2012-12-02 23:39:46,090::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1153::DEBUG::2012-12-02 23:39:46,090::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1153::DEBUG::2012-12-02 23:39:46,120::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1153::DEBUG::2012-12-02 23:39:46,120::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1153::DEBUG::2012-12-02 23:39:46,120::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1153::DEBUG::2012-12-02 23:39:46,120::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1153::ERROR::2012-12-02 23:39:46,120::task::833::TaskManager.Task::(_setError) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1153::DEBUG::2012-12-02 23:39:46,121::task::852::TaskManager.Task::(_run) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::Task._run: 10089cac-072a-420f-8eac-7991b9ad8721 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1153::DEBUG::2012-12-02 23:39:46,121::task::1177::TaskManager.Task::(stop) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::stopping in state preparing (force False) >Thread-1153::DEBUG::2012-12-02 23:39:46,121::task::957::TaskManager.Task::(_decref) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::ref 1 aborting True >Thread-1153::INFO::2012-12-02 23:39:46,121::task::1134::TaskManager.Task::(prepare) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1153::DEBUG::2012-12-02 23:39:46,121::task::1139::TaskManager.Task::(prepare) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::Prepare: aborted: Cannot find master domain >Thread-1153::DEBUG::2012-12-02 23:39:46,122::task::957::TaskManager.Task::(_decref) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::ref 0 aborting True >Thread-1153::DEBUG::2012-12-02 23:39:46,122::task::892::TaskManager.Task::(_doAbort) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::Task._doAbort: force False >Thread-1153::DEBUG::2012-12-02 23:39:46,122::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1153::DEBUG::2012-12-02 23:39:46,122::task::568::TaskManager.Task::(_updateState) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::moving from state preparing -> state aborting >Thread-1153::DEBUG::2012-12-02 23:39:46,122::task::523::TaskManager.Task::(__state_aborting) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::_aborting: recover policy none >Thread-1153::DEBUG::2012-12-02 23:39:46,122::task::568::TaskManager.Task::(_updateState) Task=`10089cac-072a-420f-8eac-7991b9ad8721`::moving from state aborting -> state failed >Thread-1153::DEBUG::2012-12-02 23:39:46,122::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1153::DEBUG::2012-12-02 23:39:46,122::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1153::ERROR::2012-12-02 23:39:46,123::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1217::DEBUG::2012-12-02 23:39:51,004::task::568::TaskManager.Task::(_updateState) Task=`03addce2-5cec-4ae7-9771-f17b8b47231c`::moving from state init -> state preparing >Thread-1217::INFO::2012-12-02 23:39:51,004::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1217::INFO::2012-12-02 23:39:51,005::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1217::DEBUG::2012-12-02 23:39:51,005::task::1151::TaskManager.Task::(prepare) Task=`03addce2-5cec-4ae7-9771-f17b8b47231c`::finished: {} >Thread-1217::DEBUG::2012-12-02 23:39:51,005::task::568::TaskManager.Task::(_updateState) Task=`03addce2-5cec-4ae7-9771-f17b8b47231c`::moving from state preparing -> state finished >Thread-1217::DEBUG::2012-12-02 23:39:51,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1217::DEBUG::2012-12-02 23:39:51,005::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1217::DEBUG::2012-12-02 23:39:51,005::task::957::TaskManager.Task::(_decref) Task=`03addce2-5cec-4ae7-9771-f17b8b47231c`::ref 0 aborting False >Thread-1221::DEBUG::2012-12-02 23:39:56,444::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1221::DEBUG::2012-12-02 23:39:56,444::task::568::TaskManager.Task::(_updateState) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::moving from state init -> state preparing >Thread-1221::INFO::2012-12-02 23:39:56,445::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1221::ERROR::2012-12-02 23:39:56,445::task::833::TaskManager.Task::(_setError) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1221::DEBUG::2012-12-02 23:39:56,445::task::852::TaskManager.Task::(_run) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::Task._run: 66fd5061-f31e-4bdd-ba79-bb9670cb2832 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1221::DEBUG::2012-12-02 23:39:56,445::task::1177::TaskManager.Task::(stop) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::stopping in state preparing (force False) >Thread-1221::DEBUG::2012-12-02 23:39:56,445::task::957::TaskManager.Task::(_decref) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::ref 1 aborting True >Thread-1221::INFO::2012-12-02 23:39:56,445::task::1134::TaskManager.Task::(prepare) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1221::DEBUG::2012-12-02 23:39:56,445::task::1139::TaskManager.Task::(prepare) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1221::DEBUG::2012-12-02 23:39:56,446::task::957::TaskManager.Task::(_decref) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::ref 0 aborting True >Thread-1221::DEBUG::2012-12-02 23:39:56,446::task::892::TaskManager.Task::(_doAbort) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::Task._doAbort: force False >Thread-1221::DEBUG::2012-12-02 23:39:56,446::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1221::DEBUG::2012-12-02 23:39:56,446::task::568::TaskManager.Task::(_updateState) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::moving from state preparing -> state aborting >Thread-1221::DEBUG::2012-12-02 23:39:56,446::task::523::TaskManager.Task::(__state_aborting) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::_aborting: recover policy none >Thread-1221::DEBUG::2012-12-02 23:39:56,446::task::568::TaskManager.Task::(_updateState) Task=`66fd5061-f31e-4bdd-ba79-bb9670cb2832`::moving from state aborting -> state failed >Thread-1221::DEBUG::2012-12-02 23:39:56,446::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1221::DEBUG::2012-12-02 23:39:56,446::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1221::ERROR::2012-12-02 23:39:56,446::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1222::DEBUG::2012-12-02 23:39:56,590::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1222::DEBUG::2012-12-02 23:39:56,591::task::568::TaskManager.Task::(_updateState) Task=`5c162cc6-a649-431b-ba60-ef55a2ae1091`::moving from state init -> state preparing >Thread-1222::INFO::2012-12-02 23:39:56,591::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1222::DEBUG::2012-12-02 23:39:56,591::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1222::DEBUG::2012-12-02 23:39:56,591::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1222::INFO::2012-12-02 23:39:56,591::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1222::DEBUG::2012-12-02 23:39:56,591::task::1151::TaskManager.Task::(prepare) Task=`5c162cc6-a649-431b-ba60-ef55a2ae1091`::finished: {'allTasksStatus': {}} >Thread-1222::DEBUG::2012-12-02 23:39:56,591::task::568::TaskManager.Task::(_updateState) Task=`5c162cc6-a649-431b-ba60-ef55a2ae1091`::moving from state preparing -> state finished >Thread-1222::DEBUG::2012-12-02 23:39:56,591::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1222::DEBUG::2012-12-02 23:39:56,592::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1222::DEBUG::2012-12-02 23:39:56,592::task::957::TaskManager.Task::(_decref) Task=`5c162cc6-a649-431b-ba60-ef55a2ae1091`::ref 0 aborting False >Thread-1223::DEBUG::2012-12-02 23:39:56,738::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1223::DEBUG::2012-12-02 23:39:56,738::task::568::TaskManager.Task::(_updateState) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::moving from state init -> state preparing >Thread-1223::INFO::2012-12-02 23:39:56,738::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1223::ERROR::2012-12-02 23:39:56,738::task::833::TaskManager.Task::(_setError) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1223::DEBUG::2012-12-02 23:39:56,738::task::852::TaskManager.Task::(_run) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::Task._run: 68c51b3c-fa3f-4ffa-ab51-87215d2bce39 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1223::DEBUG::2012-12-02 23:39:56,739::task::1177::TaskManager.Task::(stop) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::stopping in state preparing (force False) >Thread-1223::DEBUG::2012-12-02 23:39:56,739::task::957::TaskManager.Task::(_decref) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::ref 1 aborting True >Thread-1223::INFO::2012-12-02 23:39:56,739::task::1134::TaskManager.Task::(prepare) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1223::DEBUG::2012-12-02 23:39:56,739::task::1139::TaskManager.Task::(prepare) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1223::DEBUG::2012-12-02 23:39:56,739::task::957::TaskManager.Task::(_decref) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::ref 0 aborting True >Thread-1223::DEBUG::2012-12-02 23:39:56,739::task::892::TaskManager.Task::(_doAbort) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::Task._doAbort: force False >Thread-1223::DEBUG::2012-12-02 23:39:56,739::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1223::DEBUG::2012-12-02 23:39:56,740::task::568::TaskManager.Task::(_updateState) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::moving from state preparing -> state aborting >Thread-1223::DEBUG::2012-12-02 23:39:56,740::task::523::TaskManager.Task::(__state_aborting) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::_aborting: recover policy none >Thread-1223::DEBUG::2012-12-02 23:39:56,740::task::568::TaskManager.Task::(_updateState) Task=`68c51b3c-fa3f-4ffa-ab51-87215d2bce39`::moving from state aborting -> state failed >Thread-1223::DEBUG::2012-12-02 23:39:56,740::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1223::DEBUG::2012-12-02 23:39:56,740::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1223::ERROR::2012-12-02 23:39:56,740::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1224::DEBUG::2012-12-02 23:39:56,927::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1224::DEBUG::2012-12-02 23:39:56,927::task::568::TaskManager.Task::(_updateState) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::moving from state init -> state preparing >Thread-1224::INFO::2012-12-02 23:39:56,928::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1224::DEBUG::2012-12-02 23:39:56,928::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a40550e8-4f98-43d2-b212-438f3dc106d2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1224::DEBUG::2012-12-02 23:39:56,928::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1224::DEBUG::2012-12-02 23:39:56,928::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1224::DEBUG::2012-12-02 23:39:56,928::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a40550e8-4f98-43d2-b212-438f3dc106d2`::Granted request >Thread-1224::INFO::2012-12-02 23:39:56,929::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1224::DEBUG::2012-12-02 23:39:56,929::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:56,929::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:56,929::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:56,930::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:56,930::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:56,930::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:56,930::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1224::DEBUG::2012-12-02 23:39:56,930::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1224::DEBUG::2012-12-02 23:39:56,930::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1224::DEBUG::2012-12-02 23:39:56,930::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1224::DEBUG::2012-12-02 23:39:56,931::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1224::DEBUG::2012-12-02 23:39:56,955::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1224::DEBUG::2012-12-02 23:39:56,955::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1224::DEBUG::2012-12-02 23:39:57,588::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1224::DEBUG::2012-12-02 23:39:57,656::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1224::DEBUG::2012-12-02 23:39:57,656::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,657::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,657::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,657::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,657::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,657::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,657::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1224::DEBUG::2012-12-02 23:39:57,658::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,659::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1224::DEBUG::2012-12-02 23:39:57,863::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1224::WARNING::2012-12-02 23:39:57,864::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1224::DEBUG::2012-12-02 23:39:57,864::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1224::DEBUG::2012-12-02 23:39:57,899::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1224::DEBUG::2012-12-02 23:39:57,900::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1224::DEBUG::2012-12-02 23:39:57,900::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1224::DEBUG::2012-12-02 23:39:57,900::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1224::ERROR::2012-12-02 23:39:57,900::task::833::TaskManager.Task::(_setError) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1224::DEBUG::2012-12-02 23:39:57,901::task::852::TaskManager.Task::(_run) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::Task._run: e1666f30-91d0-4af5-a12a-8a8882ed934f ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1224::DEBUG::2012-12-02 23:39:57,901::task::1177::TaskManager.Task::(stop) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::stopping in state preparing (force False) >Thread-1224::DEBUG::2012-12-02 23:39:57,901::task::957::TaskManager.Task::(_decref) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::ref 1 aborting True >Thread-1224::INFO::2012-12-02 23:39:57,901::task::1134::TaskManager.Task::(prepare) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1224::DEBUG::2012-12-02 23:39:57,901::task::1139::TaskManager.Task::(prepare) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::Prepare: aborted: Cannot find master domain >Thread-1224::DEBUG::2012-12-02 23:39:57,901::task::957::TaskManager.Task::(_decref) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::ref 0 aborting True >Thread-1224::DEBUG::2012-12-02 23:39:57,901::task::892::TaskManager.Task::(_doAbort) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::Task._doAbort: force False >Thread-1224::DEBUG::2012-12-02 23:39:57,902::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1224::DEBUG::2012-12-02 23:39:57,902::task::568::TaskManager.Task::(_updateState) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::moving from state preparing -> state aborting >Thread-1224::DEBUG::2012-12-02 23:39:57,902::task::523::TaskManager.Task::(__state_aborting) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::_aborting: recover policy none >Thread-1224::DEBUG::2012-12-02 23:39:57,902::task::568::TaskManager.Task::(_updateState) Task=`e1666f30-91d0-4af5-a12a-8a8882ed934f`::moving from state aborting -> state failed >Thread-1224::DEBUG::2012-12-02 23:39:57,902::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1224::DEBUG::2012-12-02 23:39:57,902::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1224::ERROR::2012-12-02 23:39:57,902::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1288::DEBUG::2012-12-02 23:40:01,930::task::568::TaskManager.Task::(_updateState) Task=`a296b36f-a36d-4164-a5e8-f69f0e2f0195`::moving from state init -> state preparing >Thread-1288::INFO::2012-12-02 23:40:01,931::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1288::INFO::2012-12-02 23:40:01,931::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1288::DEBUG::2012-12-02 23:40:01,931::task::1151::TaskManager.Task::(prepare) Task=`a296b36f-a36d-4164-a5e8-f69f0e2f0195`::finished: {} >Thread-1288::DEBUG::2012-12-02 23:40:01,931::task::568::TaskManager.Task::(_updateState) Task=`a296b36f-a36d-4164-a5e8-f69f0e2f0195`::moving from state preparing -> state finished >Thread-1288::DEBUG::2012-12-02 23:40:01,931::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1288::DEBUG::2012-12-02 23:40:01,931::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1288::DEBUG::2012-12-02 23:40:01,932::task::957::TaskManager.Task::(_decref) Task=`a296b36f-a36d-4164-a5e8-f69f0e2f0195`::ref 0 aborting False >Thread-1292::DEBUG::2012-12-02 23:40:08,229::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1292::DEBUG::2012-12-02 23:40:08,230::task::568::TaskManager.Task::(_updateState) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::moving from state init -> state preparing >Thread-1292::INFO::2012-12-02 23:40:08,230::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1292::ERROR::2012-12-02 23:40:08,230::task::833::TaskManager.Task::(_setError) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1292::DEBUG::2012-12-02 23:40:08,233::task::852::TaskManager.Task::(_run) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::Task._run: 5e6c4482-2b5b-4575-9948-a8d6c9f3143d ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1292::DEBUG::2012-12-02 23:40:08,233::task::1177::TaskManager.Task::(stop) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::stopping in state preparing (force False) >Thread-1292::DEBUG::2012-12-02 23:40:08,233::task::957::TaskManager.Task::(_decref) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::ref 1 aborting True >Thread-1292::INFO::2012-12-02 23:40:08,233::task::1134::TaskManager.Task::(prepare) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1292::DEBUG::2012-12-02 23:40:08,233::task::1139::TaskManager.Task::(prepare) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1292::DEBUG::2012-12-02 23:40:08,234::task::957::TaskManager.Task::(_decref) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::ref 0 aborting True >Thread-1292::DEBUG::2012-12-02 23:40:08,234::task::892::TaskManager.Task::(_doAbort) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::Task._doAbort: force False >Thread-1292::DEBUG::2012-12-02 23:40:08,234::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1292::DEBUG::2012-12-02 23:40:08,234::task::568::TaskManager.Task::(_updateState) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::moving from state preparing -> state aborting >Thread-1292::DEBUG::2012-12-02 23:40:08,234::task::523::TaskManager.Task::(__state_aborting) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::_aborting: recover policy none >Thread-1292::DEBUG::2012-12-02 23:40:08,234::task::568::TaskManager.Task::(_updateState) Task=`5e6c4482-2b5b-4575-9948-a8d6c9f3143d`::moving from state aborting -> state failed >Thread-1292::DEBUG::2012-12-02 23:40:08,234::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1292::DEBUG::2012-12-02 23:40:08,234::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1292::ERROR::2012-12-02 23:40:08,235::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1293::DEBUG::2012-12-02 23:40:08,376::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1293::DEBUG::2012-12-02 23:40:08,376::task::568::TaskManager.Task::(_updateState) Task=`78ab9d5e-ec05-4d15-aa1d-e72fb6555340`::moving from state init -> state preparing >Thread-1293::INFO::2012-12-02 23:40:08,376::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1293::DEBUG::2012-12-02 23:40:08,376::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1293::DEBUG::2012-12-02 23:40:08,377::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1293::INFO::2012-12-02 23:40:08,377::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1293::DEBUG::2012-12-02 23:40:08,377::task::1151::TaskManager.Task::(prepare) Task=`78ab9d5e-ec05-4d15-aa1d-e72fb6555340`::finished: {'allTasksStatus': {}} >Thread-1293::DEBUG::2012-12-02 23:40:08,377::task::568::TaskManager.Task::(_updateState) Task=`78ab9d5e-ec05-4d15-aa1d-e72fb6555340`::moving from state preparing -> state finished >Thread-1293::DEBUG::2012-12-02 23:40:08,377::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1293::DEBUG::2012-12-02 23:40:08,377::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1293::DEBUG::2012-12-02 23:40:08,377::task::957::TaskManager.Task::(_decref) Task=`78ab9d5e-ec05-4d15-aa1d-e72fb6555340`::ref 0 aborting False >Thread-1294::DEBUG::2012-12-02 23:40:08,518::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1294::DEBUG::2012-12-02 23:40:08,518::task::568::TaskManager.Task::(_updateState) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::moving from state init -> state preparing >Thread-1294::INFO::2012-12-02 23:40:08,518::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1294::ERROR::2012-12-02 23:40:08,518::task::833::TaskManager.Task::(_setError) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1294::DEBUG::2012-12-02 23:40:08,519::task::852::TaskManager.Task::(_run) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::Task._run: f412a473-9a57-45da-b00e-b6db8504a6b9 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1294::DEBUG::2012-12-02 23:40:08,519::task::1177::TaskManager.Task::(stop) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::stopping in state preparing (force False) >Thread-1294::DEBUG::2012-12-02 23:40:08,519::task::957::TaskManager.Task::(_decref) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::ref 1 aborting True >Thread-1294::INFO::2012-12-02 23:40:08,519::task::1134::TaskManager.Task::(prepare) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1294::DEBUG::2012-12-02 23:40:08,519::task::1139::TaskManager.Task::(prepare) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1294::DEBUG::2012-12-02 23:40:08,519::task::957::TaskManager.Task::(_decref) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::ref 0 aborting True >Thread-1294::DEBUG::2012-12-02 23:40:08,519::task::892::TaskManager.Task::(_doAbort) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::Task._doAbort: force False >Thread-1294::DEBUG::2012-12-02 23:40:08,520::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1294::DEBUG::2012-12-02 23:40:08,520::task::568::TaskManager.Task::(_updateState) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::moving from state preparing -> state aborting >Thread-1294::DEBUG::2012-12-02 23:40:08,520::task::523::TaskManager.Task::(__state_aborting) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::_aborting: recover policy none >Thread-1294::DEBUG::2012-12-02 23:40:08,520::task::568::TaskManager.Task::(_updateState) Task=`f412a473-9a57-45da-b00e-b6db8504a6b9`::moving from state aborting -> state failed >Thread-1294::DEBUG::2012-12-02 23:40:08,520::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1294::DEBUG::2012-12-02 23:40:08,520::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1294::ERROR::2012-12-02 23:40:08,520::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1296::DEBUG::2012-12-02 23:40:08,712::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1296::DEBUG::2012-12-02 23:40:08,712::task::568::TaskManager.Task::(_updateState) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::moving from state init -> state preparing >Thread-1296::INFO::2012-12-02 23:40:08,712::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1296::DEBUG::2012-12-02 23:40:08,713::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c2a28e17-e9ea-44fc-ae02-49d95ffd4481`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1296::DEBUG::2012-12-02 23:40:08,713::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1296::DEBUG::2012-12-02 23:40:08,713::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1296::DEBUG::2012-12-02 23:40:08,713::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`c2a28e17-e9ea-44fc-ae02-49d95ffd4481`::Granted request >Thread-1296::INFO::2012-12-02 23:40:08,713::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1296::DEBUG::2012-12-02 23:40:08,714::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:08,714::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:08,714::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:08,714::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:08,714::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:08,715::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:08,715::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1296::DEBUG::2012-12-02 23:40:08,715::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1296::DEBUG::2012-12-02 23:40:08,715::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1296::DEBUG::2012-12-02 23:40:08,715::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1296::DEBUG::2012-12-02 23:40:08,715::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1296::DEBUG::2012-12-02 23:40:08,742::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1296::DEBUG::2012-12-02 23:40:08,742::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1296::DEBUG::2012-12-02 23:40:09,375::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1296::DEBUG::2012-12-02 23:40:09,456::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1296::DEBUG::2012-12-02 23:40:09,457::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,457::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,457::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,457::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,457::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,457::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,458::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1296::DEBUG::2012-12-02 23:40:09,458::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,459::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1296::DEBUG::2012-12-02 23:40:09,670::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1296::WARNING::2012-12-02 23:40:09,672::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1296::DEBUG::2012-12-02 23:40:09,673::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1296::DEBUG::2012-12-02 23:40:09,702::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1296::DEBUG::2012-12-02 23:40:09,702::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1296::DEBUG::2012-12-02 23:40:09,702::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1296::DEBUG::2012-12-02 23:40:09,703::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1296::ERROR::2012-12-02 23:40:09,703::task::833::TaskManager.Task::(_setError) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1296::DEBUG::2012-12-02 23:40:09,703::task::852::TaskManager.Task::(_run) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::Task._run: 0feba324-02b1-48e4-9a60-3437c185bc10 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1296::DEBUG::2012-12-02 23:40:09,703::task::1177::TaskManager.Task::(stop) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::stopping in state preparing (force False) >Thread-1296::DEBUG::2012-12-02 23:40:09,703::task::957::TaskManager.Task::(_decref) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::ref 1 aborting True >Thread-1296::INFO::2012-12-02 23:40:09,704::task::1134::TaskManager.Task::(prepare) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1296::DEBUG::2012-12-02 23:40:09,704::task::1139::TaskManager.Task::(prepare) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::Prepare: aborted: Cannot find master domain >Thread-1296::DEBUG::2012-12-02 23:40:09,704::task::957::TaskManager.Task::(_decref) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::ref 0 aborting True >Thread-1296::DEBUG::2012-12-02 23:40:09,704::task::892::TaskManager.Task::(_doAbort) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::Task._doAbort: force False >Thread-1296::DEBUG::2012-12-02 23:40:09,704::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1296::DEBUG::2012-12-02 23:40:09,704::task::568::TaskManager.Task::(_updateState) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::moving from state preparing -> state aborting >Thread-1296::DEBUG::2012-12-02 23:40:09,704::task::523::TaskManager.Task::(__state_aborting) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::_aborting: recover policy none >Thread-1296::DEBUG::2012-12-02 23:40:09,704::task::568::TaskManager.Task::(_updateState) Task=`0feba324-02b1-48e4-9a60-3437c185bc10`::moving from state aborting -> state failed >Thread-1296::DEBUG::2012-12-02 23:40:09,705::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1296::DEBUG::2012-12-02 23:40:09,705::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1296::ERROR::2012-12-02 23:40:09,705::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1359::DEBUG::2012-12-02 23:40:12,841::task::568::TaskManager.Task::(_updateState) Task=`cd4f2af7-a5f3-457d-a101-868634b806f2`::moving from state init -> state preparing >Thread-1359::INFO::2012-12-02 23:40:12,841::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1359::INFO::2012-12-02 23:40:12,841::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1359::DEBUG::2012-12-02 23:40:12,841::task::1151::TaskManager.Task::(prepare) Task=`cd4f2af7-a5f3-457d-a101-868634b806f2`::finished: {} >Thread-1359::DEBUG::2012-12-02 23:40:12,841::task::568::TaskManager.Task::(_updateState) Task=`cd4f2af7-a5f3-457d-a101-868634b806f2`::moving from state preparing -> state finished >Thread-1359::DEBUG::2012-12-02 23:40:12,841::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1359::DEBUG::2012-12-02 23:40:12,841::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1359::DEBUG::2012-12-02 23:40:12,842::task::957::TaskManager.Task::(_decref) Task=`cd4f2af7-a5f3-457d-a101-868634b806f2`::ref 0 aborting False >Thread-1364::DEBUG::2012-12-02 23:40:20,039::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1364::DEBUG::2012-12-02 23:40:20,039::task::568::TaskManager.Task::(_updateState) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::moving from state init -> state preparing >Thread-1364::INFO::2012-12-02 23:40:20,040::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1364::ERROR::2012-12-02 23:40:20,040::task::833::TaskManager.Task::(_setError) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1364::DEBUG::2012-12-02 23:40:20,040::task::852::TaskManager.Task::(_run) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::Task._run: 9cb03754-0a06-4434-ae64-51b0df4e482f ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1364::DEBUG::2012-12-02 23:40:20,040::task::1177::TaskManager.Task::(stop) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::stopping in state preparing (force False) >Thread-1364::DEBUG::2012-12-02 23:40:20,040::task::957::TaskManager.Task::(_decref) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::ref 1 aborting True >Thread-1364::INFO::2012-12-02 23:40:20,040::task::1134::TaskManager.Task::(prepare) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1364::DEBUG::2012-12-02 23:40:20,041::task::1139::TaskManager.Task::(prepare) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1364::DEBUG::2012-12-02 23:40:20,041::task::957::TaskManager.Task::(_decref) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::ref 0 aborting True >Thread-1364::DEBUG::2012-12-02 23:40:20,041::task::892::TaskManager.Task::(_doAbort) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::Task._doAbort: force False >Thread-1364::DEBUG::2012-12-02 23:40:20,041::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1364::DEBUG::2012-12-02 23:40:20,041::task::568::TaskManager.Task::(_updateState) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::moving from state preparing -> state aborting >Thread-1364::DEBUG::2012-12-02 23:40:20,041::task::523::TaskManager.Task::(__state_aborting) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::_aborting: recover policy none >Thread-1364::DEBUG::2012-12-02 23:40:20,041::task::568::TaskManager.Task::(_updateState) Task=`9cb03754-0a06-4434-ae64-51b0df4e482f`::moving from state aborting -> state failed >Thread-1364::DEBUG::2012-12-02 23:40:20,041::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1364::DEBUG::2012-12-02 23:40:20,042::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1364::ERROR::2012-12-02 23:40:20,042::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1365::DEBUG::2012-12-02 23:40:20,184::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1365::DEBUG::2012-12-02 23:40:20,184::task::568::TaskManager.Task::(_updateState) Task=`d09c23c2-00dc-490f-b62c-0a23c3f4321c`::moving from state init -> state preparing >Thread-1365::INFO::2012-12-02 23:40:20,184::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1365::DEBUG::2012-12-02 23:40:20,184::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1365::DEBUG::2012-12-02 23:40:20,185::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1365::INFO::2012-12-02 23:40:20,185::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1365::DEBUG::2012-12-02 23:40:20,185::task::1151::TaskManager.Task::(prepare) Task=`d09c23c2-00dc-490f-b62c-0a23c3f4321c`::finished: {'allTasksStatus': {}} >Thread-1365::DEBUG::2012-12-02 23:40:20,185::task::568::TaskManager.Task::(_updateState) Task=`d09c23c2-00dc-490f-b62c-0a23c3f4321c`::moving from state preparing -> state finished >Thread-1365::DEBUG::2012-12-02 23:40:20,185::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1365::DEBUG::2012-12-02 23:40:20,185::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1365::DEBUG::2012-12-02 23:40:20,185::task::957::TaskManager.Task::(_decref) Task=`d09c23c2-00dc-490f-b62c-0a23c3f4321c`::ref 0 aborting False >Thread-1366::DEBUG::2012-12-02 23:40:20,327::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1366::DEBUG::2012-12-02 23:40:20,328::task::568::TaskManager.Task::(_updateState) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::moving from state init -> state preparing >Thread-1366::INFO::2012-12-02 23:40:20,328::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1366::ERROR::2012-12-02 23:40:20,328::task::833::TaskManager.Task::(_setError) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1366::DEBUG::2012-12-02 23:40:20,328::task::852::TaskManager.Task::(_run) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::Task._run: be952f40-49f6-4064-a3aa-1ef52884e7a6 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1366::DEBUG::2012-12-02 23:40:20,328::task::1177::TaskManager.Task::(stop) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::stopping in state preparing (force False) >Thread-1366::DEBUG::2012-12-02 23:40:20,329::task::957::TaskManager.Task::(_decref) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::ref 1 aborting True >Thread-1366::INFO::2012-12-02 23:40:20,329::task::1134::TaskManager.Task::(prepare) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1366::DEBUG::2012-12-02 23:40:20,329::task::1139::TaskManager.Task::(prepare) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1366::DEBUG::2012-12-02 23:40:20,329::task::957::TaskManager.Task::(_decref) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::ref 0 aborting True >Thread-1366::DEBUG::2012-12-02 23:40:20,329::task::892::TaskManager.Task::(_doAbort) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::Task._doAbort: force False >Thread-1366::DEBUG::2012-12-02 23:40:20,329::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1366::DEBUG::2012-12-02 23:40:20,329::task::568::TaskManager.Task::(_updateState) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::moving from state preparing -> state aborting >Thread-1366::DEBUG::2012-12-02 23:40:20,329::task::523::TaskManager.Task::(__state_aborting) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::_aborting: recover policy none >Thread-1366::DEBUG::2012-12-02 23:40:20,330::task::568::TaskManager.Task::(_updateState) Task=`be952f40-49f6-4064-a3aa-1ef52884e7a6`::moving from state aborting -> state failed >Thread-1366::DEBUG::2012-12-02 23:40:20,330::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1366::DEBUG::2012-12-02 23:40:20,330::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1366::ERROR::2012-12-02 23:40:20,330::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1367::DEBUG::2012-12-02 23:40:20,510::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1367::DEBUG::2012-12-02 23:40:20,510::task::568::TaskManager.Task::(_updateState) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::moving from state init -> state preparing >Thread-1367::INFO::2012-12-02 23:40:20,510::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1367::DEBUG::2012-12-02 23:40:20,510::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`67fc2b01-820a-4dd7-a0b0-bccbb8682494`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1367::DEBUG::2012-12-02 23:40:20,511::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1367::DEBUG::2012-12-02 23:40:20,511::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1367::DEBUG::2012-12-02 23:40:20,511::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`67fc2b01-820a-4dd7-a0b0-bccbb8682494`::Granted request >Thread-1367::INFO::2012-12-02 23:40:20,511::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1367::DEBUG::2012-12-02 23:40:20,512::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:20,512::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:20,512::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:20,512::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:20,512::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:20,512::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:20,512::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1367::DEBUG::2012-12-02 23:40:20,513::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1367::DEBUG::2012-12-02 23:40:20,513::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1367::DEBUG::2012-12-02 23:40:20,513::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1367::DEBUG::2012-12-02 23:40:20,513::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1367::DEBUG::2012-12-02 23:40:20,537::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1367::DEBUG::2012-12-02 23:40:20,537::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1367::DEBUG::2012-12-02 23:40:21,170::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1367::DEBUG::2012-12-02 23:40:21,266::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1367::DEBUG::2012-12-02 23:40:21,267::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,267::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,267::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,267::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,267::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,268::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,268::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1367::DEBUG::2012-12-02 23:40:21,268::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,270::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1367::DEBUG::2012-12-02 23:40:21,474::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1367::WARNING::2012-12-02 23:40:21,475::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1367::DEBUG::2012-12-02 23:40:21,476::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1367::DEBUG::2012-12-02 23:40:21,506::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1367::DEBUG::2012-12-02 23:40:21,506::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1367::DEBUG::2012-12-02 23:40:21,506::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1367::DEBUG::2012-12-02 23:40:21,506::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1367::ERROR::2012-12-02 23:40:21,507::task::833::TaskManager.Task::(_setError) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1367::DEBUG::2012-12-02 23:40:21,507::task::852::TaskManager.Task::(_run) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::Task._run: 8f876746-b5f4-48c3-9a2e-15bcf9f54016 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1367::DEBUG::2012-12-02 23:40:21,507::task::1177::TaskManager.Task::(stop) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::stopping in state preparing (force False) >Thread-1367::DEBUG::2012-12-02 23:40:21,507::task::957::TaskManager.Task::(_decref) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::ref 1 aborting True >Thread-1367::INFO::2012-12-02 23:40:21,507::task::1134::TaskManager.Task::(prepare) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1367::DEBUG::2012-12-02 23:40:21,508::task::1139::TaskManager.Task::(prepare) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::Prepare: aborted: Cannot find master domain >Thread-1367::DEBUG::2012-12-02 23:40:21,508::task::957::TaskManager.Task::(_decref) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::ref 0 aborting True >Thread-1367::DEBUG::2012-12-02 23:40:21,508::task::892::TaskManager.Task::(_doAbort) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::Task._doAbort: force False >Thread-1367::DEBUG::2012-12-02 23:40:21,508::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1367::DEBUG::2012-12-02 23:40:21,508::task::568::TaskManager.Task::(_updateState) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::moving from state preparing -> state aborting >Thread-1367::DEBUG::2012-12-02 23:40:21,508::task::523::TaskManager.Task::(__state_aborting) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::_aborting: recover policy none >Thread-1367::DEBUG::2012-12-02 23:40:21,508::task::568::TaskManager.Task::(_updateState) Task=`8f876746-b5f4-48c3-9a2e-15bcf9f54016`::moving from state aborting -> state failed >Thread-1367::DEBUG::2012-12-02 23:40:21,508::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1367::DEBUG::2012-12-02 23:40:21,509::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1367::ERROR::2012-12-02 23:40:21,509::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1430::DEBUG::2012-12-02 23:40:23,860::task::568::TaskManager.Task::(_updateState) Task=`c43f4a45-6828-484a-9f2e-0cb6525d3b24`::moving from state init -> state preparing >Thread-1430::INFO::2012-12-02 23:40:23,860::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1430::INFO::2012-12-02 23:40:23,861::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1430::DEBUG::2012-12-02 23:40:23,861::task::1151::TaskManager.Task::(prepare) Task=`c43f4a45-6828-484a-9f2e-0cb6525d3b24`::finished: {} >Thread-1430::DEBUG::2012-12-02 23:40:23,861::task::568::TaskManager.Task::(_updateState) Task=`c43f4a45-6828-484a-9f2e-0cb6525d3b24`::moving from state preparing -> state finished >Thread-1430::DEBUG::2012-12-02 23:40:23,861::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1430::DEBUG::2012-12-02 23:40:23,861::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1430::DEBUG::2012-12-02 23:40:23,861::task::957::TaskManager.Task::(_decref) Task=`c43f4a45-6828-484a-9f2e-0cb6525d3b24`::ref 0 aborting False >Thread-1435::DEBUG::2012-12-02 23:40:31,835::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1435::DEBUG::2012-12-02 23:40:31,835::task::568::TaskManager.Task::(_updateState) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::moving from state init -> state preparing >Thread-1435::INFO::2012-12-02 23:40:31,835::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1435::ERROR::2012-12-02 23:40:31,835::task::833::TaskManager.Task::(_setError) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1435::DEBUG::2012-12-02 23:40:31,835::task::852::TaskManager.Task::(_run) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::Task._run: f8b94989-0657-45e4-bb3b-d2eb16b99051 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1435::DEBUG::2012-12-02 23:40:31,836::task::1177::TaskManager.Task::(stop) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::stopping in state preparing (force False) >Thread-1435::DEBUG::2012-12-02 23:40:31,836::task::957::TaskManager.Task::(_decref) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::ref 1 aborting True >Thread-1435::INFO::2012-12-02 23:40:31,836::task::1134::TaskManager.Task::(prepare) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1435::DEBUG::2012-12-02 23:40:31,836::task::1139::TaskManager.Task::(prepare) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1435::DEBUG::2012-12-02 23:40:31,836::task::957::TaskManager.Task::(_decref) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::ref 0 aborting True >Thread-1435::DEBUG::2012-12-02 23:40:31,836::task::892::TaskManager.Task::(_doAbort) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::Task._doAbort: force False >Thread-1435::DEBUG::2012-12-02 23:40:31,836::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1435::DEBUG::2012-12-02 23:40:31,837::task::568::TaskManager.Task::(_updateState) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::moving from state preparing -> state aborting >Thread-1435::DEBUG::2012-12-02 23:40:31,837::task::523::TaskManager.Task::(__state_aborting) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::_aborting: recover policy none >Thread-1435::DEBUG::2012-12-02 23:40:31,837::task::568::TaskManager.Task::(_updateState) Task=`f8b94989-0657-45e4-bb3b-d2eb16b99051`::moving from state aborting -> state failed >Thread-1435::DEBUG::2012-12-02 23:40:31,837::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1435::DEBUG::2012-12-02 23:40:31,837::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1435::ERROR::2012-12-02 23:40:31,837::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1436::DEBUG::2012-12-02 23:40:32,003::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1436::DEBUG::2012-12-02 23:40:32,004::task::568::TaskManager.Task::(_updateState) Task=`470c3216-4507-4594-b421-c57f49e89d04`::moving from state init -> state preparing >Thread-1436::INFO::2012-12-02 23:40:32,004::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1436::DEBUG::2012-12-02 23:40:32,004::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1436::DEBUG::2012-12-02 23:40:32,004::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1436::INFO::2012-12-02 23:40:32,004::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1436::DEBUG::2012-12-02 23:40:32,005::task::1151::TaskManager.Task::(prepare) Task=`470c3216-4507-4594-b421-c57f49e89d04`::finished: {'allTasksStatus': {}} >Thread-1436::DEBUG::2012-12-02 23:40:32,005::task::568::TaskManager.Task::(_updateState) Task=`470c3216-4507-4594-b421-c57f49e89d04`::moving from state preparing -> state finished >Thread-1436::DEBUG::2012-12-02 23:40:32,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1436::DEBUG::2012-12-02 23:40:32,005::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1436::DEBUG::2012-12-02 23:40:32,005::task::957::TaskManager.Task::(_decref) Task=`470c3216-4507-4594-b421-c57f49e89d04`::ref 0 aborting False >Thread-1437::DEBUG::2012-12-02 23:40:32,179::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1437::DEBUG::2012-12-02 23:40:32,179::task::568::TaskManager.Task::(_updateState) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::moving from state init -> state preparing >Thread-1437::INFO::2012-12-02 23:40:32,179::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1437::ERROR::2012-12-02 23:40:32,179::task::833::TaskManager.Task::(_setError) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1437::DEBUG::2012-12-02 23:40:32,179::task::852::TaskManager.Task::(_run) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::Task._run: 43645c4b-bd6e-43ac-b250-8d31abad9022 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1437::DEBUG::2012-12-02 23:40:32,179::task::1177::TaskManager.Task::(stop) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::stopping in state preparing (force False) >Thread-1437::DEBUG::2012-12-02 23:40:32,179::task::957::TaskManager.Task::(_decref) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::ref 1 aborting True >Thread-1437::INFO::2012-12-02 23:40:32,180::task::1134::TaskManager.Task::(prepare) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1437::DEBUG::2012-12-02 23:40:32,180::task::1139::TaskManager.Task::(prepare) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1437::DEBUG::2012-12-02 23:40:32,180::task::957::TaskManager.Task::(_decref) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::ref 0 aborting True >Thread-1437::DEBUG::2012-12-02 23:40:32,180::task::892::TaskManager.Task::(_doAbort) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::Task._doAbort: force False >Thread-1437::DEBUG::2012-12-02 23:40:32,180::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1437::DEBUG::2012-12-02 23:40:32,180::task::568::TaskManager.Task::(_updateState) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::moving from state preparing -> state aborting >Thread-1437::DEBUG::2012-12-02 23:40:32,180::task::523::TaskManager.Task::(__state_aborting) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::_aborting: recover policy none >Thread-1437::DEBUG::2012-12-02 23:40:32,180::task::568::TaskManager.Task::(_updateState) Task=`43645c4b-bd6e-43ac-b250-8d31abad9022`::moving from state aborting -> state failed >Thread-1437::DEBUG::2012-12-02 23:40:32,181::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1437::DEBUG::2012-12-02 23:40:32,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1437::ERROR::2012-12-02 23:40:32,181::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1438::DEBUG::2012-12-02 23:40:32,402::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1438::DEBUG::2012-12-02 23:40:32,402::task::568::TaskManager.Task::(_updateState) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::moving from state init -> state preparing >Thread-1438::INFO::2012-12-02 23:40:32,403::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1438::DEBUG::2012-12-02 23:40:32,403::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`849e3aac-f297-4ec2-a32e-9bf17b950a80`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1438::DEBUG::2012-12-02 23:40:32,403::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1438::DEBUG::2012-12-02 23:40:32,403::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1438::DEBUG::2012-12-02 23:40:32,403::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`849e3aac-f297-4ec2-a32e-9bf17b950a80`::Granted request >Thread-1438::INFO::2012-12-02 23:40:32,404::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1438::DEBUG::2012-12-02 23:40:32,404::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:32,404::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:32,404::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:32,405::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:32,405::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:32,405::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:32,405::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1438::DEBUG::2012-12-02 23:40:32,405::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1438::DEBUG::2012-12-02 23:40:32,405::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1438::DEBUG::2012-12-02 23:40:32,405::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1438::DEBUG::2012-12-02 23:40:32,405::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1438::DEBUG::2012-12-02 23:40:32,434::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1438::DEBUG::2012-12-02 23:40:32,434::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1438::DEBUG::2012-12-02 23:40:33,067::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1438::DEBUG::2012-12-02 23:40:33,143::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1438::DEBUG::2012-12-02 23:40:33,144::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,144::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,144::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,144::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,145::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,145::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,145::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1438::DEBUG::2012-12-02 23:40:33,145::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,146::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1438::DEBUG::2012-12-02 23:40:33,352::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1438::WARNING::2012-12-02 23:40:33,354::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1438::DEBUG::2012-12-02 23:40:33,354::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1438::DEBUG::2012-12-02 23:40:33,384::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1438::DEBUG::2012-12-02 23:40:33,384::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1438::DEBUG::2012-12-02 23:40:33,384::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1438::DEBUG::2012-12-02 23:40:33,384::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1438::ERROR::2012-12-02 23:40:33,384::task::833::TaskManager.Task::(_setError) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1438::DEBUG::2012-12-02 23:40:33,385::task::852::TaskManager.Task::(_run) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::Task._run: 7556d027-8f44-48de-bdd8-ec1d056e0727 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1438::DEBUG::2012-12-02 23:40:33,385::task::1177::TaskManager.Task::(stop) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::stopping in state preparing (force False) >Thread-1438::DEBUG::2012-12-02 23:40:33,385::task::957::TaskManager.Task::(_decref) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::ref 1 aborting True >Thread-1438::INFO::2012-12-02 23:40:33,385::task::1134::TaskManager.Task::(prepare) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1438::DEBUG::2012-12-02 23:40:33,385::task::1139::TaskManager.Task::(prepare) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::Prepare: aborted: Cannot find master domain >Thread-1438::DEBUG::2012-12-02 23:40:33,385::task::957::TaskManager.Task::(_decref) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::ref 0 aborting True >Thread-1438::DEBUG::2012-12-02 23:40:33,386::task::892::TaskManager.Task::(_doAbort) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::Task._doAbort: force False >Thread-1438::DEBUG::2012-12-02 23:40:33,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1438::DEBUG::2012-12-02 23:40:33,386::task::568::TaskManager.Task::(_updateState) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::moving from state preparing -> state aborting >Thread-1438::DEBUG::2012-12-02 23:40:33,386::task::523::TaskManager.Task::(__state_aborting) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::_aborting: recover policy none >Thread-1438::DEBUG::2012-12-02 23:40:33,386::task::568::TaskManager.Task::(_updateState) Task=`7556d027-8f44-48de-bdd8-ec1d056e0727`::moving from state aborting -> state failed >Thread-1438::DEBUG::2012-12-02 23:40:33,386::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1438::DEBUG::2012-12-02 23:40:33,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1438::ERROR::2012-12-02 23:40:33,386::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1501::DEBUG::2012-12-02 23:40:34,882::task::568::TaskManager.Task::(_updateState) Task=`d99a1d63-a3c7-402b-960e-0550e92115b1`::moving from state init -> state preparing >Thread-1501::INFO::2012-12-02 23:40:34,882::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1501::INFO::2012-12-02 23:40:34,882::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1501::DEBUG::2012-12-02 23:40:34,882::task::1151::TaskManager.Task::(prepare) Task=`d99a1d63-a3c7-402b-960e-0550e92115b1`::finished: {} >Thread-1501::DEBUG::2012-12-02 23:40:34,883::task::568::TaskManager.Task::(_updateState) Task=`d99a1d63-a3c7-402b-960e-0550e92115b1`::moving from state preparing -> state finished >Thread-1501::DEBUG::2012-12-02 23:40:34,883::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1501::DEBUG::2012-12-02 23:40:34,883::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1501::DEBUG::2012-12-02 23:40:34,883::task::957::TaskManager.Task::(_decref) Task=`d99a1d63-a3c7-402b-960e-0550e92115b1`::ref 0 aborting False >Thread-1507::DEBUG::2012-12-02 23:40:43,709::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1507::DEBUG::2012-12-02 23:40:43,709::task::568::TaskManager.Task::(_updateState) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::moving from state init -> state preparing >Thread-1507::INFO::2012-12-02 23:40:43,709::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1507::ERROR::2012-12-02 23:40:43,709::task::833::TaskManager.Task::(_setError) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1507::DEBUG::2012-12-02 23:40:43,710::task::852::TaskManager.Task::(_run) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::Task._run: bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1507::DEBUG::2012-12-02 23:40:43,710::task::1177::TaskManager.Task::(stop) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::stopping in state preparing (force False) >Thread-1507::DEBUG::2012-12-02 23:40:43,710::task::957::TaskManager.Task::(_decref) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::ref 1 aborting True >Thread-1507::INFO::2012-12-02 23:40:43,710::task::1134::TaskManager.Task::(prepare) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1507::DEBUG::2012-12-02 23:40:43,710::task::1139::TaskManager.Task::(prepare) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1507::DEBUG::2012-12-02 23:40:43,710::task::957::TaskManager.Task::(_decref) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::ref 0 aborting True >Thread-1507::DEBUG::2012-12-02 23:40:43,710::task::892::TaskManager.Task::(_doAbort) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::Task._doAbort: force False >Thread-1507::DEBUG::2012-12-02 23:40:43,710::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1507::DEBUG::2012-12-02 23:40:43,711::task::568::TaskManager.Task::(_updateState) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::moving from state preparing -> state aborting >Thread-1507::DEBUG::2012-12-02 23:40:43,711::task::523::TaskManager.Task::(__state_aborting) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::_aborting: recover policy none >Thread-1507::DEBUG::2012-12-02 23:40:43,711::task::568::TaskManager.Task::(_updateState) Task=`bf9ecf0b-ffcf-4d3c-9ce8-53567c067d2a`::moving from state aborting -> state failed >Thread-1507::DEBUG::2012-12-02 23:40:43,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1507::DEBUG::2012-12-02 23:40:43,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1507::ERROR::2012-12-02 23:40:43,711::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1508::DEBUG::2012-12-02 23:40:43,849::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1508::DEBUG::2012-12-02 23:40:43,849::task::568::TaskManager.Task::(_updateState) Task=`be97335b-c50d-4076-8230-b3e1f2e0a531`::moving from state init -> state preparing >Thread-1508::INFO::2012-12-02 23:40:43,850::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1508::DEBUG::2012-12-02 23:40:43,850::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1508::DEBUG::2012-12-02 23:40:43,850::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1508::INFO::2012-12-02 23:40:43,850::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1508::DEBUG::2012-12-02 23:40:43,850::task::1151::TaskManager.Task::(prepare) Task=`be97335b-c50d-4076-8230-b3e1f2e0a531`::finished: {'allTasksStatus': {}} >Thread-1508::DEBUG::2012-12-02 23:40:43,850::task::568::TaskManager.Task::(_updateState) Task=`be97335b-c50d-4076-8230-b3e1f2e0a531`::moving from state preparing -> state finished >Thread-1508::DEBUG::2012-12-02 23:40:43,850::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1508::DEBUG::2012-12-02 23:40:43,850::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1508::DEBUG::2012-12-02 23:40:43,851::task::957::TaskManager.Task::(_decref) Task=`be97335b-c50d-4076-8230-b3e1f2e0a531`::ref 0 aborting False >Thread-1509::DEBUG::2012-12-02 23:40:43,997::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1509::DEBUG::2012-12-02 23:40:43,997::task::568::TaskManager.Task::(_updateState) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::moving from state init -> state preparing >Thread-1509::INFO::2012-12-02 23:40:43,997::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1509::ERROR::2012-12-02 23:40:43,998::task::833::TaskManager.Task::(_setError) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1509::DEBUG::2012-12-02 23:40:43,998::task::852::TaskManager.Task::(_run) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::Task._run: c2195878-f4b3-43cb-abbe-9da1ad7dd410 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1509::DEBUG::2012-12-02 23:40:43,998::task::1177::TaskManager.Task::(stop) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::stopping in state preparing (force False) >Thread-1509::DEBUG::2012-12-02 23:40:43,998::task::957::TaskManager.Task::(_decref) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::ref 1 aborting True >Thread-1509::INFO::2012-12-02 23:40:43,998::task::1134::TaskManager.Task::(prepare) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1509::DEBUG::2012-12-02 23:40:43,998::task::1139::TaskManager.Task::(prepare) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1509::DEBUG::2012-12-02 23:40:43,998::task::957::TaskManager.Task::(_decref) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::ref 0 aborting True >Thread-1509::DEBUG::2012-12-02 23:40:43,999::task::892::TaskManager.Task::(_doAbort) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::Task._doAbort: force False >Thread-1509::DEBUG::2012-12-02 23:40:43,999::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1509::DEBUG::2012-12-02 23:40:43,999::task::568::TaskManager.Task::(_updateState) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::moving from state preparing -> state aborting >Thread-1509::DEBUG::2012-12-02 23:40:43,999::task::523::TaskManager.Task::(__state_aborting) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::_aborting: recover policy none >Thread-1509::DEBUG::2012-12-02 23:40:43,999::task::568::TaskManager.Task::(_updateState) Task=`c2195878-f4b3-43cb-abbe-9da1ad7dd410`::moving from state aborting -> state failed >Thread-1509::DEBUG::2012-12-02 23:40:43,999::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1509::DEBUG::2012-12-02 23:40:43,999::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1509::ERROR::2012-12-02 23:40:43,999::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1510::DEBUG::2012-12-02 23:40:44,184::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1510::DEBUG::2012-12-02 23:40:44,184::task::568::TaskManager.Task::(_updateState) Task=`9672975a-782e-4059-9060-ca70acb4b344`::moving from state init -> state preparing >Thread-1510::INFO::2012-12-02 23:40:44,184::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1510::DEBUG::2012-12-02 23:40:44,185::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6f5c10c5-c9d3-41f6-bda7-9008e6ebe839`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1510::DEBUG::2012-12-02 23:40:44,185::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1510::DEBUG::2012-12-02 23:40:44,185::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1510::DEBUG::2012-12-02 23:40:44,185::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6f5c10c5-c9d3-41f6-bda7-9008e6ebe839`::Granted request >Thread-1510::INFO::2012-12-02 23:40:44,185::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1510::DEBUG::2012-12-02 23:40:44,186::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,186::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,186::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,186::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,186::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,186::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,186::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1510::DEBUG::2012-12-02 23:40:44,186::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1510::DEBUG::2012-12-02 23:40:44,186::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1510::DEBUG::2012-12-02 23:40:44,187::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1510::DEBUG::2012-12-02 23:40:44,187::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1510::DEBUG::2012-12-02 23:40:44,211::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1510::DEBUG::2012-12-02 23:40:44,211::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1510::DEBUG::2012-12-02 23:40:44,844::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1510::DEBUG::2012-12-02 23:40:44,914::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1510::DEBUG::2012-12-02 23:40:44,914::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,914::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,914::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,915::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,915::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,915::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,915::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1510::DEBUG::2012-12-02 23:40:44,915::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:44,917::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1510::DEBUG::2012-12-02 23:40:45,122::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1510::WARNING::2012-12-02 23:40:45,123::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1510::DEBUG::2012-12-02 23:40:45,124::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1510::DEBUG::2012-12-02 23:40:45,153::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1510::DEBUG::2012-12-02 23:40:45,153::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1510::DEBUG::2012-12-02 23:40:45,153::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1510::DEBUG::2012-12-02 23:40:45,153::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1510::ERROR::2012-12-02 23:40:45,154::task::833::TaskManager.Task::(_setError) Task=`9672975a-782e-4059-9060-ca70acb4b344`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1510::DEBUG::2012-12-02 23:40:45,154::task::852::TaskManager.Task::(_run) Task=`9672975a-782e-4059-9060-ca70acb4b344`::Task._run: 9672975a-782e-4059-9060-ca70acb4b344 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1510::DEBUG::2012-12-02 23:40:45,154::task::1177::TaskManager.Task::(stop) Task=`9672975a-782e-4059-9060-ca70acb4b344`::stopping in state preparing (force False) >Thread-1510::DEBUG::2012-12-02 23:40:45,154::task::957::TaskManager.Task::(_decref) Task=`9672975a-782e-4059-9060-ca70acb4b344`::ref 1 aborting True >Thread-1510::INFO::2012-12-02 23:40:45,154::task::1134::TaskManager.Task::(prepare) Task=`9672975a-782e-4059-9060-ca70acb4b344`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1510::DEBUG::2012-12-02 23:40:45,155::task::1139::TaskManager.Task::(prepare) Task=`9672975a-782e-4059-9060-ca70acb4b344`::Prepare: aborted: Cannot find master domain >Thread-1510::DEBUG::2012-12-02 23:40:45,155::task::957::TaskManager.Task::(_decref) Task=`9672975a-782e-4059-9060-ca70acb4b344`::ref 0 aborting True >Thread-1510::DEBUG::2012-12-02 23:40:45,155::task::892::TaskManager.Task::(_doAbort) Task=`9672975a-782e-4059-9060-ca70acb4b344`::Task._doAbort: force False >Thread-1510::DEBUG::2012-12-02 23:40:45,155::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1510::DEBUG::2012-12-02 23:40:45,155::task::568::TaskManager.Task::(_updateState) Task=`9672975a-782e-4059-9060-ca70acb4b344`::moving from state preparing -> state aborting >Thread-1510::DEBUG::2012-12-02 23:40:45,155::task::523::TaskManager.Task::(__state_aborting) Task=`9672975a-782e-4059-9060-ca70acb4b344`::_aborting: recover policy none >Thread-1510::DEBUG::2012-12-02 23:40:45,155::task::568::TaskManager.Task::(_updateState) Task=`9672975a-782e-4059-9060-ca70acb4b344`::moving from state aborting -> state failed >Thread-1510::DEBUG::2012-12-02 23:40:45,155::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1510::DEBUG::2012-12-02 23:40:45,156::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1510::ERROR::2012-12-02 23:40:45,156::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1572::DEBUG::2012-12-02 23:40:45,851::task::568::TaskManager.Task::(_updateState) Task=`ab3096d0-0f4a-423c-b4f0-3466e119405d`::moving from state init -> state preparing >Thread-1572::INFO::2012-12-02 23:40:45,851::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1572::INFO::2012-12-02 23:40:45,851::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1572::DEBUG::2012-12-02 23:40:45,851::task::1151::TaskManager.Task::(prepare) Task=`ab3096d0-0f4a-423c-b4f0-3466e119405d`::finished: {} >Thread-1572::DEBUG::2012-12-02 23:40:45,852::task::568::TaskManager.Task::(_updateState) Task=`ab3096d0-0f4a-423c-b4f0-3466e119405d`::moving from state preparing -> state finished >Thread-1572::DEBUG::2012-12-02 23:40:45,852::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1572::DEBUG::2012-12-02 23:40:45,852::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1572::DEBUG::2012-12-02 23:40:45,852::task::957::TaskManager.Task::(_decref) Task=`ab3096d0-0f4a-423c-b4f0-3466e119405d`::ref 0 aborting False >Thread-1578::DEBUG::2012-12-02 23:40:55,476::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1578::DEBUG::2012-12-02 23:40:55,476::task::568::TaskManager.Task::(_updateState) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::moving from state init -> state preparing >Thread-1578::INFO::2012-12-02 23:40:55,477::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1578::ERROR::2012-12-02 23:40:55,477::task::833::TaskManager.Task::(_setError) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1578::DEBUG::2012-12-02 23:40:55,477::task::852::TaskManager.Task::(_run) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::Task._run: dd2646b2-6845-4aa3-90b8-7f325a719b23 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1578::DEBUG::2012-12-02 23:40:55,477::task::1177::TaskManager.Task::(stop) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::stopping in state preparing (force False) >Thread-1578::DEBUG::2012-12-02 23:40:55,477::task::957::TaskManager.Task::(_decref) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::ref 1 aborting True >Thread-1578::INFO::2012-12-02 23:40:55,477::task::1134::TaskManager.Task::(prepare) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1578::DEBUG::2012-12-02 23:40:55,477::task::1139::TaskManager.Task::(prepare) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1578::DEBUG::2012-12-02 23:40:55,478::task::957::TaskManager.Task::(_decref) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::ref 0 aborting True >Thread-1578::DEBUG::2012-12-02 23:40:55,478::task::892::TaskManager.Task::(_doAbort) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::Task._doAbort: force False >Thread-1578::DEBUG::2012-12-02 23:40:55,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1578::DEBUG::2012-12-02 23:40:55,478::task::568::TaskManager.Task::(_updateState) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::moving from state preparing -> state aborting >Thread-1578::DEBUG::2012-12-02 23:40:55,478::task::523::TaskManager.Task::(__state_aborting) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::_aborting: recover policy none >Thread-1578::DEBUG::2012-12-02 23:40:55,478::task::568::TaskManager.Task::(_updateState) Task=`dd2646b2-6845-4aa3-90b8-7f325a719b23`::moving from state aborting -> state failed >Thread-1578::DEBUG::2012-12-02 23:40:55,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1578::DEBUG::2012-12-02 23:40:55,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1578::ERROR::2012-12-02 23:40:55,479::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1579::DEBUG::2012-12-02 23:40:55,618::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1579::DEBUG::2012-12-02 23:40:55,619::task::568::TaskManager.Task::(_updateState) Task=`d84fcfca-80a8-4514-92d8-ac5bbde63bea`::moving from state init -> state preparing >Thread-1579::INFO::2012-12-02 23:40:55,619::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1579::DEBUG::2012-12-02 23:40:55,619::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1579::DEBUG::2012-12-02 23:40:55,619::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1579::INFO::2012-12-02 23:40:55,619::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1579::DEBUG::2012-12-02 23:40:55,619::task::1151::TaskManager.Task::(prepare) Task=`d84fcfca-80a8-4514-92d8-ac5bbde63bea`::finished: {'allTasksStatus': {}} >Thread-1579::DEBUG::2012-12-02 23:40:55,619::task::568::TaskManager.Task::(_updateState) Task=`d84fcfca-80a8-4514-92d8-ac5bbde63bea`::moving from state preparing -> state finished >Thread-1579::DEBUG::2012-12-02 23:40:55,620::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1579::DEBUG::2012-12-02 23:40:55,620::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1579::DEBUG::2012-12-02 23:40:55,620::task::957::TaskManager.Task::(_decref) Task=`d84fcfca-80a8-4514-92d8-ac5bbde63bea`::ref 0 aborting False >Thread-1580::DEBUG::2012-12-02 23:40:55,759::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1580::DEBUG::2012-12-02 23:40:55,759::task::568::TaskManager.Task::(_updateState) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::moving from state init -> state preparing >Thread-1580::INFO::2012-12-02 23:40:55,760::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1580::ERROR::2012-12-02 23:40:55,760::task::833::TaskManager.Task::(_setError) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1580::DEBUG::2012-12-02 23:40:55,760::task::852::TaskManager.Task::(_run) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::Task._run: cd67738f-9fc7-48c5-abd6-7ab823467b98 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1580::DEBUG::2012-12-02 23:40:55,760::task::1177::TaskManager.Task::(stop) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::stopping in state preparing (force False) >Thread-1580::DEBUG::2012-12-02 23:40:55,760::task::957::TaskManager.Task::(_decref) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::ref 1 aborting True >Thread-1580::INFO::2012-12-02 23:40:55,760::task::1134::TaskManager.Task::(prepare) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1580::DEBUG::2012-12-02 23:40:55,760::task::1139::TaskManager.Task::(prepare) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1580::DEBUG::2012-12-02 23:40:55,761::task::957::TaskManager.Task::(_decref) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::ref 0 aborting True >Thread-1580::DEBUG::2012-12-02 23:40:55,761::task::892::TaskManager.Task::(_doAbort) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::Task._doAbort: force False >Thread-1580::DEBUG::2012-12-02 23:40:55,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1580::DEBUG::2012-12-02 23:40:55,761::task::568::TaskManager.Task::(_updateState) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::moving from state preparing -> state aborting >Thread-1580::DEBUG::2012-12-02 23:40:55,761::task::523::TaskManager.Task::(__state_aborting) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::_aborting: recover policy none >Thread-1580::DEBUG::2012-12-02 23:40:55,761::task::568::TaskManager.Task::(_updateState) Task=`cd67738f-9fc7-48c5-abd6-7ab823467b98`::moving from state aborting -> state failed >Thread-1580::DEBUG::2012-12-02 23:40:55,761::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1580::DEBUG::2012-12-02 23:40:55,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1580::ERROR::2012-12-02 23:40:55,761::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1581::DEBUG::2012-12-02 23:40:55,944::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1581::DEBUG::2012-12-02 23:40:55,944::task::568::TaskManager.Task::(_updateState) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::moving from state init -> state preparing >Thread-1581::INFO::2012-12-02 23:40:55,944::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1581::DEBUG::2012-12-02 23:40:55,945::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9f020d6d-0b20-4d54-9e37-00907df1f018`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1581::DEBUG::2012-12-02 23:40:55,945::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1581::DEBUG::2012-12-02 23:40:55,945::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1581::DEBUG::2012-12-02 23:40:55,945::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9f020d6d-0b20-4d54-9e37-00907df1f018`::Granted request >Thread-1581::INFO::2012-12-02 23:40:55,945::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1581::DEBUG::2012-12-02 23:40:55,946::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:55,946::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:55,946::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:55,946::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:55,946::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:55,946::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:55,947::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1581::DEBUG::2012-12-02 23:40:55,947::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1581::DEBUG::2012-12-02 23:40:55,947::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1581::DEBUG::2012-12-02 23:40:55,947::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1581::DEBUG::2012-12-02 23:40:55,947::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1581::DEBUG::2012-12-02 23:40:55,971::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1581::DEBUG::2012-12-02 23:40:55,971::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1581::DEBUG::2012-12-02 23:40:56,636::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1581::DEBUG::2012-12-02 23:40:56,690::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1581::DEBUG::2012-12-02 23:40:56,690::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,691::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,691::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,691::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,691::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,691::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,691::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1581::DEBUG::2012-12-02 23:40:56,692::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,693::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1582::DEBUG::2012-12-02 23:40:56,770::task::568::TaskManager.Task::(_updateState) Task=`d297ace3-e73c-4219-9233-0fa94d6d350f`::moving from state init -> state preparing >Thread-1582::INFO::2012-12-02 23:40:56,770::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1582::INFO::2012-12-02 23:40:56,770::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1582::DEBUG::2012-12-02 23:40:56,770::task::1151::TaskManager.Task::(prepare) Task=`d297ace3-e73c-4219-9233-0fa94d6d350f`::finished: {} >Thread-1582::DEBUG::2012-12-02 23:40:56,770::task::568::TaskManager.Task::(_updateState) Task=`d297ace3-e73c-4219-9233-0fa94d6d350f`::moving from state preparing -> state finished >Thread-1582::DEBUG::2012-12-02 23:40:56,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1582::DEBUG::2012-12-02 23:40:56,770::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1582::DEBUG::2012-12-02 23:40:56,770::task::957::TaskManager.Task::(_decref) Task=`d297ace3-e73c-4219-9233-0fa94d6d350f`::ref 0 aborting False >Thread-1581::DEBUG::2012-12-02 23:40:56,900::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1581::WARNING::2012-12-02 23:40:56,901::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1581::DEBUG::2012-12-02 23:40:56,902::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1581::DEBUG::2012-12-02 23:40:56,931::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1581::DEBUG::2012-12-02 23:40:56,931::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1581::DEBUG::2012-12-02 23:40:56,931::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1581::DEBUG::2012-12-02 23:40:56,931::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1581::ERROR::2012-12-02 23:40:56,932::task::833::TaskManager.Task::(_setError) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1581::DEBUG::2012-12-02 23:40:56,932::task::852::TaskManager.Task::(_run) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::Task._run: 69ed2e32-e64d-4e14-8605-e9f18f2c39cd ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1581::DEBUG::2012-12-02 23:40:56,932::task::1177::TaskManager.Task::(stop) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::stopping in state preparing (force False) >Thread-1581::DEBUG::2012-12-02 23:40:56,932::task::957::TaskManager.Task::(_decref) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::ref 1 aborting True >Thread-1581::INFO::2012-12-02 23:40:56,932::task::1134::TaskManager.Task::(prepare) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1581::DEBUG::2012-12-02 23:40:56,933::task::1139::TaskManager.Task::(prepare) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::Prepare: aborted: Cannot find master domain >Thread-1581::DEBUG::2012-12-02 23:40:56,933::task::957::TaskManager.Task::(_decref) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::ref 0 aborting True >Thread-1581::DEBUG::2012-12-02 23:40:56,933::task::892::TaskManager.Task::(_doAbort) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::Task._doAbort: force False >Thread-1581::DEBUG::2012-12-02 23:40:56,933::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1581::DEBUG::2012-12-02 23:40:56,933::task::568::TaskManager.Task::(_updateState) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::moving from state preparing -> state aborting >Thread-1581::DEBUG::2012-12-02 23:40:56,933::task::523::TaskManager.Task::(__state_aborting) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::_aborting: recover policy none >Thread-1581::DEBUG::2012-12-02 23:40:56,933::task::568::TaskManager.Task::(_updateState) Task=`69ed2e32-e64d-4e14-8605-e9f18f2c39cd`::moving from state aborting -> state failed >Thread-1581::DEBUG::2012-12-02 23:40:56,933::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1581::DEBUG::2012-12-02 23:40:56,934::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1581::ERROR::2012-12-02 23:40:56,934::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1649::DEBUG::2012-12-02 23:41:07,204::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1649::DEBUG::2012-12-02 23:41:07,204::task::568::TaskManager.Task::(_updateState) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::moving from state init -> state preparing >Thread-1649::INFO::2012-12-02 23:41:07,205::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1649::ERROR::2012-12-02 23:41:07,205::task::833::TaskManager.Task::(_setError) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1649::DEBUG::2012-12-02 23:41:07,205::task::852::TaskManager.Task::(_run) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::Task._run: 15aa2728-6d60-49cd-b2a5-a77ed2843bb1 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1649::DEBUG::2012-12-02 23:41:07,205::task::1177::TaskManager.Task::(stop) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::stopping in state preparing (force False) >Thread-1649::DEBUG::2012-12-02 23:41:07,205::task::957::TaskManager.Task::(_decref) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::ref 1 aborting True >Thread-1649::INFO::2012-12-02 23:41:07,205::task::1134::TaskManager.Task::(prepare) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1649::DEBUG::2012-12-02 23:41:07,205::task::1139::TaskManager.Task::(prepare) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1649::DEBUG::2012-12-02 23:41:07,206::task::957::TaskManager.Task::(_decref) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::ref 0 aborting True >Thread-1649::DEBUG::2012-12-02 23:41:07,206::task::892::TaskManager.Task::(_doAbort) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::Task._doAbort: force False >Thread-1649::DEBUG::2012-12-02 23:41:07,206::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1649::DEBUG::2012-12-02 23:41:07,206::task::568::TaskManager.Task::(_updateState) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::moving from state preparing -> state aborting >Thread-1649::DEBUG::2012-12-02 23:41:07,206::task::523::TaskManager.Task::(__state_aborting) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::_aborting: recover policy none >Thread-1649::DEBUG::2012-12-02 23:41:07,206::task::568::TaskManager.Task::(_updateState) Task=`15aa2728-6d60-49cd-b2a5-a77ed2843bb1`::moving from state aborting -> state failed >Thread-1649::DEBUG::2012-12-02 23:41:07,206::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1649::DEBUG::2012-12-02 23:41:07,206::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1649::ERROR::2012-12-02 23:41:07,206::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1650::DEBUG::2012-12-02 23:41:07,340::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1650::DEBUG::2012-12-02 23:41:07,341::task::568::TaskManager.Task::(_updateState) Task=`b332ce28-fbea-48f7-bfe3-cb918914a0a6`::moving from state init -> state preparing >Thread-1650::INFO::2012-12-02 23:41:07,341::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1650::DEBUG::2012-12-02 23:41:07,341::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1650::DEBUG::2012-12-02 23:41:07,341::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1650::INFO::2012-12-02 23:41:07,341::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1650::DEBUG::2012-12-02 23:41:07,341::task::1151::TaskManager.Task::(prepare) Task=`b332ce28-fbea-48f7-bfe3-cb918914a0a6`::finished: {'allTasksStatus': {}} >Thread-1650::DEBUG::2012-12-02 23:41:07,341::task::568::TaskManager.Task::(_updateState) Task=`b332ce28-fbea-48f7-bfe3-cb918914a0a6`::moving from state preparing -> state finished >Thread-1650::DEBUG::2012-12-02 23:41:07,342::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1650::DEBUG::2012-12-02 23:41:07,342::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1650::DEBUG::2012-12-02 23:41:07,342::task::957::TaskManager.Task::(_decref) Task=`b332ce28-fbea-48f7-bfe3-cb918914a0a6`::ref 0 aborting False >Thread-1651::DEBUG::2012-12-02 23:41:07,485::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1651::DEBUG::2012-12-02 23:41:07,485::task::568::TaskManager.Task::(_updateState) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::moving from state init -> state preparing >Thread-1651::INFO::2012-12-02 23:41:07,486::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1651::ERROR::2012-12-02 23:41:07,486::task::833::TaskManager.Task::(_setError) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1651::DEBUG::2012-12-02 23:41:07,486::task::852::TaskManager.Task::(_run) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::Task._run: 5dbe7661-7e43-4143-97f6-beebe830df27 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1651::DEBUG::2012-12-02 23:41:07,486::task::1177::TaskManager.Task::(stop) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::stopping in state preparing (force False) >Thread-1651::DEBUG::2012-12-02 23:41:07,486::task::957::TaskManager.Task::(_decref) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::ref 1 aborting True >Thread-1651::INFO::2012-12-02 23:41:07,486::task::1134::TaskManager.Task::(prepare) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1651::DEBUG::2012-12-02 23:41:07,487::task::1139::TaskManager.Task::(prepare) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1651::DEBUG::2012-12-02 23:41:07,487::task::957::TaskManager.Task::(_decref) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::ref 0 aborting True >Thread-1651::DEBUG::2012-12-02 23:41:07,487::task::892::TaskManager.Task::(_doAbort) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::Task._doAbort: force False >Thread-1651::DEBUG::2012-12-02 23:41:07,487::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1651::DEBUG::2012-12-02 23:41:07,487::task::568::TaskManager.Task::(_updateState) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::moving from state preparing -> state aborting >Thread-1651::DEBUG::2012-12-02 23:41:07,487::task::523::TaskManager.Task::(__state_aborting) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::_aborting: recover policy none >Thread-1651::DEBUG::2012-12-02 23:41:07,487::task::568::TaskManager.Task::(_updateState) Task=`5dbe7661-7e43-4143-97f6-beebe830df27`::moving from state aborting -> state failed >Thread-1651::DEBUG::2012-12-02 23:41:07,487::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1651::DEBUG::2012-12-02 23:41:07,488::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1651::ERROR::2012-12-02 23:41:07,488::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1652::DEBUG::2012-12-02 23:41:07,657::task::568::TaskManager.Task::(_updateState) Task=`1f4f56ea-94ce-4d82-9690-95aa881b46b7`::moving from state init -> state preparing >Thread-1652::INFO::2012-12-02 23:41:07,658::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1652::INFO::2012-12-02 23:41:07,658::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1652::DEBUG::2012-12-02 23:41:07,658::task::1151::TaskManager.Task::(prepare) Task=`1f4f56ea-94ce-4d82-9690-95aa881b46b7`::finished: {} >Thread-1652::DEBUG::2012-12-02 23:41:07,658::task::568::TaskManager.Task::(_updateState) Task=`1f4f56ea-94ce-4d82-9690-95aa881b46b7`::moving from state preparing -> state finished >Thread-1652::DEBUG::2012-12-02 23:41:07,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1652::DEBUG::2012-12-02 23:41:07,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1652::DEBUG::2012-12-02 23:41:07,658::task::957::TaskManager.Task::(_decref) Task=`1f4f56ea-94ce-4d82-9690-95aa881b46b7`::ref 0 aborting False >Thread-1653::DEBUG::2012-12-02 23:41:07,674::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1653::DEBUG::2012-12-02 23:41:07,675::task::568::TaskManager.Task::(_updateState) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::moving from state init -> state preparing >Thread-1653::INFO::2012-12-02 23:41:07,675::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1653::DEBUG::2012-12-02 23:41:07,675::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`70f1d2b5-b528-447a-8bce-7550ff267273`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1653::DEBUG::2012-12-02 23:41:07,675::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1653::DEBUG::2012-12-02 23:41:07,676::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1653::DEBUG::2012-12-02 23:41:07,676::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`70f1d2b5-b528-447a-8bce-7550ff267273`::Granted request >Thread-1653::INFO::2012-12-02 23:41:07,676::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1653::DEBUG::2012-12-02 23:41:07,676::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:07,677::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:07,677::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:07,677::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:07,677::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:07,677::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:07,677::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1653::DEBUG::2012-12-02 23:41:07,677::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1653::DEBUG::2012-12-02 23:41:07,678::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1653::DEBUG::2012-12-02 23:41:07,678::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1653::DEBUG::2012-12-02 23:41:07,678::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1653::DEBUG::2012-12-02 23:41:07,702::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1653::DEBUG::2012-12-02 23:41:07,703::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1653::DEBUG::2012-12-02 23:41:08,336::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1653::DEBUG::2012-12-02 23:41:08,411::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1653::DEBUG::2012-12-02 23:41:08,411::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,412::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,412::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,412::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,412::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,412::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,412::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1653::DEBUG::2012-12-02 23:41:08,413::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,414::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1653::DEBUG::2012-12-02 23:41:08,624::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1653::WARNING::2012-12-02 23:41:08,626::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1653::DEBUG::2012-12-02 23:41:08,626::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1653::DEBUG::2012-12-02 23:41:08,658::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1653::DEBUG::2012-12-02 23:41:08,659::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1653::DEBUG::2012-12-02 23:41:08,659::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1653::DEBUG::2012-12-02 23:41:08,659::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1653::ERROR::2012-12-02 23:41:08,659::task::833::TaskManager.Task::(_setError) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1653::DEBUG::2012-12-02 23:41:08,660::task::852::TaskManager.Task::(_run) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::Task._run: 16ccb4e9-f893-455a-8cdb-c7568e411da3 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1653::DEBUG::2012-12-02 23:41:08,660::task::1177::TaskManager.Task::(stop) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::stopping in state preparing (force False) >Thread-1653::DEBUG::2012-12-02 23:41:08,660::task::957::TaskManager.Task::(_decref) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::ref 1 aborting True >Thread-1653::INFO::2012-12-02 23:41:08,660::task::1134::TaskManager.Task::(prepare) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1653::DEBUG::2012-12-02 23:41:08,660::task::1139::TaskManager.Task::(prepare) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::Prepare: aborted: Cannot find master domain >Thread-1653::DEBUG::2012-12-02 23:41:08,660::task::957::TaskManager.Task::(_decref) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::ref 0 aborting True >Thread-1653::DEBUG::2012-12-02 23:41:08,661::task::892::TaskManager.Task::(_doAbort) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::Task._doAbort: force False >Thread-1653::DEBUG::2012-12-02 23:41:08,661::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1653::DEBUG::2012-12-02 23:41:08,661::task::568::TaskManager.Task::(_updateState) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::moving from state preparing -> state aborting >Thread-1653::DEBUG::2012-12-02 23:41:08,661::task::523::TaskManager.Task::(__state_aborting) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::_aborting: recover policy none >Thread-1653::DEBUG::2012-12-02 23:41:08,661::task::568::TaskManager.Task::(_updateState) Task=`16ccb4e9-f893-455a-8cdb-c7568e411da3`::moving from state aborting -> state failed >Thread-1653::DEBUG::2012-12-02 23:41:08,661::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1653::DEBUG::2012-12-02 23:41:08,661::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1653::ERROR::2012-12-02 23:41:08,661::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1720::DEBUG::2012-12-02 23:41:18,691::task::568::TaskManager.Task::(_updateState) Task=`01cf1119-7eaa-4cbe-9e1b-76daa32d7797`::moving from state init -> state preparing >Thread-1720::INFO::2012-12-02 23:41:18,691::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1720::INFO::2012-12-02 23:41:18,691::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1720::DEBUG::2012-12-02 23:41:18,691::task::1151::TaskManager.Task::(prepare) Task=`01cf1119-7eaa-4cbe-9e1b-76daa32d7797`::finished: {} >Thread-1720::DEBUG::2012-12-02 23:41:18,692::task::568::TaskManager.Task::(_updateState) Task=`01cf1119-7eaa-4cbe-9e1b-76daa32d7797`::moving from state preparing -> state finished >Thread-1720::DEBUG::2012-12-02 23:41:18,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1720::DEBUG::2012-12-02 23:41:18,692::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1720::DEBUG::2012-12-02 23:41:18,692::task::957::TaskManager.Task::(_decref) Task=`01cf1119-7eaa-4cbe-9e1b-76daa32d7797`::ref 0 aborting False >Thread-1722::DEBUG::2012-12-02 23:41:19,025::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1722::DEBUG::2012-12-02 23:41:19,025::task::568::TaskManager.Task::(_updateState) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::moving from state init -> state preparing >Thread-1722::INFO::2012-12-02 23:41:19,026::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1722::ERROR::2012-12-02 23:41:19,026::task::833::TaskManager.Task::(_setError) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1722::DEBUG::2012-12-02 23:41:19,026::task::852::TaskManager.Task::(_run) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::Task._run: 08d700a6-9c6c-4610-be0d-eaf3c85ae448 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1722::DEBUG::2012-12-02 23:41:19,026::task::1177::TaskManager.Task::(stop) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::stopping in state preparing (force False) >Thread-1722::DEBUG::2012-12-02 23:41:19,026::task::957::TaskManager.Task::(_decref) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::ref 1 aborting True >Thread-1722::INFO::2012-12-02 23:41:19,026::task::1134::TaskManager.Task::(prepare) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1722::DEBUG::2012-12-02 23:41:19,026::task::1139::TaskManager.Task::(prepare) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1722::DEBUG::2012-12-02 23:41:19,027::task::957::TaskManager.Task::(_decref) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::ref 0 aborting True >Thread-1722::DEBUG::2012-12-02 23:41:19,027::task::892::TaskManager.Task::(_doAbort) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::Task._doAbort: force False >Thread-1722::DEBUG::2012-12-02 23:41:19,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1722::DEBUG::2012-12-02 23:41:19,027::task::568::TaskManager.Task::(_updateState) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::moving from state preparing -> state aborting >Thread-1722::DEBUG::2012-12-02 23:41:19,027::task::523::TaskManager.Task::(__state_aborting) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::_aborting: recover policy none >Thread-1722::DEBUG::2012-12-02 23:41:19,027::task::568::TaskManager.Task::(_updateState) Task=`08d700a6-9c6c-4610-be0d-eaf3c85ae448`::moving from state aborting -> state failed >Thread-1722::DEBUG::2012-12-02 23:41:19,027::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1722::DEBUG::2012-12-02 23:41:19,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1722::ERROR::2012-12-02 23:41:19,028::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1723::DEBUG::2012-12-02 23:41:19,168::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1723::DEBUG::2012-12-02 23:41:19,168::task::568::TaskManager.Task::(_updateState) Task=`97b7c72c-8c70-4938-ab3d-5478cbe03be1`::moving from state init -> state preparing >Thread-1723::INFO::2012-12-02 23:41:19,168::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1723::DEBUG::2012-12-02 23:41:19,168::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1723::DEBUG::2012-12-02 23:41:19,168::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1723::INFO::2012-12-02 23:41:19,168::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1723::DEBUG::2012-12-02 23:41:19,169::task::1151::TaskManager.Task::(prepare) Task=`97b7c72c-8c70-4938-ab3d-5478cbe03be1`::finished: {'allTasksStatus': {}} >Thread-1723::DEBUG::2012-12-02 23:41:19,169::task::568::TaskManager.Task::(_updateState) Task=`97b7c72c-8c70-4938-ab3d-5478cbe03be1`::moving from state preparing -> state finished >Thread-1723::DEBUG::2012-12-02 23:41:19,169::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1723::DEBUG::2012-12-02 23:41:19,169::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1723::DEBUG::2012-12-02 23:41:19,169::task::957::TaskManager.Task::(_decref) Task=`97b7c72c-8c70-4938-ab3d-5478cbe03be1`::ref 0 aborting False >Thread-1724::DEBUG::2012-12-02 23:41:19,308::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1724::DEBUG::2012-12-02 23:41:19,308::task::568::TaskManager.Task::(_updateState) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::moving from state init -> state preparing >Thread-1724::INFO::2012-12-02 23:41:19,308::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1724::ERROR::2012-12-02 23:41:19,309::task::833::TaskManager.Task::(_setError) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1724::DEBUG::2012-12-02 23:41:19,309::task::852::TaskManager.Task::(_run) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::Task._run: 56bd4aea-1403-41a4-be10-1cd13f373deb ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1724::DEBUG::2012-12-02 23:41:19,309::task::1177::TaskManager.Task::(stop) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::stopping in state preparing (force False) >Thread-1724::DEBUG::2012-12-02 23:41:19,309::task::957::TaskManager.Task::(_decref) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::ref 1 aborting True >Thread-1724::INFO::2012-12-02 23:41:19,309::task::1134::TaskManager.Task::(prepare) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1724::DEBUG::2012-12-02 23:41:19,309::task::1139::TaskManager.Task::(prepare) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1724::DEBUG::2012-12-02 23:41:19,309::task::957::TaskManager.Task::(_decref) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::ref 0 aborting True >Thread-1724::DEBUG::2012-12-02 23:41:19,310::task::892::TaskManager.Task::(_doAbort) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::Task._doAbort: force False >Thread-1724::DEBUG::2012-12-02 23:41:19,310::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1724::DEBUG::2012-12-02 23:41:19,310::task::568::TaskManager.Task::(_updateState) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::moving from state preparing -> state aborting >Thread-1724::DEBUG::2012-12-02 23:41:19,310::task::523::TaskManager.Task::(__state_aborting) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::_aborting: recover policy none >Thread-1724::DEBUG::2012-12-02 23:41:19,310::task::568::TaskManager.Task::(_updateState) Task=`56bd4aea-1403-41a4-be10-1cd13f373deb`::moving from state aborting -> state failed >Thread-1724::DEBUG::2012-12-02 23:41:19,310::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1724::DEBUG::2012-12-02 23:41:19,310::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1724::ERROR::2012-12-02 23:41:19,310::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1725::DEBUG::2012-12-02 23:41:19,488::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1725::DEBUG::2012-12-02 23:41:19,488::task::568::TaskManager.Task::(_updateState) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::moving from state init -> state preparing >Thread-1725::INFO::2012-12-02 23:41:19,488::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1725::DEBUG::2012-12-02 23:41:19,489::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`17ab13be-93c2-4ee0-822a-eefdb9ecb721`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1725::DEBUG::2012-12-02 23:41:19,489::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1725::DEBUG::2012-12-02 23:41:19,489::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1725::DEBUG::2012-12-02 23:41:19,489::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`17ab13be-93c2-4ee0-822a-eefdb9ecb721`::Granted request >Thread-1725::INFO::2012-12-02 23:41:19,490::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1725::DEBUG::2012-12-02 23:41:19,490::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:19,490::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:19,490::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:19,490::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:19,491::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:19,491::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:19,491::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1725::DEBUG::2012-12-02 23:41:19,491::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1725::DEBUG::2012-12-02 23:41:19,491::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1725::DEBUG::2012-12-02 23:41:19,491::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1725::DEBUG::2012-12-02 23:41:19,491::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1725::DEBUG::2012-12-02 23:41:19,520::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1725::DEBUG::2012-12-02 23:41:19,520::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1725::DEBUG::2012-12-02 23:41:20,153::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1725::DEBUG::2012-12-02 23:41:20,212::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1725::DEBUG::2012-12-02 23:41:20,213::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,213::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,213::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,213::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,214::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,214::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,214::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1725::DEBUG::2012-12-02 23:41:20,214::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,216::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1725::DEBUG::2012-12-02 23:41:20,421::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1725::WARNING::2012-12-02 23:41:20,422::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1725::DEBUG::2012-12-02 23:41:20,423::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1725::DEBUG::2012-12-02 23:41:20,452::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1725::DEBUG::2012-12-02 23:41:20,452::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1725::DEBUG::2012-12-02 23:41:20,452::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1725::DEBUG::2012-12-02 23:41:20,452::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1725::ERROR::2012-12-02 23:41:20,452::task::833::TaskManager.Task::(_setError) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1725::DEBUG::2012-12-02 23:41:20,453::task::852::TaskManager.Task::(_run) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::Task._run: 8e4e6731-2fa3-47a9-ab37-21dbc21e9e50 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1725::DEBUG::2012-12-02 23:41:20,453::task::1177::TaskManager.Task::(stop) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::stopping in state preparing (force False) >Thread-1725::DEBUG::2012-12-02 23:41:20,453::task::957::TaskManager.Task::(_decref) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::ref 1 aborting True >Thread-1725::INFO::2012-12-02 23:41:20,453::task::1134::TaskManager.Task::(prepare) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1725::DEBUG::2012-12-02 23:41:20,453::task::1139::TaskManager.Task::(prepare) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::Prepare: aborted: Cannot find master domain >Thread-1725::DEBUG::2012-12-02 23:41:20,453::task::957::TaskManager.Task::(_decref) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::ref 0 aborting True >Thread-1725::DEBUG::2012-12-02 23:41:20,453::task::892::TaskManager.Task::(_doAbort) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::Task._doAbort: force False >Thread-1725::DEBUG::2012-12-02 23:41:20,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1725::DEBUG::2012-12-02 23:41:20,454::task::568::TaskManager.Task::(_updateState) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::moving from state preparing -> state aborting >Thread-1725::DEBUG::2012-12-02 23:41:20,454::task::523::TaskManager.Task::(__state_aborting) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::_aborting: recover policy none >Thread-1725::DEBUG::2012-12-02 23:41:20,454::task::568::TaskManager.Task::(_updateState) Task=`8e4e6731-2fa3-47a9-ab37-21dbc21e9e50`::moving from state aborting -> state failed >Thread-1725::DEBUG::2012-12-02 23:41:20,454::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1725::DEBUG::2012-12-02 23:41:20,454::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1725::ERROR::2012-12-02 23:41:20,454::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1791::DEBUG::2012-12-02 23:41:29,597::task::568::TaskManager.Task::(_updateState) Task=`23f5c32c-79b4-48e1-a14e-03075563eaa8`::moving from state init -> state preparing >Thread-1791::INFO::2012-12-02 23:41:29,597::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1791::INFO::2012-12-02 23:41:29,597::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1791::DEBUG::2012-12-02 23:41:29,597::task::1151::TaskManager.Task::(prepare) Task=`23f5c32c-79b4-48e1-a14e-03075563eaa8`::finished: {} >Thread-1791::DEBUG::2012-12-02 23:41:29,597::task::568::TaskManager.Task::(_updateState) Task=`23f5c32c-79b4-48e1-a14e-03075563eaa8`::moving from state preparing -> state finished >Thread-1791::DEBUG::2012-12-02 23:41:29,597::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1791::DEBUG::2012-12-02 23:41:29,598::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1791::DEBUG::2012-12-02 23:41:29,598::task::957::TaskManager.Task::(_decref) Task=`23f5c32c-79b4-48e1-a14e-03075563eaa8`::ref 0 aborting False >Thread-1793::DEBUG::2012-12-02 23:41:30,759::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1793::DEBUG::2012-12-02 23:41:30,759::task::568::TaskManager.Task::(_updateState) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::moving from state init -> state preparing >Thread-1793::INFO::2012-12-02 23:41:30,759::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1793::ERROR::2012-12-02 23:41:30,760::task::833::TaskManager.Task::(_setError) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1793::DEBUG::2012-12-02 23:41:30,760::task::852::TaskManager.Task::(_run) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::Task._run: fb83197f-8c3b-4427-9e6f-0a9342e6ffa5 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1793::DEBUG::2012-12-02 23:41:30,760::task::1177::TaskManager.Task::(stop) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::stopping in state preparing (force False) >Thread-1793::DEBUG::2012-12-02 23:41:30,760::task::957::TaskManager.Task::(_decref) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::ref 1 aborting True >Thread-1793::INFO::2012-12-02 23:41:30,760::task::1134::TaskManager.Task::(prepare) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1793::DEBUG::2012-12-02 23:41:30,760::task::1139::TaskManager.Task::(prepare) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1793::DEBUG::2012-12-02 23:41:30,760::task::957::TaskManager.Task::(_decref) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::ref 0 aborting True >Thread-1793::DEBUG::2012-12-02 23:41:30,761::task::892::TaskManager.Task::(_doAbort) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::Task._doAbort: force False >Thread-1793::DEBUG::2012-12-02 23:41:30,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1793::DEBUG::2012-12-02 23:41:30,761::task::568::TaskManager.Task::(_updateState) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::moving from state preparing -> state aborting >Thread-1793::DEBUG::2012-12-02 23:41:30,761::task::523::TaskManager.Task::(__state_aborting) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::_aborting: recover policy none >Thread-1793::DEBUG::2012-12-02 23:41:30,761::task::568::TaskManager.Task::(_updateState) Task=`fb83197f-8c3b-4427-9e6f-0a9342e6ffa5`::moving from state aborting -> state failed >Thread-1793::DEBUG::2012-12-02 23:41:30,761::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1793::DEBUG::2012-12-02 23:41:30,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1793::ERROR::2012-12-02 23:41:30,761::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1794::DEBUG::2012-12-02 23:41:30,898::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1794::DEBUG::2012-12-02 23:41:30,898::task::568::TaskManager.Task::(_updateState) Task=`22bddbab-9da2-45ea-9508-17739469ded5`::moving from state init -> state preparing >Thread-1794::INFO::2012-12-02 23:41:30,898::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1794::DEBUG::2012-12-02 23:41:30,899::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1794::DEBUG::2012-12-02 23:41:30,899::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1794::INFO::2012-12-02 23:41:30,899::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1794::DEBUG::2012-12-02 23:41:30,899::task::1151::TaskManager.Task::(prepare) Task=`22bddbab-9da2-45ea-9508-17739469ded5`::finished: {'allTasksStatus': {}} >Thread-1794::DEBUG::2012-12-02 23:41:30,899::task::568::TaskManager.Task::(_updateState) Task=`22bddbab-9da2-45ea-9508-17739469ded5`::moving from state preparing -> state finished >Thread-1794::DEBUG::2012-12-02 23:41:30,899::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1794::DEBUG::2012-12-02 23:41:30,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1794::DEBUG::2012-12-02 23:41:30,899::task::957::TaskManager.Task::(_decref) Task=`22bddbab-9da2-45ea-9508-17739469ded5`::ref 0 aborting False >Thread-1795::DEBUG::2012-12-02 23:41:31,048::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1795::DEBUG::2012-12-02 23:41:31,048::task::568::TaskManager.Task::(_updateState) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::moving from state init -> state preparing >Thread-1795::INFO::2012-12-02 23:41:31,048::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1795::ERROR::2012-12-02 23:41:31,048::task::833::TaskManager.Task::(_setError) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1795::DEBUG::2012-12-02 23:41:31,049::task::852::TaskManager.Task::(_run) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::Task._run: 89572de3-20d8-4dd2-b9c2-c6c69f2548a1 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1795::DEBUG::2012-12-02 23:41:31,049::task::1177::TaskManager.Task::(stop) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::stopping in state preparing (force False) >Thread-1795::DEBUG::2012-12-02 23:41:31,049::task::957::TaskManager.Task::(_decref) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::ref 1 aborting True >Thread-1795::INFO::2012-12-02 23:41:31,049::task::1134::TaskManager.Task::(prepare) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1795::DEBUG::2012-12-02 23:41:31,049::task::1139::TaskManager.Task::(prepare) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1795::DEBUG::2012-12-02 23:41:31,049::task::957::TaskManager.Task::(_decref) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::ref 0 aborting True >Thread-1795::DEBUG::2012-12-02 23:41:31,050::task::892::TaskManager.Task::(_doAbort) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::Task._doAbort: force False >Thread-1795::DEBUG::2012-12-02 23:41:31,050::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1795::DEBUG::2012-12-02 23:41:31,050::task::568::TaskManager.Task::(_updateState) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::moving from state preparing -> state aborting >Thread-1795::DEBUG::2012-12-02 23:41:31,050::task::523::TaskManager.Task::(__state_aborting) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::_aborting: recover policy none >Thread-1795::DEBUG::2012-12-02 23:41:31,050::task::568::TaskManager.Task::(_updateState) Task=`89572de3-20d8-4dd2-b9c2-c6c69f2548a1`::moving from state aborting -> state failed >Thread-1795::DEBUG::2012-12-02 23:41:31,050::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1795::DEBUG::2012-12-02 23:41:31,050::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1795::ERROR::2012-12-02 23:41:31,050::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1796::DEBUG::2012-12-02 23:41:31,235::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1796::DEBUG::2012-12-02 23:41:31,236::task::568::TaskManager.Task::(_updateState) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::moving from state init -> state preparing >Thread-1796::INFO::2012-12-02 23:41:31,236::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1796::DEBUG::2012-12-02 23:41:31,236::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2634f2bf-00d1-47fe-bbb9-d1e0ec45c095`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1796::DEBUG::2012-12-02 23:41:31,236::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1796::DEBUG::2012-12-02 23:41:31,237::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1796::DEBUG::2012-12-02 23:41:31,237::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2634f2bf-00d1-47fe-bbb9-d1e0ec45c095`::Granted request >Thread-1796::INFO::2012-12-02 23:41:31,237::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1796::DEBUG::2012-12-02 23:41:31,237::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,238::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,238::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,238::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,238::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,238::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,238::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1796::DEBUG::2012-12-02 23:41:31,238::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1796::DEBUG::2012-12-02 23:41:31,239::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1796::DEBUG::2012-12-02 23:41:31,239::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1796::DEBUG::2012-12-02 23:41:31,239::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1796::DEBUG::2012-12-02 23:41:31,264::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1796::DEBUG::2012-12-02 23:41:31,264::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1796::DEBUG::2012-12-02 23:41:31,897::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1796::DEBUG::2012-12-02 23:41:31,969::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,970::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1796::DEBUG::2012-12-02 23:41:31,970::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:31,972::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1796::DEBUG::2012-12-02 23:41:32,175::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1796::WARNING::2012-12-02 23:41:32,177::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1796::DEBUG::2012-12-02 23:41:32,177::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1796::DEBUG::2012-12-02 23:41:32,206::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1796::DEBUG::2012-12-02 23:41:32,206::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1796::DEBUG::2012-12-02 23:41:32,206::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1796::DEBUG::2012-12-02 23:41:32,206::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1796::ERROR::2012-12-02 23:41:32,207::task::833::TaskManager.Task::(_setError) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1796::DEBUG::2012-12-02 23:41:32,207::task::852::TaskManager.Task::(_run) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::Task._run: ce0b4d43-01e3-43f8-8182-3476580392e7 ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1796::DEBUG::2012-12-02 23:41:32,207::task::1177::TaskManager.Task::(stop) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::stopping in state preparing (force False) >Thread-1796::DEBUG::2012-12-02 23:41:32,207::task::957::TaskManager.Task::(_decref) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::ref 1 aborting True >Thread-1796::INFO::2012-12-02 23:41:32,207::task::1134::TaskManager.Task::(prepare) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1796::DEBUG::2012-12-02 23:41:32,208::task::1139::TaskManager.Task::(prepare) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::Prepare: aborted: Cannot find master domain >Thread-1796::DEBUG::2012-12-02 23:41:32,208::task::957::TaskManager.Task::(_decref) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::ref 0 aborting True >Thread-1796::DEBUG::2012-12-02 23:41:32,208::task::892::TaskManager.Task::(_doAbort) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::Task._doAbort: force False >Thread-1796::DEBUG::2012-12-02 23:41:32,208::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1796::DEBUG::2012-12-02 23:41:32,208::task::568::TaskManager.Task::(_updateState) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::moving from state preparing -> state aborting >Thread-1796::DEBUG::2012-12-02 23:41:32,208::task::523::TaskManager.Task::(__state_aborting) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::_aborting: recover policy none >Thread-1796::DEBUG::2012-12-02 23:41:32,208::task::568::TaskManager.Task::(_updateState) Task=`ce0b4d43-01e3-43f8-8182-3476580392e7`::moving from state aborting -> state failed >Thread-1796::DEBUG::2012-12-02 23:41:32,208::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1796::DEBUG::2012-12-02 23:41:32,209::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1796::ERROR::2012-12-02 23:41:32,209::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1862::DEBUG::2012-12-02 23:41:40,489::task::568::TaskManager.Task::(_updateState) Task=`cbe7880c-5b23-4931-b0d8-1705decaa8c1`::moving from state init -> state preparing >Thread-1862::INFO::2012-12-02 23:41:40,490::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1862::INFO::2012-12-02 23:41:40,490::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1862::DEBUG::2012-12-02 23:41:40,490::task::1151::TaskManager.Task::(prepare) Task=`cbe7880c-5b23-4931-b0d8-1705decaa8c1`::finished: {} >Thread-1862::DEBUG::2012-12-02 23:41:40,490::task::568::TaskManager.Task::(_updateState) Task=`cbe7880c-5b23-4931-b0d8-1705decaa8c1`::moving from state preparing -> state finished >Thread-1862::DEBUG::2012-12-02 23:41:40,490::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1862::DEBUG::2012-12-02 23:41:40,490::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1862::DEBUG::2012-12-02 23:41:40,490::task::957::TaskManager.Task::(_decref) Task=`cbe7880c-5b23-4931-b0d8-1705decaa8c1`::ref 0 aborting False >Thread-1864::DEBUG::2012-12-02 23:41:42,547::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1864::DEBUG::2012-12-02 23:41:42,548::task::568::TaskManager.Task::(_updateState) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::moving from state init -> state preparing >Thread-1864::INFO::2012-12-02 23:41:42,548::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1864::ERROR::2012-12-02 23:41:42,548::task::833::TaskManager.Task::(_setError) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1864::DEBUG::2012-12-02 23:41:42,548::task::852::TaskManager.Task::(_run) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::Task._run: d3ab3207-85ee-4718-9332-737ead9542c1 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1864::DEBUG::2012-12-02 23:41:42,548::task::1177::TaskManager.Task::(stop) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::stopping in state preparing (force False) >Thread-1864::DEBUG::2012-12-02 23:41:42,548::task::957::TaskManager.Task::(_decref) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::ref 1 aborting True >Thread-1864::INFO::2012-12-02 23:41:42,549::task::1134::TaskManager.Task::(prepare) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1864::DEBUG::2012-12-02 23:41:42,549::task::1139::TaskManager.Task::(prepare) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1864::DEBUG::2012-12-02 23:41:42,549::task::957::TaskManager.Task::(_decref) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::ref 0 aborting True >Thread-1864::DEBUG::2012-12-02 23:41:42,549::task::892::TaskManager.Task::(_doAbort) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::Task._doAbort: force False >Thread-1864::DEBUG::2012-12-02 23:41:42,549::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1864::DEBUG::2012-12-02 23:41:42,549::task::568::TaskManager.Task::(_updateState) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::moving from state preparing -> state aborting >Thread-1864::DEBUG::2012-12-02 23:41:42,549::task::523::TaskManager.Task::(__state_aborting) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::_aborting: recover policy none >Thread-1864::DEBUG::2012-12-02 23:41:42,549::task::568::TaskManager.Task::(_updateState) Task=`d3ab3207-85ee-4718-9332-737ead9542c1`::moving from state aborting -> state failed >Thread-1864::DEBUG::2012-12-02 23:41:42,550::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1864::DEBUG::2012-12-02 23:41:42,550::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1864::ERROR::2012-12-02 23:41:42,550::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1865::DEBUG::2012-12-02 23:41:42,718::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1865::DEBUG::2012-12-02 23:41:42,719::task::568::TaskManager.Task::(_updateState) Task=`6b06c6df-06d0-441d-86a1-23ca8bf2629c`::moving from state init -> state preparing >Thread-1865::INFO::2012-12-02 23:41:42,719::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-1865::DEBUG::2012-12-02 23:41:42,719::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-1865::DEBUG::2012-12-02 23:41:42,719::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-1865::INFO::2012-12-02 23:41:42,719::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-1865::DEBUG::2012-12-02 23:41:42,719::task::1151::TaskManager.Task::(prepare) Task=`6b06c6df-06d0-441d-86a1-23ca8bf2629c`::finished: {'allTasksStatus': {}} >Thread-1865::DEBUG::2012-12-02 23:41:42,719::task::568::TaskManager.Task::(_updateState) Task=`6b06c6df-06d0-441d-86a1-23ca8bf2629c`::moving from state preparing -> state finished >Thread-1865::DEBUG::2012-12-02 23:41:42,719::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1865::DEBUG::2012-12-02 23:41:42,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1865::DEBUG::2012-12-02 23:41:42,720::task::957::TaskManager.Task::(_decref) Task=`6b06c6df-06d0-441d-86a1-23ca8bf2629c`::ref 0 aborting False >Thread-1867::DEBUG::2012-12-02 23:41:42,865::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1867::DEBUG::2012-12-02 23:41:42,866::task::568::TaskManager.Task::(_updateState) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::moving from state init -> state preparing >Thread-1867::INFO::2012-12-02 23:41:42,866::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1867::ERROR::2012-12-02 23:41:42,866::task::833::TaskManager.Task::(_setError) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-1867::DEBUG::2012-12-02 23:41:42,866::task::852::TaskManager.Task::(_run) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::Task._run: ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-1867::DEBUG::2012-12-02 23:41:42,867::task::1177::TaskManager.Task::(stop) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::stopping in state preparing (force False) >Thread-1867::DEBUG::2012-12-02 23:41:42,867::task::957::TaskManager.Task::(_decref) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::ref 1 aborting True >Thread-1867::INFO::2012-12-02 23:41:42,867::task::1134::TaskManager.Task::(prepare) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-1867::DEBUG::2012-12-02 23:41:42,867::task::1139::TaskManager.Task::(prepare) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::Prepare: aborted: Unknown pool id, pool not connected >Thread-1867::DEBUG::2012-12-02 23:41:42,867::task::957::TaskManager.Task::(_decref) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::ref 0 aborting True >Thread-1867::DEBUG::2012-12-02 23:41:42,867::task::892::TaskManager.Task::(_doAbort) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::Task._doAbort: force False >Thread-1867::DEBUG::2012-12-02 23:41:42,867::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1867::DEBUG::2012-12-02 23:41:42,867::task::568::TaskManager.Task::(_updateState) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::moving from state preparing -> state aborting >Thread-1867::DEBUG::2012-12-02 23:41:42,868::task::523::TaskManager.Task::(__state_aborting) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::_aborting: recover policy none >Thread-1867::DEBUG::2012-12-02 23:41:42,868::task::568::TaskManager.Task::(_updateState) Task=`ab6c82ff-3ff2-41af-8e7a-f3ed88dd29f7`::moving from state aborting -> state failed >Thread-1867::DEBUG::2012-12-02 23:41:42,868::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1867::DEBUG::2012-12-02 23:41:42,868::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1867::ERROR::2012-12-02 23:41:42,868::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-1868::DEBUG::2012-12-02 23:41:43,028::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1868::DEBUG::2012-12-02 23:41:43,028::task::568::TaskManager.Task::(_updateState) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::moving from state init -> state preparing >Thread-1868::INFO::2012-12-02 23:41:43,029::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='225bca28-f0a0-4899-b137-32acd04f57d4', masterVersion=2, options=None) >Thread-1868::DEBUG::2012-12-02 23:41:43,029::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ae2b3eb7-cee5-4859-a54d-c67906da482e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1868::DEBUG::2012-12-02 23:41:43,029::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1868::DEBUG::2012-12-02 23:41:43,029::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1868::DEBUG::2012-12-02 23:41:43,029::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ae2b3eb7-cee5-4859-a54d-c67906da482e`::Granted request >Thread-1868::INFO::2012-12-02 23:41:43,030::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 225bca28-f0a0-4899-b137-32acd04f57d4 (ver = 2) >Thread-1868::DEBUG::2012-12-02 23:41:43,030::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,030::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,030::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,030::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,031::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,031::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,031::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1868::DEBUG::2012-12-02 23:41:43,031::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1868::DEBUG::2012-12-02 23:41:43,031::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1868::DEBUG::2012-12-02 23:41:43,031::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1868::DEBUG::2012-12-02 23:41:43,031::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1868::DEBUG::2012-12-02 23:41:43,056::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1868::DEBUG::2012-12-02 23:41:43,057::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1868::DEBUG::2012-12-02 23:41:43,691::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1868::DEBUG::2012-12-02 23:41:43,761::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1868::DEBUG::2012-12-02 23:41:43,761::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,762::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,762::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,762::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,762::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,762::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,763::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1868::DEBUG::2012-12-02 23:41:43,763::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:43,764::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1868::DEBUG::2012-12-02 23:41:43,972::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1868::WARNING::2012-12-02 23:41:43,973::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1868::DEBUG::2012-12-02 23:41:43,973::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1868::DEBUG::2012-12-02 23:41:44,004::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1868::DEBUG::2012-12-02 23:41:44,004::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1868::DEBUG::2012-12-02 23:41:44,004::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1868::DEBUG::2012-12-02 23:41:44,004::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1868::ERROR::2012-12-02 23:41:44,004::task::833::TaskManager.Task::(_setError) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-1868::DEBUG::2012-12-02 23:41:44,005::task::852::TaskManager.Task::(_run) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::Task._run: ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '225bca28-f0a0-4899-b137-32acd04f57d4', 2) {} failed - stopping task >Thread-1868::DEBUG::2012-12-02 23:41:44,005::task::1177::TaskManager.Task::(stop) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::stopping in state preparing (force False) >Thread-1868::DEBUG::2012-12-02 23:41:44,005::task::957::TaskManager.Task::(_decref) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::ref 1 aborting True >Thread-1868::INFO::2012-12-02 23:41:44,005::task::1134::TaskManager.Task::(prepare) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-1868::DEBUG::2012-12-02 23:41:44,005::task::1139::TaskManager.Task::(prepare) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::Prepare: aborted: Cannot find master domain >Thread-1868::DEBUG::2012-12-02 23:41:44,006::task::957::TaskManager.Task::(_decref) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::ref 0 aborting True >Thread-1868::DEBUG::2012-12-02 23:41:44,006::task::892::TaskManager.Task::(_doAbort) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::Task._doAbort: force False >Thread-1868::DEBUG::2012-12-02 23:41:44,006::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1868::DEBUG::2012-12-02 23:41:44,006::task::568::TaskManager.Task::(_updateState) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::moving from state preparing -> state aborting >Thread-1868::DEBUG::2012-12-02 23:41:44,006::task::523::TaskManager.Task::(__state_aborting) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::_aborting: recover policy none >Thread-1868::DEBUG::2012-12-02 23:41:44,006::task::568::TaskManager.Task::(_updateState) Task=`ac2e9d3f-2a97-4e50-aa33-70e0bfaf022e`::moving from state aborting -> state failed >Thread-1868::DEBUG::2012-12-02 23:41:44,006::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1868::DEBUG::2012-12-02 23:41:44,006::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1868::ERROR::2012-12-02 23:41:44,007::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=225bca28-f0a0-4899-b137-32acd04f57d4'", 'code': 304}} >Thread-1931::DEBUG::2012-12-02 23:41:45,053::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1931::DEBUG::2012-12-02 23:41:45,053::task::568::TaskManager.Task::(_updateState) Task=`ce9f4be5-85ce-4426-ab6d-ee196d80e8a4`::moving from state init -> state preparing >Thread-1931::INFO::2012-12-02 23:41:45,053::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-1931::INFO::2012-12-02 23:41:45,053::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-1931::DEBUG::2012-12-02 23:41:45,053::task::1151::TaskManager.Task::(prepare) Task=`ce9f4be5-85ce-4426-ab6d-ee196d80e8a4`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-1931::DEBUG::2012-12-02 23:41:45,054::task::568::TaskManager.Task::(_updateState) Task=`ce9f4be5-85ce-4426-ab6d-ee196d80e8a4`::moving from state preparing -> state finished >Thread-1931::DEBUG::2012-12-02 23:41:45,054::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1931::DEBUG::2012-12-02 23:41:45,054::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1931::DEBUG::2012-12-02 23:41:45,054::task::957::TaskManager.Task::(_decref) Task=`ce9f4be5-85ce-4426-ab6d-ee196d80e8a4`::ref 0 aborting False >Thread-1932::DEBUG::2012-12-02 23:41:45,198::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1932::DEBUG::2012-12-02 23:41:45,198::task::568::TaskManager.Task::(_updateState) Task=`4c338dbd-4cdb-416e-b127-9a94d502ca09`::moving from state init -> state preparing >Thread-1932::INFO::2012-12-02 23:41:45,198::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-1932::DEBUG::2012-12-02 23:41:45,205::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328' (cwd None) >Thread-1932::DEBUG::2012-12-02 23:41:45,318::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1932::DEBUG::2012-12-02 23:41:45,318::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1932::DEBUG::2012-12-02 23:41:45,319::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1932::DEBUG::2012-12-02 23:41:45,319::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1932::DEBUG::2012-12-02 23:41:45,319::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1932::DEBUG::2012-12-02 23:41:45,319::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1932::INFO::2012-12-02 23:41:45,319::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-1932::DEBUG::2012-12-02 23:41:45,319::task::1151::TaskManager.Task::(prepare) Task=`4c338dbd-4cdb-416e-b127-9a94d502ca09`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-1932::DEBUG::2012-12-02 23:41:45,320::task::568::TaskManager.Task::(_updateState) Task=`4c338dbd-4cdb-416e-b127-9a94d502ca09`::moving from state preparing -> state finished >Thread-1932::DEBUG::2012-12-02 23:41:45,320::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1932::DEBUG::2012-12-02 23:41:45,320::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1932::DEBUG::2012-12-02 23:41:45,320::task::957::TaskManager.Task::(_decref) Task=`4c338dbd-4cdb-416e-b127-9a94d502ca09`::ref 0 aborting False >Thread-1933::DEBUG::2012-12-02 23:41:45,522::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1933::DEBUG::2012-12-02 23:41:45,523::task::568::TaskManager.Task::(_updateState) Task=`51c70f0e-de31-4cca-8fd7-fc9da551d7f3`::moving from state init -> state preparing >Thread-1933::INFO::2012-12-02 23:41:45,523::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', remove=False, options=None) >Thread-1933::WARNING::2012-12-02 23:41:45,523::hsm::887::Storage.HSM::(disconnectStoragePool) disconnect sp: 479cf195-bb99-45c5-b400-433d810eaa48 failed. Known pools {} >Thread-1933::INFO::2012-12-02 23:41:45,523::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None >Thread-1933::DEBUG::2012-12-02 23:41:45,523::task::1151::TaskManager.Task::(prepare) Task=`51c70f0e-de31-4cca-8fd7-fc9da551d7f3`::finished: None >Thread-1933::DEBUG::2012-12-02 23:41:45,523::task::568::TaskManager.Task::(_updateState) Task=`51c70f0e-de31-4cca-8fd7-fc9da551d7f3`::moving from state preparing -> state finished >Thread-1933::DEBUG::2012-12-02 23:41:45,523::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1933::DEBUG::2012-12-02 23:41:45,524::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1933::DEBUG::2012-12-02 23:41:45,524::task::957::TaskManager.Task::(_decref) Task=`51c70f0e-de31-4cca-8fd7-fc9da551d7f3`::ref 0 aborting False >Thread-1934::DEBUG::2012-12-02 23:41:45,675::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-1934::DEBUG::2012-12-02 23:41:45,676::task::568::TaskManager.Task::(_updateState) Task=`513ee25a-3791-49c7-ae24-bb3daac09534`::moving from state init -> state preparing >Thread-1934::INFO::2012-12-02 23:41:45,676::logUtils::37::dispatcher::(wrapper) Run and protect: reconstructMaster(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', poolName='dc', masterDom='5971512c-ee51-475e-b956-3872c38ce193', domDict={'225bca28-f0a0-4899-b137-32acd04f57d4': 'active', '2ee486e4-abfd-4f11-9b29-42bfc74d7064': 'active', '5971512c-ee51-475e-b956-3872c38ce193': 'active'}, masterVersion=3, lockPolicy=None, lockRenewalIntervalSec=5, leaseTimeSec=60, ioOpTimeoutSec=10, leaseRetries=3, hostId=2, options=None) >Thread-1934::INFO::2012-12-02 23:41:45,676::hsm::1483::Storage.HSM::(reconstructMaster) spUUID=479cf195-bb99-45c5-b400-433d810eaa48 master=5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::DEBUG::2012-12-02 23:41:45,676::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1934::DEBUG::2012-12-02 23:41:45,676::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1934::DEBUG::2012-12-02 23:41:45,677::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1934::DEBUG::2012-12-02 23:41:45,677::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1934::DEBUG::2012-12-02 23:41:45,677::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:41:45,706::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1934::DEBUG::2012-12-02 23:41:45,706::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1934::DEBUG::2012-12-02 23:41:46,342::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:41:46,396::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1934::DEBUG::2012-12-02 23:41:46,396::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,396::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,396::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,397::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,397::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,397::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,397::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1934::DEBUG::2012-12-02 23:41:46,397::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,399::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:41:46,606::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-1934::WARNING::2012-12-02 23:41:46,608::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-1934::DEBUG::2012-12-02 23:41:46,608::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:41:46,644::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::DEBUG::2012-12-02 23:41:46,644::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-1934::DEBUG::2012-12-02 23:41:46,684::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=ea2a83cb9e6d502aabc1445069efd526bb38c4e4'] >Thread-1934::DEBUG::2012-12-02 23:41:46,718::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-1934::DEBUG::2012-12-02 23:41:46,718::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '5971512c-ee51-475e-b956-3872c38ce193_imageNS' >Thread-1934::DEBUG::2012-12-02 23:41:46,718::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace '5971512c-ee51-475e-b956-3872c38ce193_volumeNS' >Thread-1934::INFO::2012-12-02 23:41:46,719::fileSD::260::Storage.StorageDomain::(validate) sdUUID=5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::DEBUG::2012-12-02 23:41:46,725::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=ea2a83cb9e6d502aabc1445069efd526bb38c4e4'] >Thread-1934::DEBUG::2012-12-02 23:41:46,726::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ccf5685b-3c7f-49f7-84bd-019581dbb1ea`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1934::DEBUG::2012-12-02 23:41:46,726::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1934::DEBUG::2012-12-02 23:41:46,726::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1934::DEBUG::2012-12-02 23:41:46,726::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ccf5685b-3c7f-49f7-84bd-019581dbb1ea`::Granted request >Thread-1934::DEBUG::2012-12-02 23:41:46,727::task::794::TaskManager.Task::(resourceAcquired) Task=`513ee25a-3791-49c7-ae24-bb3daac09534`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-1934::DEBUG::2012-12-02 23:41:46,727::task::957::TaskManager.Task::(_decref) Task=`513ee25a-3791-49c7-ae24-bb3daac09534`::ref 1 aborting False >Thread-1934::INFO::2012-12-02 23:41:46,727::sp::727::Storage.StoragePool::(reconstructMaster) spUUID=479cf195-bb99-45c5-b400-433d810eaa48 hostId=2 poolName=dc msdUUID=5971512c-ee51-475e-b956-3872c38ce193 domDict={'225bca28-f0a0-4899-b137-32acd04f57d4': 'active', '2ee486e4-abfd-4f11-9b29-42bfc74d7064': 'active', '5971512c-ee51-475e-b956-3872c38ce193': 'active'} masterVersion=3 leaseparams=({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5}) >Thread-1934::INFO::2012-12-02 23:41:46,727::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-1999::DEBUG::2012-12-02 23:41:51,416::task::568::TaskManager.Task::(_updateState) Task=`ffad3cce-16c6-4fed-8933-bfce1b8662ff`::moving from state init -> state preparing >Thread-1999::INFO::2012-12-02 23:41:51,416::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1999::INFO::2012-12-02 23:41:51,416::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} >Thread-1999::DEBUG::2012-12-02 23:41:51,416::task::1151::TaskManager.Task::(prepare) Task=`ffad3cce-16c6-4fed-8933-bfce1b8662ff`::finished: {} >Thread-1999::DEBUG::2012-12-02 23:41:51,416::task::568::TaskManager.Task::(_updateState) Task=`ffad3cce-16c6-4fed-8933-bfce1b8662ff`::moving from state preparing -> state finished >Thread-1999::DEBUG::2012-12-02 23:41:51,417::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1999::DEBUG::2012-12-02 23:41:51,417::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1999::DEBUG::2012-12-02 23:41:51,417::task::957::TaskManager.Task::(_decref) Task=`ffad3cce-16c6-4fed-8933-bfce1b8662ff`::ref 0 aborting False >Thread-2002::DEBUG::2012-12-02 23:41:54,326::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2002::DEBUG::2012-12-02 23:41:54,326::task::568::TaskManager.Task::(_updateState) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::moving from state init -> state preparing >Thread-2002::INFO::2012-12-02 23:41:54,327::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2002::ERROR::2012-12-02 23:41:54,327::task::833::TaskManager.Task::(_setError) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-2002::DEBUG::2012-12-02 23:41:54,327::task::852::TaskManager.Task::(_run) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::Task._run: 1ae00a11-9604-4158-96f5-00424f12174c ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2002::DEBUG::2012-12-02 23:41:54,327::task::1177::TaskManager.Task::(stop) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::stopping in state preparing (force False) >Thread-2002::DEBUG::2012-12-02 23:41:54,327::task::957::TaskManager.Task::(_decref) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::ref 1 aborting True >Thread-2002::INFO::2012-12-02 23:41:54,327::task::1134::TaskManager.Task::(prepare) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-2002::DEBUG::2012-12-02 23:41:54,328::task::1139::TaskManager.Task::(prepare) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::Prepare: aborted: Unknown pool id, pool not connected >Thread-2002::DEBUG::2012-12-02 23:41:54,328::task::957::TaskManager.Task::(_decref) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::ref 0 aborting True >Thread-2002::DEBUG::2012-12-02 23:41:54,328::task::892::TaskManager.Task::(_doAbort) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::Task._doAbort: force False >Thread-2002::DEBUG::2012-12-02 23:41:54,328::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2002::DEBUG::2012-12-02 23:41:54,328::task::568::TaskManager.Task::(_updateState) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::moving from state preparing -> state aborting >Thread-2002::DEBUG::2012-12-02 23:41:54,328::task::523::TaskManager.Task::(__state_aborting) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::_aborting: recover policy none >Thread-2002::DEBUG::2012-12-02 23:41:54,328::task::568::TaskManager.Task::(_updateState) Task=`1ae00a11-9604-4158-96f5-00424f12174c`::moving from state aborting -> state failed >Thread-2002::DEBUG::2012-12-02 23:41:54,328::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2002::DEBUG::2012-12-02 23:41:54,329::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2002::ERROR::2012-12-02 23:41:54,329::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-2003::DEBUG::2012-12-02 23:41:54,467::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2003::DEBUG::2012-12-02 23:41:54,467::task::568::TaskManager.Task::(_updateState) Task=`a1976abe-da50-4023-b5be-5947d97ccca2`::moving from state init -> state preparing >Thread-2003::INFO::2012-12-02 23:41:54,468::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-2003::DEBUG::2012-12-02 23:41:54,468::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-2003::DEBUG::2012-12-02 23:41:54,468::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-2003::INFO::2012-12-02 23:41:54,468::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-2003::DEBUG::2012-12-02 23:41:54,468::task::1151::TaskManager.Task::(prepare) Task=`a1976abe-da50-4023-b5be-5947d97ccca2`::finished: {'allTasksStatus': {}} >Thread-2003::DEBUG::2012-12-02 23:41:54,468::task::568::TaskManager.Task::(_updateState) Task=`a1976abe-da50-4023-b5be-5947d97ccca2`::moving from state preparing -> state finished >Thread-2003::DEBUG::2012-12-02 23:41:54,468::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2003::DEBUG::2012-12-02 23:41:54,468::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2003::DEBUG::2012-12-02 23:41:54,468::task::957::TaskManager.Task::(_decref) Task=`a1976abe-da50-4023-b5be-5947d97ccca2`::ref 0 aborting False >Thread-2004::DEBUG::2012-12-02 23:41:54,620::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2004::DEBUG::2012-12-02 23:41:54,620::task::568::TaskManager.Task::(_updateState) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::moving from state init -> state preparing >Thread-2004::INFO::2012-12-02 23:41:54,620::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2004::ERROR::2012-12-02 23:41:54,621::task::833::TaskManager.Task::(_setError) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-2004::DEBUG::2012-12-02 23:41:54,621::task::852::TaskManager.Task::(_run) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::Task._run: be60d026-68e3-4a28-9fce-a3e02f7cbeb9 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-2004::DEBUG::2012-12-02 23:41:54,621::task::1177::TaskManager.Task::(stop) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::stopping in state preparing (force False) >Thread-2004::DEBUG::2012-12-02 23:41:54,621::task::957::TaskManager.Task::(_decref) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::ref 1 aborting True >Thread-2004::INFO::2012-12-02 23:41:54,621::task::1134::TaskManager.Task::(prepare) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-2004::DEBUG::2012-12-02 23:41:54,621::task::1139::TaskManager.Task::(prepare) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::Prepare: aborted: Unknown pool id, pool not connected >Thread-2004::DEBUG::2012-12-02 23:41:54,622::task::957::TaskManager.Task::(_decref) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::ref 0 aborting True >Thread-2004::DEBUG::2012-12-02 23:41:54,622::task::892::TaskManager.Task::(_doAbort) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::Task._doAbort: force False >Thread-2004::DEBUG::2012-12-02 23:41:54,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2004::DEBUG::2012-12-02 23:41:54,622::task::568::TaskManager.Task::(_updateState) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::moving from state preparing -> state aborting >Thread-2004::DEBUG::2012-12-02 23:41:54,622::task::523::TaskManager.Task::(__state_aborting) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::_aborting: recover policy none >Thread-2004::DEBUG::2012-12-02 23:41:54,622::task::568::TaskManager.Task::(_updateState) Task=`be60d026-68e3-4a28-9fce-a3e02f7cbeb9`::moving from state aborting -> state failed >Thread-2004::DEBUG::2012-12-02 23:41:54,622::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2004::DEBUG::2012-12-02 23:41:54,622::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2004::ERROR::2012-12-02 23:41:54,622::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-2005::DEBUG::2012-12-02 23:41:54,825::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2005::DEBUG::2012-12-02 23:41:54,826::task::568::TaskManager.Task::(_updateState) Task=`a067ac0b-a20e-4da7-b5fe-2eeb913b8673`::moving from state init -> state preparing >Thread-2005::INFO::2012-12-02 23:41:54,826::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='5971512c-ee51-475e-b956-3872c38ce193', masterVersion=3, options=None) >Thread-2005::DEBUG::2012-12-02 23:41:54,826::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a5d9d8f-b47f-4688-8c09-3ff8ad8eea71`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2005::DEBUG::2012-12-02 23:41:54,826::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-2005::DEBUG::2012-12-02 23:41:54,826::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is currently locked, Entering queue (1 in queue) >Thread-1934::DEBUG::2012-12-02 23:42:07,730::safelease::178::SANLock::(acquireHostId) Host id for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully acquired (id: 2) >Thread-1934::INFO::2012-12-02 23:42:07,730::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-1934::DEBUG::2012-12-02 23:42:08,105::safelease::235::SANLock::(acquire) Cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully acquired (id: 2) >Thread-1934::INFO::2012-12-02 23:42:08,105::sp::702::Storage.StoragePool::(createMaster) setting master domain for spUUID 479cf195-bb99-45c5-b400-433d810eaa48 on sdUUID=5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::DEBUG::2012-12-02 23:42:08,105::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,106::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,111::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Regular', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=ea2a83cb9e6d502aabc1445069efd526bb38c4e4'] >Thread-1934::DEBUG::2012-12-02 23:42:08,111::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,111::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,112::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,112::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-1934::DEBUG::2012-12-02 23:42:08,112::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=2e6254cda857483b030ccaeca55706f9b9af9067'] >Thread-1934::DEBUG::2012-12-02 23:42:08,161::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-1934::INFO::2012-12-02 23:42:08,161::sp::757::Storage.StoragePool::(reconstructMaster) Set storage pool domains: {'225bca28-f0a0-4899-b137-32acd04f57d4': 'Active', '2ee486e4-abfd-4f11-9b29-42bfc74d7064': 'Active', '5971512c-ee51-475e-b956-3872c38ce193': 'Active'} >Thread-1934::DEBUG::2012-12-02 23:42:08,161::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,162::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-1934::DEBUG::2012-12-02 23:42:08,162::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-1934::DEBUG::2012-12-02 23:42:08,217::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-1934::DEBUG::2012-12-02 23:42:08,218::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,218::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,218::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,218::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,218::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,218::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,219::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-1934::DEBUG::2012-12-02 23:42:08,219::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1934::DEBUG::2012-12-02 23:42:08,219::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-1934::DEBUG::2012-12-02 23:42:08,219::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1934::DEBUG::2012-12-02 23:42:08,219::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:42:08,243::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1934::DEBUG::2012-12-02 23:42:08,244::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1934::DEBUG::2012-12-02 23:42:08,876::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:42:08,950::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-1934::DEBUG::2012-12-02 23:42:08,950::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,950::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,951::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,951::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,951::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,951::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,951::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1934::DEBUG::2012-12-02 23:42:08,951::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:08,953::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:42:09,162::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-1934::WARNING::2012-12-02 23:42:09,163::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-1934::DEBUG::2012-12-02 23:42:09,164::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:09,201::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::DEBUG::2012-12-02 23:42:09,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-1934::DEBUG::2012-12-02 23:42:09,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-1934::DEBUG::2012-12-02 23:42:09,214::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-1934::WARNING::2012-12-02 23:42:09,214::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >Thread-1934::WARNING::2012-12-02 23:42:09,214::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >Thread-1934::DEBUG::2012-12-02 23:42:09,214::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 5971512c-ee51-475e-b956-3872c38ce193 verified, version 3 >Thread-1934::DEBUG::2012-12-02 23:42:09,215::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-1934::DEBUG::2012-12-02 23:42:09,215::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-1934::DEBUG::2012-12-02 23:42:09,221::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-1934::DEBUG::2012-12-02 23:42:09,222::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:09,222::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:42:09,426::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1934::WARNING::2012-12-02 23:42:09,427::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1934::DEBUG::2012-12-02 23:42:09,428::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1934::ERROR::2012-12-02 23:42:09,457::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-1934::DEBUG::2012-12-02 23:42:09,458::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:09,458::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:42:09,663::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1934::WARNING::2012-12-02 23:42:09,665::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1934::DEBUG::2012-12-02 23:42:09,665::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1934::ERROR::2012-12-02 23:42:09,693::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-1934::DEBUG::2012-12-02 23:42:09,694::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `479cf195-bb99-45c5-b400-433d810eaa48` started monitoring domain `5971512c-ee51-475e-b956-3872c38ce193` >Thread-1934::DEBUG::2012-12-02 23:42:09,694::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-1934::DEBUG::2012-12-02 23:42:09,699::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1934::DEBUG::2012-12-02 23:42:09,701::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1934::DEBUG::2012-12-02 23:42:09,955::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1934::WARNING::2012-12-02 23:42:09,957::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found', ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1934::DEBUG::2012-12-02 23:42:09,957::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1934::INFO::2012-12-02 23:42:09,986::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::INFO::2012-12-02 23:42:09,986::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-1934::INFO::2012-12-02 23:42:09,987::safelease::239::SANLock::(release) Releasing cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 >Thread-1934::DEBUG::2012-12-02 23:42:10,003::safelease::250::SANLock::(release) Cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully released >Thread-1934::INFO::2012-12-02 23:42:10,003::logUtils::39::dispatcher::(wrapper) Run and protect: reconstructMaster, Return response: None >Thread-1934::DEBUG::2012-12-02 23:42:10,003::task::1151::TaskManager.Task::(prepare) Task=`513ee25a-3791-49c7-ae24-bb3daac09534`::finished: None >Thread-1934::DEBUG::2012-12-02 23:42:10,003::task::568::TaskManager.Task::(_updateState) Task=`513ee25a-3791-49c7-ae24-bb3daac09534`::moving from state preparing -> state finished >Thread-2192::DEBUG::2012-12-02 23:42:10,004::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 5971512c-ee51-475e-b956-3872c38ce193 is stopping >Thread-1934::DEBUG::2012-12-02 23:42:10,004::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-2192::INFO::2012-12-02 23:42:10,004::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-1934::DEBUG::2012-12-02 23:42:10,004::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1934::DEBUG::2012-12-02 23:42:10,004::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1934::DEBUG::2012-12-02 23:42:10,005::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1934::DEBUG::2012-12-02 23:42:10,005::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1934::DEBUG::2012-12-02 23:42:10,005::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' has 1 requests in queue. Handling top request. >Thread-1934::DEBUG::2012-12-02 23:42:10,005::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a5d9d8f-b47f-4688-8c09-3ff8ad8eea71`::Granted request >Thread-1934::DEBUG::2012-12-02 23:42:10,005::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.479cf195-bb99-45c5-b400-433d810eaa48 - exclusive: granted' was granted >Thread-1934::DEBUG::2012-12-02 23:42:10,006::task::957::TaskManager.Task::(_decref) Task=`513ee25a-3791-49c7-ae24-bb3daac09534`::ref 0 aborting False >Thread-2005::INFO::2012-12-02 23:42:10,006::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 5971512c-ee51-475e-b956-3872c38ce193 (ver = 3) >Thread-2005::DEBUG::2012-12-02 23:42:10,007::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,007::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,007::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,007::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,007::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,008::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,008::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-2005::DEBUG::2012-12-02 23:42:10,008::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2005::DEBUG::2012-12-02 23:42:10,008::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-2005::DEBUG::2012-12-02 23:42:10,008::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2005::DEBUG::2012-12-02 23:42:10,008::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-2005::DEBUG::2012-12-02 23:42:10,034::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-2005::DEBUG::2012-12-02 23:42:10,035::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2256::DEBUG::2012-12-02 23:42:10,282::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2256::DEBUG::2012-12-02 23:42:10,283::task::568::TaskManager.Task::(_updateState) Task=`4923de99-7416-42cb-a2e2-bf01ada97a65`::moving from state init -> state preparing >Thread-2256::INFO::2012-12-02 23:42:10,283::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='5971512c-ee51-475e-b956-3872c38ce193', masterVersion=3, options=None) >Thread-2256::DEBUG::2012-12-02 23:42:10,283::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a7194662-39a9-466f-911d-88ca9185d8cb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2256::DEBUG::2012-12-02 23:42:10,283::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-2256::DEBUG::2012-12-02 23:42:10,284::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is currently locked, Entering queue (1 in queue) >Thread-2005::DEBUG::2012-12-02 23:42:10,667::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-2005::DEBUG::2012-12-02 23:42:10,737::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-2005::DEBUG::2012-12-02 23:42:10,737::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,738::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,738::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,738::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,738::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,738::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,739::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2005::DEBUG::2012-12-02 23:42:10,739::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,740::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-2005::DEBUG::2012-12-02 23:42:10,944::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-2005::WARNING::2012-12-02 23:42:10,945::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-2005::DEBUG::2012-12-02 23:42:10,946::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:10,983::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-2005::DEBUG::2012-12-02 23:42:10,983::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-2005::DEBUG::2012-12-02 23:42:10,992::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-2005::DEBUG::2012-12-02 23:42:10,994::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-2005::WARNING::2012-12-02 23:42:10,994::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >Thread-2005::WARNING::2012-12-02 23:42:10,994::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >Thread-2005::DEBUG::2012-12-02 23:42:10,994::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 5971512c-ee51-475e-b956-3872c38ce193 verified, version 3 >Thread-2005::DEBUG::2012-12-02 23:42:10,994::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-2005::DEBUG::2012-12-02 23:42:10,995::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2005::DEBUG::2012-12-02 23:42:11,002::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-2005::DEBUG::2012-12-02 23:42:11,002::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:11,002::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-2005::DEBUG::2012-12-02 23:42:11,210::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-2005::WARNING::2012-12-02 23:42:11,212::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-2005::DEBUG::2012-12-02 23:42:11,212::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2005::ERROR::2012-12-02 23:42:11,246::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-2005::DEBUG::2012-12-02 23:42:11,246::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:11,246::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2005::DEBUG::2012-12-02 23:42:11,454::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2005::WARNING::2012-12-02 23:42:11,455::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2005::DEBUG::2012-12-02 23:42:11,456::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2005::ERROR::2012-12-02 23:42:11,485::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2005::DEBUG::2012-12-02 23:42:11,486::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `479cf195-bb99-45c5-b400-433d810eaa48` started monitoring domain `5971512c-ee51-475e-b956-3872c38ce193` >Thread-2005::DEBUG::2012-12-02 23:42:11,486::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2005::DEBUG::2012-12-02 23:42:11,488::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2005::DEBUG::2012-12-02 23:42:11,489::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2005::DEBUG::2012-12-02 23:42:11,746::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2005::WARNING::2012-12-02 23:42:11,748::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found', ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2005::DEBUG::2012-12-02 23:42:11,748::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2005::INFO::2012-12-02 23:42:11,776::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/5971512c-ee51-475e-b956-3872c38ce193 >Thread-2005::INFO::2012-12-02 23:42:11,776::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-2005::DEBUG::2012-12-02 23:42:11,777::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-2005::DEBUG::2012-12-02 23:42:11,777::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-2005::DEBUG::2012-12-02 23:42:11,777::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-2005::DEBUG::2012-12-02 23:42:11,777::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' has 1 requests in queue. Handling top request. >Thread-2005::DEBUG::2012-12-02 23:42:11,777::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a7194662-39a9-466f-911d-88ca9185d8cb`::Granted request >Thread-2005::DEBUG::2012-12-02 23:42:11,778::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.479cf195-bb99-45c5-b400-433d810eaa48 - shared: granted' was granted >Thread-2005::DEBUG::2012-12-02 23:42:11,778::resourceManager::597::ResourceManager::(releaseResource) This is a shared lock. Granting all shared requests >Thread-2005::INFO::2012-12-02 23:42:11,778::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-2256::DEBUG::2012-12-02 23:42:11,778::task::794::TaskManager.Task::(resourceAcquired) Task=`4923de99-7416-42cb-a2e2-bf01ada97a65`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-2005::DEBUG::2012-12-02 23:42:11,778::task::1151::TaskManager.Task::(prepare) Task=`a067ac0b-a20e-4da7-b5fe-2eeb913b8673`::finished: True >Thread-2256::DEBUG::2012-12-02 23:42:11,778::task::957::TaskManager.Task::(_decref) Task=`4923de99-7416-42cb-a2e2-bf01ada97a65`::ref 1 aborting False >Thread-2005::DEBUG::2012-12-02 23:42:11,779::task::568::TaskManager.Task::(_updateState) Task=`a067ac0b-a20e-4da7-b5fe-2eeb913b8673`::moving from state preparing -> state finished >Thread-2256::INFO::2012-12-02 23:42:11,779::fileSD::260::Storage.StorageDomain::(validate) sdUUID=5971512c-ee51-475e-b956-3872c38ce193 >Thread-2005::DEBUG::2012-12-02 23:42:11,779::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2005::DEBUG::2012-12-02 23:42:11,779::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2005::DEBUG::2012-12-02 23:42:11,780::task::957::TaskManager.Task::(_decref) Task=`a067ac0b-a20e-4da7-b5fe-2eeb913b8673`::ref 0 aborting False >Thread-2256::DEBUG::2012-12-02 23:42:11,786::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-2256::DEBUG::2012-12-02 23:42:11,786::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:11,787::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:11,787::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:11,787::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:11,787::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:11,787::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:11,787::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-2256::DEBUG::2012-12-02 23:42:11,788::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2256::DEBUG::2012-12-02 23:42:11,788::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-2256::DEBUG::2012-12-02 23:42:11,788::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2256::DEBUG::2012-12-02 23:42:11,788::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-2256::DEBUG::2012-12-02 23:42:11,812::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-2256::DEBUG::2012-12-02 23:42:11,812::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2505::DEBUG::2012-12-02 23:42:11,922::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2505::DEBUG::2012-12-02 23:42:11,923::task::568::TaskManager.Task::(_updateState) Task=`f424502b-f6f7-472a-9da9-f4842f13bc0f`::moving from state init -> state preparing >Thread-2505::INFO::2012-12-02 23:42:11,923::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2192::DEBUG::2012-12-02 23:42:12,005::safelease::193::SANLock::(releaseHostId) Host id for domain 5971512c-ee51-475e-b956-3872c38ce193 released successfully (id: 2) >Thread-2256::DEBUG::2012-12-02 23:42:12,445::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-2256::DEBUG::2012-12-02 23:42:12,520::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-2256::DEBUG::2012-12-02 23:42:12,521::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,521::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,521::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,521::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,521::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,521::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,522::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2256::DEBUG::2012-12-02 23:42:12,522::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,523::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-2256::DEBUG::2012-12-02 23:42:12,731::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-2256::WARNING::2012-12-02 23:42:12,733::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-2256::DEBUG::2012-12-02 23:42:12,733::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,767::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-2256::DEBUG::2012-12-02 23:42:12,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-2256::DEBUG::2012-12-02 23:42:12,774::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-2256::DEBUG::2012-12-02 23:42:12,776::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-2256::WARNING::2012-12-02 23:42:12,776::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >Thread-2256::WARNING::2012-12-02 23:42:12,776::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >Thread-2505::INFO::2012-12-02 23:42:12,776::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-2256::DEBUG::2012-12-02 23:42:12,776::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 5971512c-ee51-475e-b956-3872c38ce193 verified, version 3 >Thread-2505::DEBUG::2012-12-02 23:42:12,777::task::1151::TaskManager.Task::(prepare) Task=`f424502b-f6f7-472a-9da9-f4842f13bc0f`::finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': -1}} >Thread-2256::DEBUG::2012-12-02 23:42:12,777::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-2505::DEBUG::2012-12-02 23:42:12,777::task::568::TaskManager.Task::(_updateState) Task=`f424502b-f6f7-472a-9da9-f4842f13bc0f`::moving from state preparing -> state finished >Thread-2256::DEBUG::2012-12-02 23:42:12,777::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2505::DEBUG::2012-12-02 23:42:12,778::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2505::DEBUG::2012-12-02 23:42:12,778::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2505::DEBUG::2012-12-02 23:42:12,778::task::957::TaskManager.Task::(_decref) Task=`f424502b-f6f7-472a-9da9-f4842f13bc0f`::ref 0 aborting False >Thread-2256::DEBUG::2012-12-02 23:42:12,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >Thread-2256::DEBUG::2012-12-02 23:42:12,785::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:12,785::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-2256::DEBUG::2012-12-02 23:42:12,993::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-2256::WARNING::2012-12-02 23:42:12,995::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-2256::DEBUG::2012-12-02 23:42:12,995::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2256::ERROR::2012-12-02 23:42:13,023::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-2256::DEBUG::2012-12-02 23:42:13,024::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:13,024::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2631::DEBUG::2012-12-02 23:42:13,051::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2631::DEBUG::2012-12-02 23:42:13,051::task::568::TaskManager.Task::(_updateState) Task=`fec5421a-317e-429c-a472-8bebef790541`::moving from state init -> state preparing >Thread-2631::INFO::2012-12-02 23:42:13,051::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', prevID=-1, prevLVER='-1', recoveryMode=None, scsiFencing='false', maxHostID=250, domVersion='3', options=None) >Thread-2631::DEBUG::2012-12-02 23:42:13,052::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ada5ef78-d864-4990-a2a1-d1fc7a1a05cc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2631::DEBUG::2012-12-02 23:42:13,052::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-2631::DEBUG::2012-12-02 23:42:13,052::resourceManager::510::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is currently locked, Entering queue (1 in queue) >Thread-2256::DEBUG::2012-12-02 23:42:13,233::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2256::WARNING::2012-12-02 23:42:13,234::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2256::DEBUG::2012-12-02 23:42:13,235::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2256::ERROR::2012-12-02 23:42:13,263::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2256::DEBUG::2012-12-02 23:42:13,264::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2256::DEBUG::2012-12-02 23:42:13,264::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2256::DEBUG::2012-12-02 23:42:13,265::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2256::DEBUG::2012-12-02 23:42:13,517::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2256::WARNING::2012-12-02 23:42:13,518::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found', ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2256::DEBUG::2012-12-02 23:42:13,518::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2256::INFO::2012-12-02 23:42:13,546::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/5971512c-ee51-475e-b956-3872c38ce193 >Thread-2256::INFO::2012-12-02 23:42:13,547::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-2256::INFO::2012-12-02 23:42:13,547::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-2256::DEBUG::2012-12-02 23:42:13,547::task::1151::TaskManager.Task::(prepare) Task=`4923de99-7416-42cb-a2e2-bf01ada97a65`::finished: None >Thread-2256::DEBUG::2012-12-02 23:42:13,547::task::568::TaskManager.Task::(_updateState) Task=`4923de99-7416-42cb-a2e2-bf01ada97a65`::moving from state preparing -> state finished >Thread-2256::DEBUG::2012-12-02 23:42:13,547::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-2256::DEBUG::2012-12-02 23:42:13,548::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2256::DEBUG::2012-12-02 23:42:13,548::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-2256::DEBUG::2012-12-02 23:42:13,548::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-2256::DEBUG::2012-12-02 23:42:13,548::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-2256::DEBUG::2012-12-02 23:42:13,548::resourceManager::568::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' has 1 requests in queue. Handling top request. >Thread-2256::DEBUG::2012-12-02 23:42:13,548::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ada5ef78-d864-4990-a2a1-d1fc7a1a05cc`::Granted request >Thread-2256::DEBUG::2012-12-02 23:42:13,549::resourceManager::589::ResourceManager::(releaseResource) Request 'Request for Storage.479cf195-bb99-45c5-b400-433d810eaa48 - exclusive: granted' was granted >Thread-2256::DEBUG::2012-12-02 23:42:13,549::task::957::TaskManager.Task::(_decref) Task=`4923de99-7416-42cb-a2e2-bf01ada97a65`::ref 0 aborting False >Thread-2631::DEBUG::2012-12-02 23:42:13,549::task::794::TaskManager.Task::(resourceAcquired) Task=`fec5421a-317e-429c-a472-8bebef790541`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-2631::DEBUG::2012-12-02 23:42:13,550::task::957::TaskManager.Task::(_decref) Task=`fec5421a-317e-429c-a472-8bebef790541`::ref 1 aborting False >Thread-2631::INFO::2012-12-02 23:42:13,550::logUtils::39::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-2631::DEBUG::2012-12-02 23:42:13,550::task::1146::TaskManager.Task::(prepare) Task=`fec5421a-317e-429c-a472-8bebef790541`::Prepare: 1 jobs exist, move to acquiring >Thread-2631::DEBUG::2012-12-02 23:42:13,550::task::568::TaskManager.Task::(_updateState) Task=`fec5421a-317e-429c-a472-8bebef790541`::moving from state preparing -> state acquiring >Thread-2631::DEBUG::2012-12-02 23:42:13,551::task::568::TaskManager.Task::(_updateState) Task=`fec5421a-317e-429c-a472-8bebef790541`::moving from state acquiring -> state queued >Thread-2631::DEBUG::2012-12-02 23:42:13,551::taskManager::50::TaskManager::(_queueTask) queuing task: fec5421a-317e-429c-a472-8bebef790541 >Thread-2631::DEBUG::2012-12-02 23:42:13,551::taskManager::56::TaskManager::(_queueTask) task queued: fec5421a-317e-429c-a472-8bebef790541 >Thread-1::DEBUG::2012-12-02 23:42:13,551::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-2631::DEBUG::2012-12-02 23:42:13,551::task::1148::TaskManager.Task::(prepare) Task=`fec5421a-317e-429c-a472-8bebef790541`::returning >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,551::threadPool::212::Misc.ThreadPool.WorkerThread::(run) Task: fec5421a-317e-429c-a472-8bebef790541 running: <bound method Task.commit of <storage.task.Task instance at 0x7fed4445dc68>> with: None >Thread-2631::DEBUG::2012-12-02 23:42:13,552::task::957::TaskManager.Task::(_decref) Task=`fec5421a-317e-429c-a472-8bebef790541`::ref 0 aborting False >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,552::task::1159::TaskManager.Task::(commit) Task=`fec5421a-317e-429c-a472-8bebef790541`::committing task: fec5421a-317e-429c-a472-8bebef790541 >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,552::task::568::TaskManager.Task::(_updateState) Task=`fec5421a-317e-429c-a472-8bebef790541`::moving from state queued -> state running >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,553::task::872::TaskManager.Task::(_runJobs) Task=`fec5421a-317e-429c-a472-8bebef790541`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fed4444e5d0>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,553::task::304::TaskManager.Task::(run) Task=`fec5421a-317e-429c-a472-8bebef790541`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x7fed4444e5d0>> (args: (-1, '-1', 'false', 250, 3) kwargs: {}) callback None >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,553::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,553::misc::1063::SamplingMethod::(__call__) Got in to sampling method >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,560::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,560::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,561::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-2755::DEBUG::2012-12-02 23:42:13,709::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2755::DEBUG::2012-12-02 23:42:13,710::task::568::TaskManager.Task::(_updateState) Task=`3b6eda9d-0124-4883-9568-b66834ccc522`::moving from state init -> state preparing >Thread-2755::INFO::2012-12-02 23:42:13,710::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2252', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21', 'port': ''}], options=None) >Thread-2755::DEBUG::2012-12-02 23:42:13,711::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2252' (cwd None) >Thread-2755::DEBUG::2012-12-02 23:42:13,742::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-2755::DEBUG::2012-12-02 23:42:13,742::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2755::DEBUG::2012-12-02 23:42:13,742::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-2755::DEBUG::2012-12-02 23:42:13,742::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-2755::DEBUG::2012-12-02 23:42:13,742::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,770::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:13,771::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,772::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:13,776::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-2755::DEBUG::2012-12-02 23:42:13,778::misc::1071::SamplingMethod::(__call__) Returning last result >fec5421a-317e-429c-a472-8bebef790541::ERROR::2012-12-02 23:42:13,802::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,803::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:13,803::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:14,012::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:14,013::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:14,013::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >fec5421a-317e-429c-a472-8bebef790541::ERROR::2012-12-02 23:42:14,042::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:14,043::misc::1071::SamplingMethod::(__call__) Returning last result >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:14,051::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >fec5421a-317e-429c-a472-8bebef790541::INFO::2012-12-02 23:42:14,051::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-2755::DEBUG::2012-12-02 23:42:14,414::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-2755::DEBUG::2012-12-02 23:42:14,492::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-2755::DEBUG::2012-12-02 23:42:14,493::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:14,493::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:14,493::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:14,493::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:14,493::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:14,494::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-2755::DEBUG::2012-12-02 23:42:14,494::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-2755::INFO::2012-12-02 23:42:14,494::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-2755::DEBUG::2012-12-02 23:42:14,494::task::1151::TaskManager.Task::(prepare) Task=`3b6eda9d-0124-4883-9568-b66834ccc522`::finished: {'statuslist': [{'status': 0, 'id': 'dee4a670-d422-4ad2-aa97-a84016f50e21'}]} >Thread-2755::DEBUG::2012-12-02 23:42:14,494::task::568::TaskManager.Task::(_updateState) Task=`3b6eda9d-0124-4883-9568-b66834ccc522`::moving from state preparing -> state finished >Thread-2755::DEBUG::2012-12-02 23:42:14,494::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2755::DEBUG::2012-12-02 23:42:14,495::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2755::DEBUG::2012-12-02 23:42:14,495::task::957::TaskManager.Task::(_decref) Task=`3b6eda9d-0124-4883-9568-b66834ccc522`::ref 0 aborting False >Thread-2878::DEBUG::2012-12-02 23:42:14,694::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2878::DEBUG::2012-12-02 23:42:14,695::task::568::TaskManager.Task::(_updateState) Task=`975ba92d-6698-4610-9c46-bc9bb040fb5e`::moving from state init -> state preparing >Thread-2878::INFO::2012-12-02 23:42:14,695::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2878::DEBUG::2012-12-02 23:42:14,695::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2878::DEBUG::2012-12-02 23:42:14,695::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2878::INFO::2012-12-02 23:42:14,695::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2878::DEBUG::2012-12-02 23:42:14,695::task::1151::TaskManager.Task::(prepare) Task=`975ba92d-6698-4610-9c46-bc9bb040fb5e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2878::DEBUG::2012-12-02 23:42:14,696::task::568::TaskManager.Task::(_updateState) Task=`975ba92d-6698-4610-9c46-bc9bb040fb5e`::moving from state preparing -> state finished >Thread-2878::DEBUG::2012-12-02 23:42:14,696::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2878::DEBUG::2012-12-02 23:42:14,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2878::DEBUG::2012-12-02 23:42:14,696::task::957::TaskManager.Task::(_decref) Task=`975ba92d-6698-4610-9c46-bc9bb040fb5e`::ref 0 aborting False >Thread-2880::DEBUG::2012-12-02 23:42:15,839::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2880::DEBUG::2012-12-02 23:42:15,839::task::568::TaskManager.Task::(_updateState) Task=`4df7f75a-e4f6-4f7d-8091-835d213b4e5e`::moving from state init -> state preparing >Thread-2880::INFO::2012-12-02 23:42:15,840::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2880::DEBUG::2012-12-02 23:42:15,840::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2880::DEBUG::2012-12-02 23:42:15,840::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2880::INFO::2012-12-02 23:42:15,840::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2880::DEBUG::2012-12-02 23:42:15,840::task::1151::TaskManager.Task::(prepare) Task=`4df7f75a-e4f6-4f7d-8091-835d213b4e5e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2880::DEBUG::2012-12-02 23:42:15,840::task::568::TaskManager.Task::(_updateState) Task=`4df7f75a-e4f6-4f7d-8091-835d213b4e5e`::moving from state preparing -> state finished >Thread-2880::DEBUG::2012-12-02 23:42:15,840::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2880::DEBUG::2012-12-02 23:42:15,840::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2880::DEBUG::2012-12-02 23:42:15,841::task::957::TaskManager.Task::(_decref) Task=`4df7f75a-e4f6-4f7d-8091-835d213b4e5e`::ref 0 aborting False >Thread-2881::DEBUG::2012-12-02 23:42:16,973::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2881::DEBUG::2012-12-02 23:42:16,973::task::568::TaskManager.Task::(_updateState) Task=`515b2237-1b36-4616-8f7c-bbb60e5b6922`::moving from state init -> state preparing >Thread-2881::INFO::2012-12-02 23:42:16,973::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2881::DEBUG::2012-12-02 23:42:16,974::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2881::DEBUG::2012-12-02 23:42:16,974::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2881::INFO::2012-12-02 23:42:16,974::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2881::DEBUG::2012-12-02 23:42:16,974::task::1151::TaskManager.Task::(prepare) Task=`515b2237-1b36-4616-8f7c-bbb60e5b6922`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2881::DEBUG::2012-12-02 23:42:16,974::task::568::TaskManager.Task::(_updateState) Task=`515b2237-1b36-4616-8f7c-bbb60e5b6922`::moving from state preparing -> state finished >Thread-2881::DEBUG::2012-12-02 23:42:16,974::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2881::DEBUG::2012-12-02 23:42:16,974::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2881::DEBUG::2012-12-02 23:42:16,975::task::957::TaskManager.Task::(_decref) Task=`515b2237-1b36-4616-8f7c-bbb60e5b6922`::ref 0 aborting False >Thread-2882::DEBUG::2012-12-02 23:42:17,197::task::568::TaskManager.Task::(_updateState) Task=`a1f59d06-15ac-411d-a4bc-1ed148873b87`::moving from state init -> state preparing >Thread-2882::INFO::2012-12-02 23:42:17,197::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2882::INFO::2012-12-02 23:42:17,197::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00693392753601', 'lastCheck': 1354484531.5073919, 'code': 0, 'valid': True}} >Thread-2882::DEBUG::2012-12-02 23:42:17,197::task::1151::TaskManager.Task::(prepare) Task=`a1f59d06-15ac-411d-a4bc-1ed148873b87`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00693392753601', 'lastCheck': 1354484531.5073919, 'code': 0, 'valid': True}} >Thread-2882::DEBUG::2012-12-02 23:42:17,197::task::568::TaskManager.Task::(_updateState) Task=`a1f59d06-15ac-411d-a4bc-1ed148873b87`::moving from state preparing -> state finished >Thread-2882::DEBUG::2012-12-02 23:42:17,197::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2882::DEBUG::2012-12-02 23:42:17,197::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2882::DEBUG::2012-12-02 23:42:17,197::task::957::TaskManager.Task::(_decref) Task=`a1f59d06-15ac-411d-a4bc-1ed148873b87`::ref 0 aborting False >Thread-2884::DEBUG::2012-12-02 23:42:18,113::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2884::DEBUG::2012-12-02 23:42:18,113::task::568::TaskManager.Task::(_updateState) Task=`9156791b-677f-490b-8766-cc93fba7ba4b`::moving from state init -> state preparing >Thread-2884::INFO::2012-12-02 23:42:18,113::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2884::DEBUG::2012-12-02 23:42:18,113::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2884::DEBUG::2012-12-02 23:42:18,114::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2884::INFO::2012-12-02 23:42:18,114::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2884::DEBUG::2012-12-02 23:42:18,114::task::1151::TaskManager.Task::(prepare) Task=`9156791b-677f-490b-8766-cc93fba7ba4b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2884::DEBUG::2012-12-02 23:42:18,114::task::568::TaskManager.Task::(_updateState) Task=`9156791b-677f-490b-8766-cc93fba7ba4b`::moving from state preparing -> state finished >Thread-2884::DEBUG::2012-12-02 23:42:18,114::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2884::DEBUG::2012-12-02 23:42:18,114::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2884::DEBUG::2012-12-02 23:42:18,114::task::957::TaskManager.Task::(_decref) Task=`9156791b-677f-490b-8766-cc93fba7ba4b`::ref 0 aborting False >Thread-2885::DEBUG::2012-12-02 23:42:19,248::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2885::DEBUG::2012-12-02 23:42:19,248::task::568::TaskManager.Task::(_updateState) Task=`c1f1320e-7b79-44f9-97cc-1cf8664aea94`::moving from state init -> state preparing >Thread-2885::INFO::2012-12-02 23:42:19,249::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2885::DEBUG::2012-12-02 23:42:19,249::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2885::DEBUG::2012-12-02 23:42:19,249::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2885::INFO::2012-12-02 23:42:19,249::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2885::DEBUG::2012-12-02 23:42:19,249::task::1151::TaskManager.Task::(prepare) Task=`c1f1320e-7b79-44f9-97cc-1cf8664aea94`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2885::DEBUG::2012-12-02 23:42:19,249::task::568::TaskManager.Task::(_updateState) Task=`c1f1320e-7b79-44f9-97cc-1cf8664aea94`::moving from state preparing -> state finished >Thread-2885::DEBUG::2012-12-02 23:42:19,249::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2885::DEBUG::2012-12-02 23:42:19,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2885::DEBUG::2012-12-02 23:42:19,250::task::957::TaskManager.Task::(_decref) Task=`c1f1320e-7b79-44f9-97cc-1cf8664aea94`::ref 0 aborting False >Thread-2887::DEBUG::2012-12-02 23:42:20,387::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2887::DEBUG::2012-12-02 23:42:20,388::task::568::TaskManager.Task::(_updateState) Task=`55f5098a-0cbf-4f56-a0bf-53b8a080d24b`::moving from state init -> state preparing >Thread-2887::INFO::2012-12-02 23:42:20,388::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2887::DEBUG::2012-12-02 23:42:20,388::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2887::DEBUG::2012-12-02 23:42:20,388::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2887::INFO::2012-12-02 23:42:20,388::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2887::DEBUG::2012-12-02 23:42:20,388::task::1151::TaskManager.Task::(prepare) Task=`55f5098a-0cbf-4f56-a0bf-53b8a080d24b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2887::DEBUG::2012-12-02 23:42:20,388::task::568::TaskManager.Task::(_updateState) Task=`55f5098a-0cbf-4f56-a0bf-53b8a080d24b`::moving from state preparing -> state finished >Thread-2887::DEBUG::2012-12-02 23:42:20,389::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2887::DEBUG::2012-12-02 23:42:20,389::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2887::DEBUG::2012-12-02 23:42:20,389::task::957::TaskManager.Task::(_decref) Task=`55f5098a-0cbf-4f56-a0bf-53b8a080d24b`::ref 0 aborting False >Thread-2888::DEBUG::2012-12-02 23:42:21,527::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2888::DEBUG::2012-12-02 23:42:21,528::task::568::TaskManager.Task::(_updateState) Task=`e7370980-e36f-4227-b9df-74e5e14dcb42`::moving from state init -> state preparing >Thread-2888::INFO::2012-12-02 23:42:21,528::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2888::DEBUG::2012-12-02 23:42:21,528::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2888::DEBUG::2012-12-02 23:42:21,528::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2888::INFO::2012-12-02 23:42:21,529::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2888::DEBUG::2012-12-02 23:42:21,529::task::1151::TaskManager.Task::(prepare) Task=`e7370980-e36f-4227-b9df-74e5e14dcb42`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2888::DEBUG::2012-12-02 23:42:21,529::task::568::TaskManager.Task::(_updateState) Task=`e7370980-e36f-4227-b9df-74e5e14dcb42`::moving from state preparing -> state finished >Thread-2888::DEBUG::2012-12-02 23:42:21,529::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2888::DEBUG::2012-12-02 23:42:21,529::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2888::DEBUG::2012-12-02 23:42:21,529::task::957::TaskManager.Task::(_decref) Task=`e7370980-e36f-4227-b9df-74e5e14dcb42`::ref 0 aborting False >Thread-2890::DEBUG::2012-12-02 23:42:22,668::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2890::DEBUG::2012-12-02 23:42:22,668::task::568::TaskManager.Task::(_updateState) Task=`a2fb0ae8-f2f4-493c-afe9-1f91adc64d9e`::moving from state init -> state preparing >Thread-2890::INFO::2012-12-02 23:42:22,668::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2890::DEBUG::2012-12-02 23:42:22,668::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2890::DEBUG::2012-12-02 23:42:22,668::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2890::INFO::2012-12-02 23:42:22,669::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2890::DEBUG::2012-12-02 23:42:22,669::task::1151::TaskManager.Task::(prepare) Task=`a2fb0ae8-f2f4-493c-afe9-1f91adc64d9e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2890::DEBUG::2012-12-02 23:42:22,669::task::568::TaskManager.Task::(_updateState) Task=`a2fb0ae8-f2f4-493c-afe9-1f91adc64d9e`::moving from state preparing -> state finished >Thread-2890::DEBUG::2012-12-02 23:42:22,669::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2890::DEBUG::2012-12-02 23:42:22,669::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2890::DEBUG::2012-12-02 23:42:22,669::task::957::TaskManager.Task::(_decref) Task=`a2fb0ae8-f2f4-493c-afe9-1f91adc64d9e`::ref 0 aborting False >Thread-2891::DEBUG::2012-12-02 23:42:23,811::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2891::DEBUG::2012-12-02 23:42:23,812::task::568::TaskManager.Task::(_updateState) Task=`c7143e7e-57d6-4375-b090-7b94886a30e2`::moving from state init -> state preparing >Thread-2891::INFO::2012-12-02 23:42:23,812::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2891::DEBUG::2012-12-02 23:42:23,812::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2891::DEBUG::2012-12-02 23:42:23,812::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2891::INFO::2012-12-02 23:42:23,812::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2891::DEBUG::2012-12-02 23:42:23,813::task::1151::TaskManager.Task::(prepare) Task=`c7143e7e-57d6-4375-b090-7b94886a30e2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2891::DEBUG::2012-12-02 23:42:23,813::task::568::TaskManager.Task::(_updateState) Task=`c7143e7e-57d6-4375-b090-7b94886a30e2`::moving from state preparing -> state finished >Thread-2891::DEBUG::2012-12-02 23:42:23,813::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2891::DEBUG::2012-12-02 23:42:23,813::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2891::DEBUG::2012-12-02 23:42:23,813::task::957::TaskManager.Task::(_decref) Task=`c7143e7e-57d6-4375-b090-7b94886a30e2`::ref 0 aborting False >Thread-2893::DEBUG::2012-12-02 23:42:24,950::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2893::DEBUG::2012-12-02 23:42:24,950::task::568::TaskManager.Task::(_updateState) Task=`8f3f3153-e1b9-4613-a319-863a5a070906`::moving from state init -> state preparing >Thread-2893::INFO::2012-12-02 23:42:24,950::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2893::DEBUG::2012-12-02 23:42:24,950::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2893::DEBUG::2012-12-02 23:42:24,950::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2893::INFO::2012-12-02 23:42:24,951::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2893::DEBUG::2012-12-02 23:42:24,951::task::1151::TaskManager.Task::(prepare) Task=`8f3f3153-e1b9-4613-a319-863a5a070906`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2893::DEBUG::2012-12-02 23:42:24,951::task::568::TaskManager.Task::(_updateState) Task=`8f3f3153-e1b9-4613-a319-863a5a070906`::moving from state preparing -> state finished >Thread-2893::DEBUG::2012-12-02 23:42:24,951::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2893::DEBUG::2012-12-02 23:42:24,951::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2893::DEBUG::2012-12-02 23:42:24,951::task::957::TaskManager.Task::(_decref) Task=`8f3f3153-e1b9-4613-a319-863a5a070906`::ref 0 aborting False >Thread-2895::DEBUG::2012-12-02 23:42:26,093::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2895::DEBUG::2012-12-02 23:42:26,093::task::568::TaskManager.Task::(_updateState) Task=`e19eb2ed-e890-4f7d-86ab-118a06ebc36a`::moving from state init -> state preparing >Thread-2895::INFO::2012-12-02 23:42:26,093::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2895::DEBUG::2012-12-02 23:42:26,094::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2895::DEBUG::2012-12-02 23:42:26,094::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2895::INFO::2012-12-02 23:42:26,094::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2895::DEBUG::2012-12-02 23:42:26,094::task::1151::TaskManager.Task::(prepare) Task=`e19eb2ed-e890-4f7d-86ab-118a06ebc36a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2895::DEBUG::2012-12-02 23:42:26,094::task::568::TaskManager.Task::(_updateState) Task=`e19eb2ed-e890-4f7d-86ab-118a06ebc36a`::moving from state preparing -> state finished >Thread-2895::DEBUG::2012-12-02 23:42:26,094::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2895::DEBUG::2012-12-02 23:42:26,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2895::DEBUG::2012-12-02 23:42:26,094::task::957::TaskManager.Task::(_decref) Task=`e19eb2ed-e890-4f7d-86ab-118a06ebc36a`::ref 0 aborting False >Thread-2896::DEBUG::2012-12-02 23:42:27,243::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2896::DEBUG::2012-12-02 23:42:27,243::task::568::TaskManager.Task::(_updateState) Task=`ffc587ce-75d0-4c1f-a86b-3a930a06588a`::moving from state init -> state preparing >Thread-2896::INFO::2012-12-02 23:42:27,243::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2896::DEBUG::2012-12-02 23:42:27,243::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2896::DEBUG::2012-12-02 23:42:27,243::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2896::INFO::2012-12-02 23:42:27,243::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2896::DEBUG::2012-12-02 23:42:27,244::task::1151::TaskManager.Task::(prepare) Task=`ffc587ce-75d0-4c1f-a86b-3a930a06588a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2896::DEBUG::2012-12-02 23:42:27,244::task::568::TaskManager.Task::(_updateState) Task=`ffc587ce-75d0-4c1f-a86b-3a930a06588a`::moving from state preparing -> state finished >Thread-2896::DEBUG::2012-12-02 23:42:27,244::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2896::DEBUG::2012-12-02 23:42:27,244::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2896::DEBUG::2012-12-02 23:42:27,244::task::957::TaskManager.Task::(_decref) Task=`ffc587ce-75d0-4c1f-a86b-3a930a06588a`::ref 0 aborting False >Thread-2897::DEBUG::2012-12-02 23:42:28,127::task::568::TaskManager.Task::(_updateState) Task=`e5eb24eb-9fb1-4a59-ba17-b2e343e1439c`::moving from state init -> state preparing >Thread-2897::INFO::2012-12-02 23:42:28,127::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2897::INFO::2012-12-02 23:42:28,127::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00693392753601', 'lastCheck': 1354484531.5073919, 'code': 0, 'valid': True}} >Thread-2897::DEBUG::2012-12-02 23:42:28,127::task::1151::TaskManager.Task::(prepare) Task=`e5eb24eb-9fb1-4a59-ba17-b2e343e1439c`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00693392753601', 'lastCheck': 1354484531.5073919, 'code': 0, 'valid': True}} >Thread-2897::DEBUG::2012-12-02 23:42:28,127::task::568::TaskManager.Task::(_updateState) Task=`e5eb24eb-9fb1-4a59-ba17-b2e343e1439c`::moving from state preparing -> state finished >Thread-2897::DEBUG::2012-12-02 23:42:28,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2897::DEBUG::2012-12-02 23:42:28,128::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2897::DEBUG::2012-12-02 23:42:28,128::task::957::TaskManager.Task::(_decref) Task=`e5eb24eb-9fb1-4a59-ba17-b2e343e1439c`::ref 0 aborting False >Thread-2899::DEBUG::2012-12-02 23:42:28,385::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2899::DEBUG::2012-12-02 23:42:28,386::task::568::TaskManager.Task::(_updateState) Task=`6af526ad-e702-4382-8a4f-b6343bb21eee`::moving from state init -> state preparing >Thread-2899::INFO::2012-12-02 23:42:28,386::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2899::DEBUG::2012-12-02 23:42:28,386::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2899::DEBUG::2012-12-02 23:42:28,386::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2899::INFO::2012-12-02 23:42:28,386::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2899::DEBUG::2012-12-02 23:42:28,386::task::1151::TaskManager.Task::(prepare) Task=`6af526ad-e702-4382-8a4f-b6343bb21eee`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2899::DEBUG::2012-12-02 23:42:28,387::task::568::TaskManager.Task::(_updateState) Task=`6af526ad-e702-4382-8a4f-b6343bb21eee`::moving from state preparing -> state finished >Thread-2899::DEBUG::2012-12-02 23:42:28,387::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2899::DEBUG::2012-12-02 23:42:28,387::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2899::DEBUG::2012-12-02 23:42:28,387::task::957::TaskManager.Task::(_decref) Task=`6af526ad-e702-4382-8a4f-b6343bb21eee`::ref 0 aborting False >Thread-2900::DEBUG::2012-12-02 23:42:29,532::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2900::DEBUG::2012-12-02 23:42:29,532::task::568::TaskManager.Task::(_updateState) Task=`4449a5a0-563c-4edb-bfa1-fdb72d593c07`::moving from state init -> state preparing >Thread-2900::INFO::2012-12-02 23:42:29,532::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2900::DEBUG::2012-12-02 23:42:29,532::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2900::DEBUG::2012-12-02 23:42:29,532::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2900::INFO::2012-12-02 23:42:29,532::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2900::DEBUG::2012-12-02 23:42:29,532::task::1151::TaskManager.Task::(prepare) Task=`4449a5a0-563c-4edb-bfa1-fdb72d593c07`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2900::DEBUG::2012-12-02 23:42:29,533::task::568::TaskManager.Task::(_updateState) Task=`4449a5a0-563c-4edb-bfa1-fdb72d593c07`::moving from state preparing -> state finished >Thread-2900::DEBUG::2012-12-02 23:42:29,533::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2900::DEBUG::2012-12-02 23:42:29,533::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2900::DEBUG::2012-12-02 23:42:29,533::task::957::TaskManager.Task::(_decref) Task=`4449a5a0-563c-4edb-bfa1-fdb72d593c07`::ref 0 aborting False >Thread-2902::DEBUG::2012-12-02 23:42:30,673::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2902::DEBUG::2012-12-02 23:42:30,673::task::568::TaskManager.Task::(_updateState) Task=`6389fb86-6927-458c-84fe-d0e75f7e252f`::moving from state init -> state preparing >Thread-2902::INFO::2012-12-02 23:42:30,673::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2902::DEBUG::2012-12-02 23:42:30,673::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2902::DEBUG::2012-12-02 23:42:30,673::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2902::INFO::2012-12-02 23:42:30,674::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2902::DEBUG::2012-12-02 23:42:30,674::task::1151::TaskManager.Task::(prepare) Task=`6389fb86-6927-458c-84fe-d0e75f7e252f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2902::DEBUG::2012-12-02 23:42:30,674::task::568::TaskManager.Task::(_updateState) Task=`6389fb86-6927-458c-84fe-d0e75f7e252f`::moving from state preparing -> state finished >Thread-2902::DEBUG::2012-12-02 23:42:30,674::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2902::DEBUG::2012-12-02 23:42:30,674::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2902::DEBUG::2012-12-02 23:42:30,674::task::957::TaskManager.Task::(_decref) Task=`6389fb86-6927-458c-84fe-d0e75f7e252f`::ref 0 aborting False >Thread-2903::DEBUG::2012-12-02 23:42:31,853::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2903::DEBUG::2012-12-02 23:42:31,853::task::568::TaskManager.Task::(_updateState) Task=`2a7fc8ce-0f2c-4286-8dbb-09432feaacdd`::moving from state init -> state preparing >Thread-2903::INFO::2012-12-02 23:42:31,853::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2903::DEBUG::2012-12-02 23:42:31,853::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2903::DEBUG::2012-12-02 23:42:31,853::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2903::INFO::2012-12-02 23:42:31,854::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2903::DEBUG::2012-12-02 23:42:31,854::task::1151::TaskManager.Task::(prepare) Task=`2a7fc8ce-0f2c-4286-8dbb-09432feaacdd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2903::DEBUG::2012-12-02 23:42:31,854::task::568::TaskManager.Task::(_updateState) Task=`2a7fc8ce-0f2c-4286-8dbb-09432feaacdd`::moving from state preparing -> state finished >Thread-2903::DEBUG::2012-12-02 23:42:31,854::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2903::DEBUG::2012-12-02 23:42:31,854::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2903::DEBUG::2012-12-02 23:42:31,854::task::957::TaskManager.Task::(_decref) Task=`2a7fc8ce-0f2c-4286-8dbb-09432feaacdd`::ref 0 aborting False >Thread-2905::DEBUG::2012-12-02 23:42:32,993::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2905::DEBUG::2012-12-02 23:42:32,994::task::568::TaskManager.Task::(_updateState) Task=`300cf085-f5c6-4874-9c43-1327e9d2607f`::moving from state init -> state preparing >Thread-2905::INFO::2012-12-02 23:42:32,994::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2905::DEBUG::2012-12-02 23:42:32,994::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2905::DEBUG::2012-12-02 23:42:32,994::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2905::INFO::2012-12-02 23:42:32,994::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2905::DEBUG::2012-12-02 23:42:32,994::task::1151::TaskManager.Task::(prepare) Task=`300cf085-f5c6-4874-9c43-1327e9d2607f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2905::DEBUG::2012-12-02 23:42:32,994::task::568::TaskManager.Task::(_updateState) Task=`300cf085-f5c6-4874-9c43-1327e9d2607f`::moving from state preparing -> state finished >Thread-2905::DEBUG::2012-12-02 23:42:32,995::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2905::DEBUG::2012-12-02 23:42:32,995::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2905::DEBUG::2012-12-02 23:42:32,995::task::957::TaskManager.Task::(_decref) Task=`300cf085-f5c6-4874-9c43-1327e9d2607f`::ref 0 aborting False >Thread-2906::DEBUG::2012-12-02 23:42:34,141::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2906::DEBUG::2012-12-02 23:42:34,141::task::568::TaskManager.Task::(_updateState) Task=`4a005542-3383-4e6e-a89e-200f62405cc4`::moving from state init -> state preparing >Thread-2906::INFO::2012-12-02 23:42:34,141::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2906::DEBUG::2012-12-02 23:42:34,142::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2906::DEBUG::2012-12-02 23:42:34,142::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2906::INFO::2012-12-02 23:42:34,142::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2906::DEBUG::2012-12-02 23:42:34,142::task::1151::TaskManager.Task::(prepare) Task=`4a005542-3383-4e6e-a89e-200f62405cc4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2906::DEBUG::2012-12-02 23:42:34,142::task::568::TaskManager.Task::(_updateState) Task=`4a005542-3383-4e6e-a89e-200f62405cc4`::moving from state preparing -> state finished >Thread-2906::DEBUG::2012-12-02 23:42:34,142::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2906::DEBUG::2012-12-02 23:42:34,142::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2906::DEBUG::2012-12-02 23:42:34,142::task::957::TaskManager.Task::(_decref) Task=`4a005542-3383-4e6e-a89e-200f62405cc4`::ref 0 aborting False >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,052::safelease::178::SANLock::(acquireHostId) Host id for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully acquired (id: 2) >fec5421a-317e-429c-a472-8bebef790541::INFO::2012-12-02 23:42:35,053::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-2908::DEBUG::2012-12-02 23:42:35,297::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-2908::DEBUG::2012-12-02 23:42:35,297::task::568::TaskManager.Task::(_updateState) Task=`0b320c25-b51c-49f3-be21-18bd2af2b98a`::moving from state init -> state preparing >Thread-2908::INFO::2012-12-02 23:42:35,298::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-2908::DEBUG::2012-12-02 23:42:35,298::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-2908::DEBUG::2012-12-02 23:42:35,298::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-2908::INFO::2012-12-02 23:42:35,298::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2908::DEBUG::2012-12-02 23:42:35,298::task::1151::TaskManager.Task::(prepare) Task=`0b320c25-b51c-49f3-be21-18bd2af2b98a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-2908::DEBUG::2012-12-02 23:42:35,298::task::568::TaskManager.Task::(_updateState) Task=`0b320c25-b51c-49f3-be21-18bd2af2b98a`::moving from state preparing -> state finished >Thread-2908::DEBUG::2012-12-02 23:42:35,298::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2908::DEBUG::2012-12-02 23:42:35,299::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2908::DEBUG::2012-12-02 23:42:35,299::task::957::TaskManager.Task::(_decref) Task=`0b320c25-b51c-49f3-be21-18bd2af2b98a`::ref 0 aborting False >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,371::safelease::235::SANLock::(acquire) Cluster lock for domain 5971512c-ee51-475e-b956-3872c38ce193 successfully acquired (id: 2) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,371::sp::256::Storage.StoragePool::(startSpm) spm lock acquired successfully >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,376::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=-1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=059380d675669a49a658e84dd3dca1d02a84933c'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,376::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,376::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,376::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,410::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,410::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a50461c-69fa-4a1f-bb37-50f74dd85b11`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,410::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,410::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,411::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a50461c-69fa-4a1f-bb37-50f74dd85b11`::Granted request >fec5421a-317e-429c-a472-8bebef790541::INFO::2012-12-02 23:42:35,411::sp::403::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `5971512c-ee51-475e-b956-3872c38ce193` >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,411::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`1438f13e-1669-4579-96db-6e69c93db319`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,411::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' for lock type 'exclusive' >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,411::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free. Now locking as 'exclusive' (1 active user) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,412::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.5971512c-ee51-475e-b956-3872c38ce193`ReqID=`1438f13e-1669-4579-96db-6e69c93db319`::Granted request >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,412::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,412::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' (0 active users) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,412::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193' is free, finding out if anyone is waiting for it. >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,412::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5971512c-ee51-475e-b956-3872c38ce193', Clearing records. >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,413::sp::407::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,413::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,414::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,623::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:35,624::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,624::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,658::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,659::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,668::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,670::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:35,670::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:35,671::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,671::sp::414::Storage.StoragePool::(_upgradePool) Registering with state change event >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,671::sp::416::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-2970::DEBUG::2012-12-02 23:42:35,671::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,672::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' >Thread-2970::DEBUG::2012-12-02 23:42:35,672::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,673::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,676::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:35,676::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-2970::DEBUG::2012-12-02 23:42:35,879::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-2970::WARNING::2012-12-02 23:42:35,880::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-2970::DEBUG::2012-12-02 23:42:35,881::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2971::DEBUG::2012-12-02 23:42:35,909::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2971::DEBUG::2012-12-02 23:42:35,910::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2971::DEBUG::2012-12-02 23:42:36,116::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2971::WARNING::2012-12-02 23:42:36,118::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2971::DEBUG::2012-12-02 23:42:36,118::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:36,166::sd::583::Storage.StorageDomain::(createMasterTree) vms dir not found, creating (/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193/master/vms) >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:36,181::sd::588::Storage.StorageDomain::(createMasterTree) tasks dir not found, creating (/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193/master/tasks) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,195::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,195::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,400::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >fec5421a-317e-429c-a472-8bebef790541::WARNING::2012-12-02 23:42:36,402::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,402::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >fec5421a-317e-429c-a472-8bebef790541::ERROR::2012-12-02 23:42:36,432::sp::277::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 275, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper > return f(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1568, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,436::sp::305::Storage.StoragePool::(startSpm) ended. >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,436::task::568::TaskManager.Task::(_updateState) Task=`fec5421a-317e-429c-a472-8bebef790541`::moving from state running -> state finished >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,436::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,437::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,437::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,437::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,437::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,437::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,437::task::880::TaskManager.Task::(_runJobs) Task=`fec5421a-317e-429c-a472-8bebef790541`::Task.run: exit - success: result >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,438::task::957::TaskManager.Task::(_decref) Task=`fec5421a-317e-429c-a472-8bebef790541`::ref 0 aborting False >fec5421a-317e-429c-a472-8bebef790541::DEBUG::2012-12-02 23:42:36,438::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-3155::DEBUG::2012-12-02 23:42:36,449::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3155::DEBUG::2012-12-02 23:42:36,449::task::568::TaskManager.Task::(_updateState) Task=`5e55e5c4-950c-40e0-bf9b-b7bd425ce833`::moving from state init -> state preparing >Thread-3155::INFO::2012-12-02 23:42:36,450::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-3155::DEBUG::2012-12-02 23:42:36,450::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-3155::DEBUG::2012-12-02 23:42:36,450::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'} >Thread-3155::INFO::2012-12-02 23:42:36,450::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-3155::DEBUG::2012-12-02 23:42:36,450::task::1151::TaskManager.Task::(prepare) Task=`5e55e5c4-950c-40e0-bf9b-b7bd425ce833`::finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'fec5421a-317e-429c-a472-8bebef790541'}} >Thread-3155::DEBUG::2012-12-02 23:42:36,450::task::568::TaskManager.Task::(_updateState) Task=`5e55e5c4-950c-40e0-bf9b-b7bd425ce833`::moving from state preparing -> state finished >Thread-3155::DEBUG::2012-12-02 23:42:36,450::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3155::DEBUG::2012-12-02 23:42:36,450::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3155::DEBUG::2012-12-02 23:42:36,451::task::957::TaskManager.Task::(_decref) Task=`5e55e5c4-950c-40e0-bf9b-b7bd425ce833`::ref 0 aborting False >Thread-3156::DEBUG::2012-12-02 23:42:36,593::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3156::DEBUG::2012-12-02 23:42:36,593::task::568::TaskManager.Task::(_updateState) Task=`62c2799c-ab89-4c1a-9f69-bdb2c6153a7d`::moving from state init -> state preparing >Thread-3156::INFO::2012-12-02 23:42:36,593::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3156::INFO::2012-12-02 23:42:36,594::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3156::DEBUG::2012-12-02 23:42:36,594::task::1151::TaskManager.Task::(prepare) Task=`62c2799c-ab89-4c1a-9f69-bdb2c6153a7d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3156::DEBUG::2012-12-02 23:42:36,594::task::568::TaskManager.Task::(_updateState) Task=`62c2799c-ab89-4c1a-9f69-bdb2c6153a7d`::moving from state preparing -> state finished >Thread-3156::DEBUG::2012-12-02 23:42:36,594::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3156::DEBUG::2012-12-02 23:42:36,594::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3156::DEBUG::2012-12-02 23:42:36,594::task::957::TaskManager.Task::(_decref) Task=`62c2799c-ab89-4c1a-9f69-bdb2c6153a7d`::ref 0 aborting False >Thread-3157::DEBUG::2012-12-02 23:42:36,735::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3157::DEBUG::2012-12-02 23:42:36,736::task::568::TaskManager.Task::(_updateState) Task=`1ac8977b-3f8a-4839-9332-09644bffef2e`::moving from state init -> state preparing >Thread-3157::INFO::2012-12-02 23:42:36,736::logUtils::37::dispatcher::(wrapper) Run and protect: clearTask(taskID='fec5421a-317e-429c-a472-8bebef790541', spUUID=None, options=None) >Thread-3157::DEBUG::2012-12-02 23:42:36,736::taskManager::157::TaskManager::(clearTask) Entry. taskID: fec5421a-317e-429c-a472-8bebef790541 >Thread-3157::DEBUG::2012-12-02 23:42:36,736::taskManager::162::TaskManager::(clearTask) Return. >Thread-3157::INFO::2012-12-02 23:42:36,736::logUtils::39::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-3157::DEBUG::2012-12-02 23:42:36,736::task::1151::TaskManager.Task::(prepare) Task=`1ac8977b-3f8a-4839-9332-09644bffef2e`::finished: None >Thread-3157::DEBUG::2012-12-02 23:42:36,736::task::568::TaskManager.Task::(_updateState) Task=`1ac8977b-3f8a-4839-9332-09644bffef2e`::moving from state preparing -> state finished >Thread-3157::DEBUG::2012-12-02 23:42:36,736::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3157::DEBUG::2012-12-02 23:42:36,736::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3157::DEBUG::2012-12-02 23:42:36,737::task::957::TaskManager.Task::(_decref) Task=`1ac8977b-3f8a-4839-9332-09644bffef2e`::ref 0 aborting False >Thread-3159::DEBUG::2012-12-02 23:42:36,938::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3159::DEBUG::2012-12-02 23:42:36,938::task::568::TaskManager.Task::(_updateState) Task=`3c880b0d-5051-4260-a126-fd0c8d0560b2`::moving from state init -> state preparing >Thread-3159::INFO::2012-12-02 23:42:36,938::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3159::DEBUG::2012-12-02 23:42:36,938::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1bc093d6-f79f-44f8-998e-599008b3f976`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3159::DEBUG::2012-12-02 23:42:36,939::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3159::DEBUG::2012-12-02 23:42:36,939::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3159::DEBUG::2012-12-02 23:42:36,939::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1bc093d6-f79f-44f8-998e-599008b3f976`::Granted request >Thread-3159::DEBUG::2012-12-02 23:42:36,939::task::794::TaskManager.Task::(resourceAcquired) Task=`3c880b0d-5051-4260-a126-fd0c8d0560b2`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3159::DEBUG::2012-12-02 23:42:36,939::task::957::TaskManager.Task::(_decref) Task=`3c880b0d-5051-4260-a126-fd0c8d0560b2`::ref 1 aborting False >Thread-3159::DEBUG::2012-12-02 23:42:36,946::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3159::DEBUG::2012-12-02 23:42:36,946::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3159::DEBUG::2012-12-02 23:42:37,152::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3159::WARNING::2012-12-02 23:42:37,153::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3159::DEBUG::2012-12-02 23:42:37,154::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3159::WARNING::2012-12-02 23:42:37,182::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3159::DEBUG::2012-12-02 23:42:37,183::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3159::DEBUG::2012-12-02 23:42:37,183::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3159::DEBUG::2012-12-02 23:42:37,389::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3159::WARNING::2012-12-02 23:42:37,391::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3159::DEBUG::2012-12-02 23:42:37,391::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3159::WARNING::2012-12-02 23:42:37,420::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3159::INFO::2012-12-02 23:42:37,421::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3159::DEBUG::2012-12-02 23:42:37,421::task::1151::TaskManager.Task::(prepare) Task=`3c880b0d-5051-4260-a126-fd0c8d0560b2`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3159::DEBUG::2012-12-02 23:42:37,421::task::568::TaskManager.Task::(_updateState) Task=`3c880b0d-5051-4260-a126-fd0c8d0560b2`::moving from state preparing -> state finished >Thread-3159::DEBUG::2012-12-02 23:42:37,421::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3159::DEBUG::2012-12-02 23:42:37,421::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3159::DEBUG::2012-12-02 23:42:37,422::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3159::DEBUG::2012-12-02 23:42:37,422::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3159::DEBUG::2012-12-02 23:42:37,422::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3159::DEBUG::2012-12-02 23:42:37,422::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3159::DEBUG::2012-12-02 23:42:37,422::task::957::TaskManager.Task::(_decref) Task=`3c880b0d-5051-4260-a126-fd0c8d0560b2`::ref 0 aborting False >Thread-3282::DEBUG::2012-12-02 23:42:37,718::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3282::DEBUG::2012-12-02 23:42:37,718::task::568::TaskManager.Task::(_updateState) Task=`cf69673e-6206-4f0c-bed3-c003972d5896`::moving from state init -> state preparing >Thread-3282::INFO::2012-12-02 23:42:37,718::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-3282::DEBUG::2012-12-02 23:42:37,718::taskManager::177::TaskManager::(getAllTasksInfo) Entry. >Thread-3282::DEBUG::2012-12-02 23:42:37,718::taskManager::188::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-3282::INFO::2012-12-02 23:42:37,719::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-3282::DEBUG::2012-12-02 23:42:37,719::task::1151::TaskManager.Task::(prepare) Task=`cf69673e-6206-4f0c-bed3-c003972d5896`::finished: {'allTasksInfo': {}} >Thread-3282::DEBUG::2012-12-02 23:42:37,719::task::568::TaskManager.Task::(_updateState) Task=`cf69673e-6206-4f0c-bed3-c003972d5896`::moving from state preparing -> state finished >Thread-3282::DEBUG::2012-12-02 23:42:37,719::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3282::DEBUG::2012-12-02 23:42:37,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3282::DEBUG::2012-12-02 23:42:37,719::task::957::TaskManager.Task::(_decref) Task=`cf69673e-6206-4f0c-bed3-c003972d5896`::ref 0 aborting False >Thread-3283::DEBUG::2012-12-02 23:42:37,858::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3283::DEBUG::2012-12-02 23:42:37,859::task::568::TaskManager.Task::(_updateState) Task=`99cbb5b3-7655-43b3-97d2-e3f790784242`::moving from state init -> state preparing >Thread-3283::INFO::2012-12-02 23:42:37,859::logUtils::37::dispatcher::(wrapper) Run and protect: updateVM(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', vmList=[], sdUUID=None, options=None) >Thread-3283::DEBUG::2012-12-02 23:42:37,859::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9b2ee7e8-d2a5-42a9-9419-7e144c070e1f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3283::DEBUG::2012-12-02 23:42:37,859::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3283::DEBUG::2012-12-02 23:42:37,860::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3283::DEBUG::2012-12-02 23:42:37,860::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9b2ee7e8-d2a5-42a9-9419-7e144c070e1f`::Granted request >Thread-3283::DEBUG::2012-12-02 23:42:37,860::task::794::TaskManager.Task::(resourceAcquired) Task=`99cbb5b3-7655-43b3-97d2-e3f790784242`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3283::DEBUG::2012-12-02 23:42:37,860::task::957::TaskManager.Task::(_decref) Task=`99cbb5b3-7655-43b3-97d2-e3f790784242`::ref 1 aborting False >Thread-3283::INFO::2012-12-02 23:42:37,860::sp::1247::Storage.StoragePool::(updateVM) spUUID=479cf195-bb99-45c5-b400-433d810eaa48 sdUUID=5971512c-ee51-475e-b956-3872c38ce193 >Thread-3283::INFO::2012-12-02 23:42:37,862::logUtils::39::dispatcher::(wrapper) Run and protect: updateVM, Return response: None >Thread-3283::DEBUG::2012-12-02 23:42:37,862::task::1151::TaskManager.Task::(prepare) Task=`99cbb5b3-7655-43b3-97d2-e3f790784242`::finished: None >Thread-3283::DEBUG::2012-12-02 23:42:37,862::task::568::TaskManager.Task::(_updateState) Task=`99cbb5b3-7655-43b3-97d2-e3f790784242`::moving from state preparing -> state finished >Thread-3283::DEBUG::2012-12-02 23:42:37,862::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3283::DEBUG::2012-12-02 23:42:37,862::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3283::DEBUG::2012-12-02 23:42:37,862::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3283::DEBUG::2012-12-02 23:42:37,863::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3283::DEBUG::2012-12-02 23:42:37,863::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3283::DEBUG::2012-12-02 23:42:37,863::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3283::DEBUG::2012-12-02 23:42:37,863::task::957::TaskManager.Task::(_decref) Task=`99cbb5b3-7655-43b3-97d2-e3f790784242`::ref 0 aborting False >Thread-3284::DEBUG::2012-12-02 23:42:39,037::task::568::TaskManager.Task::(_updateState) Task=`9f71ee4a-088c-490e-aa33-c5d4b3fc76a5`::moving from state init -> state preparing >Thread-3284::INFO::2012-12-02 23:42:39,037::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3284::INFO::2012-12-02 23:42:39,038::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0075159072876', 'lastCheck': 1354484555.053232, 'code': 0, 'valid': True}} >Thread-3284::DEBUG::2012-12-02 23:42:39,038::task::1151::TaskManager.Task::(prepare) Task=`9f71ee4a-088c-490e-aa33-c5d4b3fc76a5`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0075159072876', 'lastCheck': 1354484555.053232, 'code': 0, 'valid': True}} >Thread-3284::DEBUG::2012-12-02 23:42:39,038::task::568::TaskManager.Task::(_updateState) Task=`9f71ee4a-088c-490e-aa33-c5d4b3fc76a5`::moving from state preparing -> state finished >Thread-3284::DEBUG::2012-12-02 23:42:39,038::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3284::DEBUG::2012-12-02 23:42:39,038::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3284::DEBUG::2012-12-02 23:42:39,038::task::957::TaskManager.Task::(_decref) Task=`9f71ee4a-088c-490e-aa33-c5d4b3fc76a5`::ref 0 aborting False >Thread-3289::DEBUG::2012-12-02 23:42:47,691::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3289::DEBUG::2012-12-02 23:42:47,692::task::568::TaskManager.Task::(_updateState) Task=`180ae739-bc3f-480d-8d0d-0fc83bf493ff`::moving from state init -> state preparing >Thread-3289::INFO::2012-12-02 23:42:47,692::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3289::INFO::2012-12-02 23:42:47,692::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3289::DEBUG::2012-12-02 23:42:47,692::task::1151::TaskManager.Task::(prepare) Task=`180ae739-bc3f-480d-8d0d-0fc83bf493ff`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3289::DEBUG::2012-12-02 23:42:47,693::task::568::TaskManager.Task::(_updateState) Task=`180ae739-bc3f-480d-8d0d-0fc83bf493ff`::moving from state preparing -> state finished >Thread-3289::DEBUG::2012-12-02 23:42:47,693::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3289::DEBUG::2012-12-02 23:42:47,693::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3289::DEBUG::2012-12-02 23:42:47,693::task::957::TaskManager.Task::(_decref) Task=`180ae739-bc3f-480d-8d0d-0fc83bf493ff`::ref 0 aborting False >Thread-3291::DEBUG::2012-12-02 23:42:47,852::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3291::DEBUG::2012-12-02 23:42:47,852::task::568::TaskManager.Task::(_updateState) Task=`9f191c50-68a6-465a-b307-cec6f7484123`::moving from state init -> state preparing >Thread-3291::INFO::2012-12-02 23:42:47,853::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3291::DEBUG::2012-12-02 23:42:47,853::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`eb8505f3-0c10-41aa-a994-2d9172549e8d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3291::DEBUG::2012-12-02 23:42:47,853::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3291::DEBUG::2012-12-02 23:42:47,853::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3291::DEBUG::2012-12-02 23:42:47,853::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`eb8505f3-0c10-41aa-a994-2d9172549e8d`::Granted request >Thread-3291::DEBUG::2012-12-02 23:42:47,854::task::794::TaskManager.Task::(resourceAcquired) Task=`9f191c50-68a6-465a-b307-cec6f7484123`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3291::DEBUG::2012-12-02 23:42:47,854::task::957::TaskManager.Task::(_decref) Task=`9f191c50-68a6-465a-b307-cec6f7484123`::ref 1 aborting False >Thread-3291::DEBUG::2012-12-02 23:42:47,858::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3291::DEBUG::2012-12-02 23:42:47,859::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3291::DEBUG::2012-12-02 23:42:48,065::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3291::WARNING::2012-12-02 23:42:48,066::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3291::DEBUG::2012-12-02 23:42:48,067::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3291::WARNING::2012-12-02 23:42:48,097::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3291::DEBUG::2012-12-02 23:42:48,098::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3291::DEBUG::2012-12-02 23:42:48,098::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3291::DEBUG::2012-12-02 23:42:48,306::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3291::WARNING::2012-12-02 23:42:48,308::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3291::DEBUG::2012-12-02 23:42:48,308::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3291::WARNING::2012-12-02 23:42:48,337::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3291::INFO::2012-12-02 23:42:48,337::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3291::DEBUG::2012-12-02 23:42:48,337::task::1151::TaskManager.Task::(prepare) Task=`9f191c50-68a6-465a-b307-cec6f7484123`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3291::DEBUG::2012-12-02 23:42:48,338::task::568::TaskManager.Task::(_updateState) Task=`9f191c50-68a6-465a-b307-cec6f7484123`::moving from state preparing -> state finished >Thread-3291::DEBUG::2012-12-02 23:42:48,338::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3291::DEBUG::2012-12-02 23:42:48,338::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3291::DEBUG::2012-12-02 23:42:48,338::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3291::DEBUG::2012-12-02 23:42:48,338::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3291::DEBUG::2012-12-02 23:42:48,338::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3291::DEBUG::2012-12-02 23:42:48,338::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3291::DEBUG::2012-12-02 23:42:48,339::task::957::TaskManager.Task::(_decref) Task=`9f191c50-68a6-465a-b307-cec6f7484123`::ref 0 aborting False >Thread-3414::DEBUG::2012-12-02 23:45:30,591::task::568::TaskManager.Task::(_updateState) Task=`8d95aeda-cfe2-473d-92e3-7461f3e2a166`::moving from state init -> state preparing >Thread-3414::INFO::2012-12-02 23:45:30,591::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3414::INFO::2012-12-02 23:45:30,591::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00755214691162', 'lastCheck': 1354484725.381917, 'code': 0, 'valid': True}} >Thread-3414::DEBUG::2012-12-02 23:45:30,592::task::1151::TaskManager.Task::(prepare) Task=`8d95aeda-cfe2-473d-92e3-7461f3e2a166`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00755214691162', 'lastCheck': 1354484725.381917, 'code': 0, 'valid': True}} >Thread-3414::DEBUG::2012-12-02 23:45:30,592::task::568::TaskManager.Task::(_updateState) Task=`8d95aeda-cfe2-473d-92e3-7461f3e2a166`::moving from state preparing -> state finished >Thread-3414::DEBUG::2012-12-02 23:45:30,592::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3414::DEBUG::2012-12-02 23:45:30,592::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3414::DEBUG::2012-12-02 23:45:30,592::task::957::TaskManager.Task::(_decref) Task=`8d95aeda-cfe2-473d-92e3-7461f3e2a166`::ref 0 aborting False >Thread-3415::DEBUG::2012-12-02 23:45:30,853::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-3415::DEBUG::2012-12-02 23:45:30,866::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-3415::DEBUG::2012-12-02 23:45:30,878::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-3415::DEBUG::2012-12-02 23:45:30,878::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-3415::DEBUG::2012-12-02 23:45:30,887::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-3415::DEBUG::2012-12-02 23:45:30,888::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-3415::DEBUG::2012-12-02 23:45:30,899::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-3415::DEBUG::2012-12-02 23:45:30,899::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-3415::DEBUG::2012-12-02 23:45:30,911::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-3415::DEBUG::2012-12-02 23:45:30,912::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-3415::DEBUG::2012-12-02 23:45:30,924::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-3415::DEBUG::2012-12-02 23:45:30,925::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.202.48'}} >Thread-3420::DEBUG::2012-12-02 23:45:38,657::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3420::DEBUG::2012-12-02 23:45:38,657::task::568::TaskManager.Task::(_updateState) Task=`d90fa747-1c4b-47c8-a8b7-41f42ea58ff8`::moving from state init -> state preparing >Thread-3420::INFO::2012-12-02 23:45:38,657::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3420::INFO::2012-12-02 23:45:38,658::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3420::DEBUG::2012-12-02 23:45:38,658::task::1151::TaskManager.Task::(prepare) Task=`d90fa747-1c4b-47c8-a8b7-41f42ea58ff8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3420::DEBUG::2012-12-02 23:45:38,658::task::568::TaskManager.Task::(_updateState) Task=`d90fa747-1c4b-47c8-a8b7-41f42ea58ff8`::moving from state preparing -> state finished >Thread-3420::DEBUG::2012-12-02 23:45:38,658::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3420::DEBUG::2012-12-02 23:45:38,658::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3420::DEBUG::2012-12-02 23:45:38,658::task::957::TaskManager.Task::(_decref) Task=`d90fa747-1c4b-47c8-a8b7-41f42ea58ff8`::ref 0 aborting False >Thread-3421::DEBUG::2012-12-02 23:45:38,844::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3421::DEBUG::2012-12-02 23:45:38,844::task::568::TaskManager.Task::(_updateState) Task=`86e25baa-84db-4fe5-adb2-963afaf92e32`::moving from state init -> state preparing >Thread-3421::INFO::2012-12-02 23:45:38,844::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3421::DEBUG::2012-12-02 23:45:38,845::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f7d0220e-47f8-4a3a-a777-92e05c37ac11`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3421::DEBUG::2012-12-02 23:45:38,845::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3421::DEBUG::2012-12-02 23:45:38,845::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3421::DEBUG::2012-12-02 23:45:38,845::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f7d0220e-47f8-4a3a-a777-92e05c37ac11`::Granted request >Thread-3421::DEBUG::2012-12-02 23:45:38,845::task::794::TaskManager.Task::(resourceAcquired) Task=`86e25baa-84db-4fe5-adb2-963afaf92e32`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3421::DEBUG::2012-12-02 23:45:38,845::task::957::TaskManager.Task::(_decref) Task=`86e25baa-84db-4fe5-adb2-963afaf92e32`::ref 1 aborting False >Thread-3421::DEBUG::2012-12-02 23:45:38,850::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3421::DEBUG::2012-12-02 23:45:38,850::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3421::DEBUG::2012-12-02 23:45:39,062::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3421::WARNING::2012-12-02 23:45:39,064::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3421::DEBUG::2012-12-02 23:45:39,064::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3479::WARNING::2012-12-02 23:45:39,089::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3480::WARNING::2012-12-02 23:45:39,090::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3421::WARNING::2012-12-02 23:45:39,097::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3421::DEBUG::2012-12-02 23:45:39,098::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3421::DEBUG::2012-12-02 23:45:39,098::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3421::DEBUG::2012-12-02 23:45:39,304::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3421::WARNING::2012-12-02 23:45:39,306::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3421::DEBUG::2012-12-02 23:45:39,306::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3421::WARNING::2012-12-02 23:45:39,336::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3421::INFO::2012-12-02 23:45:39,336::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3421::DEBUG::2012-12-02 23:45:39,337::task::1151::TaskManager.Task::(prepare) Task=`86e25baa-84db-4fe5-adb2-963afaf92e32`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3421::DEBUG::2012-12-02 23:45:39,337::task::568::TaskManager.Task::(_updateState) Task=`86e25baa-84db-4fe5-adb2-963afaf92e32`::moving from state preparing -> state finished >Thread-3421::DEBUG::2012-12-02 23:45:39,337::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3421::DEBUG::2012-12-02 23:45:39,337::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3421::DEBUG::2012-12-02 23:45:39,337::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3421::DEBUG::2012-12-02 23:45:39,337::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3421::DEBUG::2012-12-02 23:45:39,338::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3421::DEBUG::2012-12-02 23:45:39,338::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3421::DEBUG::2012-12-02 23:45:39,338::task::957::TaskManager.Task::(_decref) Task=`86e25baa-84db-4fe5-adb2-963afaf92e32`::ref 0 aborting False >Thread-3544::DEBUG::2012-12-02 23:45:39,590::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3544::DEBUG::2012-12-02 23:45:39,591::task::568::TaskManager.Task::(_updateState) Task=`fe7bcfc3-23bd-4343-83df-d1fcfa49435b`::moving from state init -> state preparing >Thread-3544::INFO::2012-12-02 23:45:39,591::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-3544::DEBUG::2012-12-02 23:45:39,591::taskManager::177::TaskManager::(getAllTasksInfo) Entry. >Thread-3544::DEBUG::2012-12-02 23:45:39,591::taskManager::188::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-3544::INFO::2012-12-02 23:45:39,591::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-3544::DEBUG::2012-12-02 23:45:39,592::task::1151::TaskManager.Task::(prepare) Task=`fe7bcfc3-23bd-4343-83df-d1fcfa49435b`::finished: {'allTasksInfo': {}} >Thread-3544::DEBUG::2012-12-02 23:45:39,592::task::568::TaskManager.Task::(_updateState) Task=`fe7bcfc3-23bd-4343-83df-d1fcfa49435b`::moving from state preparing -> state finished >Thread-3544::DEBUG::2012-12-02 23:45:39,592::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3544::DEBUG::2012-12-02 23:45:39,592::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3544::DEBUG::2012-12-02 23:45:39,592::task::957::TaskManager.Task::(_decref) Task=`fe7bcfc3-23bd-4343-83df-d1fcfa49435b`::ref 0 aborting False >Thread-3546::DEBUG::2012-12-02 23:45:42,183::task::568::TaskManager.Task::(_updateState) Task=`f3c1e79f-5718-44c8-ba6e-042f0de11ad4`::moving from state init -> state preparing >Thread-3546::INFO::2012-12-02 23:45:42,183::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3546::INFO::2012-12-02 23:45:42,184::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00767993927002', 'lastCheck': 1354484735.4000139, 'code': 0, 'valid': True}} >Thread-3546::DEBUG::2012-12-02 23:45:42,184::task::1151::TaskManager.Task::(prepare) Task=`f3c1e79f-5718-44c8-ba6e-042f0de11ad4`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00767993927002', 'lastCheck': 1354484735.4000139, 'code': 0, 'valid': True}} >Thread-3546::DEBUG::2012-12-02 23:45:42,184::task::568::TaskManager.Task::(_updateState) Task=`f3c1e79f-5718-44c8-ba6e-042f0de11ad4`::moving from state preparing -> state finished >Thread-3546::DEBUG::2012-12-02 23:45:42,184::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3546::DEBUG::2012-12-02 23:45:42,184::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3546::DEBUG::2012-12-02 23:45:42,184::task::957::TaskManager.Task::(_decref) Task=`f3c1e79f-5718-44c8-ba6e-042f0de11ad4`::ref 0 aborting False >Thread-3551::DEBUG::2012-12-02 23:45:49,609::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3551::DEBUG::2012-12-02 23:45:49,609::task::568::TaskManager.Task::(_updateState) Task=`faa98f19-302f-406a-abd1-89cf68a96aea`::moving from state init -> state preparing >Thread-3551::INFO::2012-12-02 23:45:49,609::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3551::INFO::2012-12-02 23:45:49,609::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3551::DEBUG::2012-12-02 23:45:49,610::task::1151::TaskManager.Task::(prepare) Task=`faa98f19-302f-406a-abd1-89cf68a96aea`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3551::DEBUG::2012-12-02 23:45:49,610::task::568::TaskManager.Task::(_updateState) Task=`faa98f19-302f-406a-abd1-89cf68a96aea`::moving from state preparing -> state finished >Thread-3551::DEBUG::2012-12-02 23:45:49,610::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3551::DEBUG::2012-12-02 23:45:49,610::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3551::DEBUG::2012-12-02 23:45:49,610::task::957::TaskManager.Task::(_decref) Task=`faa98f19-302f-406a-abd1-89cf68a96aea`::ref 0 aborting False >Thread-3552::DEBUG::2012-12-02 23:45:49,748::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3552::DEBUG::2012-12-02 23:45:49,749::task::568::TaskManager.Task::(_updateState) Task=`ee863d0d-e51a-4ba7-9669-aef1a5fd91de`::moving from state init -> state preparing >Thread-3552::INFO::2012-12-02 23:45:49,749::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3552::DEBUG::2012-12-02 23:45:49,749::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b4d40c38-9b30-40b4-8d93-7495ed574a4d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3552::DEBUG::2012-12-02 23:45:49,749::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3552::DEBUG::2012-12-02 23:45:49,750::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3552::DEBUG::2012-12-02 23:45:49,750::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b4d40c38-9b30-40b4-8d93-7495ed574a4d`::Granted request >Thread-3552::DEBUG::2012-12-02 23:45:49,750::task::794::TaskManager.Task::(resourceAcquired) Task=`ee863d0d-e51a-4ba7-9669-aef1a5fd91de`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3552::DEBUG::2012-12-02 23:45:49,750::task::957::TaskManager.Task::(_decref) Task=`ee863d0d-e51a-4ba7-9669-aef1a5fd91de`::ref 1 aborting False >Thread-3552::DEBUG::2012-12-02 23:45:49,756::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3552::DEBUG::2012-12-02 23:45:49,756::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3552::DEBUG::2012-12-02 23:45:49,963::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3552::WARNING::2012-12-02 23:45:49,965::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3552::DEBUG::2012-12-02 23:45:49,965::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3590::WARNING::2012-12-02 23:45:49,983::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3592::WARNING::2012-12-02 23:45:49,984::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3593::WARNING::2012-12-02 23:45:49,985::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3594::WARNING::2012-12-02 23:45:49,985::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3595::WARNING::2012-12-02 23:45:49,986::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3596::WARNING::2012-12-02 23:45:49,986::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3597::WARNING::2012-12-02 23:45:49,987::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3598::WARNING::2012-12-02 23:45:49,987::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3599::WARNING::2012-12-02 23:45:49,988::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3600::WARNING::2012-12-02 23:45:49,988::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3601::WARNING::2012-12-02 23:45:49,989::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3602::WARNING::2012-12-02 23:45:49,990::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3603::WARNING::2012-12-02 23:45:49,990::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3604::WARNING::2012-12-02 23:45:49,991::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3605::WARNING::2012-12-02 23:45:49,991::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3606::WARNING::2012-12-02 23:45:49,992::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3607::WARNING::2012-12-02 23:45:49,993::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3608::WARNING::2012-12-02 23:45:49,993::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3609::WARNING::2012-12-02 23:45:49,995::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3610::WARNING::2012-12-02 23:45:49,995::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3611::WARNING::2012-12-02 23:45:49,995::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3612::WARNING::2012-12-02 23:45:49,995::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3613::WARNING::2012-12-02 23:45:49,996::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3552::WARNING::2012-12-02 23:45:49,999::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3552::DEBUG::2012-12-02 23:45:50,000::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3552::DEBUG::2012-12-02 23:45:50,000::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3552::DEBUG::2012-12-02 23:45:50,205::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3552::WARNING::2012-12-02 23:45:50,206::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3552::DEBUG::2012-12-02 23:45:50,206::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3552::WARNING::2012-12-02 23:45:50,235::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3552::INFO::2012-12-02 23:45:50,236::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3552::DEBUG::2012-12-02 23:45:50,236::task::1151::TaskManager.Task::(prepare) Task=`ee863d0d-e51a-4ba7-9669-aef1a5fd91de`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3552::DEBUG::2012-12-02 23:45:50,236::task::568::TaskManager.Task::(_updateState) Task=`ee863d0d-e51a-4ba7-9669-aef1a5fd91de`::moving from state preparing -> state finished >Thread-3552::DEBUG::2012-12-02 23:45:50,236::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3552::DEBUG::2012-12-02 23:45:50,236::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3552::DEBUG::2012-12-02 23:45:50,237::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3552::DEBUG::2012-12-02 23:45:50,237::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3552::DEBUG::2012-12-02 23:45:50,237::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3552::DEBUG::2012-12-02 23:45:50,237::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3552::DEBUG::2012-12-02 23:45:50,237::task::957::TaskManager.Task::(_decref) Task=`ee863d0d-e51a-4ba7-9669-aef1a5fd91de`::ref 0 aborting False >Thread-3676::DEBUG::2012-12-02 23:45:53,167::task::568::TaskManager.Task::(_updateState) Task=`8659591d-e753-4c9d-be0a-1f4ca8527d38`::moving from state init -> state preparing >Thread-3676::INFO::2012-12-02 23:45:53,167::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3676::INFO::2012-12-02 23:45:53,167::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0088939666748', 'lastCheck': 1354484745.418879, 'code': 0, 'valid': True}} >Thread-3676::DEBUG::2012-12-02 23:45:53,167::task::1151::TaskManager.Task::(prepare) Task=`8659591d-e753-4c9d-be0a-1f4ca8527d38`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0088939666748', 'lastCheck': 1354484745.418879, 'code': 0, 'valid': True}} >Thread-3676::DEBUG::2012-12-02 23:45:53,167::task::568::TaskManager.Task::(_updateState) Task=`8659591d-e753-4c9d-be0a-1f4ca8527d38`::moving from state preparing -> state finished >Thread-3676::DEBUG::2012-12-02 23:45:53,167::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3676::DEBUG::2012-12-02 23:45:53,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3676::DEBUG::2012-12-02 23:45:53,168::task::957::TaskManager.Task::(_decref) Task=`8659591d-e753-4c9d-be0a-1f4ca8527d38`::ref 0 aborting False >Thread-3681::DEBUG::2012-12-02 23:46:00,450::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3681::DEBUG::2012-12-02 23:46:00,450::task::568::TaskManager.Task::(_updateState) Task=`f6fa200b-8191-4fa2-9066-9060ded8ad0d`::moving from state init -> state preparing >Thread-3681::INFO::2012-12-02 23:46:00,450::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3681::INFO::2012-12-02 23:46:00,450::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3681::DEBUG::2012-12-02 23:46:00,451::task::1151::TaskManager.Task::(prepare) Task=`f6fa200b-8191-4fa2-9066-9060ded8ad0d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3681::DEBUG::2012-12-02 23:46:00,451::task::568::TaskManager.Task::(_updateState) Task=`f6fa200b-8191-4fa2-9066-9060ded8ad0d`::moving from state preparing -> state finished >Thread-3681::DEBUG::2012-12-02 23:46:00,451::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3681::DEBUG::2012-12-02 23:46:00,451::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3681::DEBUG::2012-12-02 23:46:00,451::task::957::TaskManager.Task::(_decref) Task=`f6fa200b-8191-4fa2-9066-9060ded8ad0d`::ref 0 aborting False >Thread-3682::DEBUG::2012-12-02 23:46:00,594::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3682::DEBUG::2012-12-02 23:46:00,594::task::568::TaskManager.Task::(_updateState) Task=`43003edb-6f49-4574-a294-410729713d41`::moving from state init -> state preparing >Thread-3682::INFO::2012-12-02 23:46:00,595::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3682::DEBUG::2012-12-02 23:46:00,595::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`835245f7-031c-4422-89f7-86367519d51f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3682::DEBUG::2012-12-02 23:46:00,595::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3682::DEBUG::2012-12-02 23:46:00,595::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3682::DEBUG::2012-12-02 23:46:00,595::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`835245f7-031c-4422-89f7-86367519d51f`::Granted request >Thread-3682::DEBUG::2012-12-02 23:46:00,596::task::794::TaskManager.Task::(resourceAcquired) Task=`43003edb-6f49-4574-a294-410729713d41`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3682::DEBUG::2012-12-02 23:46:00,596::task::957::TaskManager.Task::(_decref) Task=`43003edb-6f49-4574-a294-410729713d41`::ref 1 aborting False >Thread-3682::DEBUG::2012-12-02 23:46:00,602::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3682::DEBUG::2012-12-02 23:46:00,603::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3682::DEBUG::2012-12-02 23:46:00,812::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3682::WARNING::2012-12-02 23:46:00,814::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3682::DEBUG::2012-12-02 23:46:00,814::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3682::WARNING::2012-12-02 23:46:00,844::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3682::DEBUG::2012-12-02 23:46:00,844::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3682::DEBUG::2012-12-02 23:46:00,845::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3682::DEBUG::2012-12-02 23:46:01,075::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3682::WARNING::2012-12-02 23:46:01,077::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3682::DEBUG::2012-12-02 23:46:01,077::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3682::WARNING::2012-12-02 23:46:01,108::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3682::INFO::2012-12-02 23:46:01,108::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3682::DEBUG::2012-12-02 23:46:01,108::task::1151::TaskManager.Task::(prepare) Task=`43003edb-6f49-4574-a294-410729713d41`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3682::DEBUG::2012-12-02 23:46:01,109::task::568::TaskManager.Task::(_updateState) Task=`43003edb-6f49-4574-a294-410729713d41`::moving from state preparing -> state finished >Thread-3682::DEBUG::2012-12-02 23:46:01,109::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3682::DEBUG::2012-12-02 23:46:01,109::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3682::DEBUG::2012-12-02 23:46:01,109::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3682::DEBUG::2012-12-02 23:46:01,109::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3682::DEBUG::2012-12-02 23:46:01,109::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3682::DEBUG::2012-12-02 23:46:01,110::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3682::DEBUG::2012-12-02 23:46:01,110::task::957::TaskManager.Task::(_decref) Task=`43003edb-6f49-4574-a294-410729713d41`::ref 0 aborting False >Thread-3806::DEBUG::2012-12-02 23:46:04,196::task::568::TaskManager.Task::(_updateState) Task=`5a954f9a-bb55-40a9-afde-1fad4cce2d1a`::moving from state init -> state preparing >Thread-3806::INFO::2012-12-02 23:46:04,196::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3806::INFO::2012-12-02 23:46:04,197::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00734496116638', 'lastCheck': 1354484755.4362919, 'code': 0, 'valid': True}} >Thread-3806::DEBUG::2012-12-02 23:46:04,197::task::1151::TaskManager.Task::(prepare) Task=`5a954f9a-bb55-40a9-afde-1fad4cce2d1a`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00734496116638', 'lastCheck': 1354484755.4362919, 'code': 0, 'valid': True}} >Thread-3806::DEBUG::2012-12-02 23:46:04,197::task::568::TaskManager.Task::(_updateState) Task=`5a954f9a-bb55-40a9-afde-1fad4cce2d1a`::moving from state preparing -> state finished >Thread-3806::DEBUG::2012-12-02 23:46:04,197::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3806::DEBUG::2012-12-02 23:46:04,197::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3806::DEBUG::2012-12-02 23:46:04,197::task::957::TaskManager.Task::(_decref) Task=`5a954f9a-bb55-40a9-afde-1fad4cce2d1a`::ref 0 aborting False >Thread-3811::DEBUG::2012-12-02 23:46:11,386::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3811::DEBUG::2012-12-02 23:46:11,386::task::568::TaskManager.Task::(_updateState) Task=`e8fa0075-a929-42f3-bdb0-768d819134ed`::moving from state init -> state preparing >Thread-3811::INFO::2012-12-02 23:46:11,387::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3811::INFO::2012-12-02 23:46:11,387::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3811::DEBUG::2012-12-02 23:46:11,387::task::1151::TaskManager.Task::(prepare) Task=`e8fa0075-a929-42f3-bdb0-768d819134ed`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3811::DEBUG::2012-12-02 23:46:11,387::task::568::TaskManager.Task::(_updateState) Task=`e8fa0075-a929-42f3-bdb0-768d819134ed`::moving from state preparing -> state finished >Thread-3811::DEBUG::2012-12-02 23:46:11,387::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3811::DEBUG::2012-12-02 23:46:11,387::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3811::DEBUG::2012-12-02 23:46:11,387::task::957::TaskManager.Task::(_decref) Task=`e8fa0075-a929-42f3-bdb0-768d819134ed`::ref 0 aborting False >Thread-3812::DEBUG::2012-12-02 23:46:11,566::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3812::DEBUG::2012-12-02 23:46:11,566::task::568::TaskManager.Task::(_updateState) Task=`3b854cc3-3631-426b-b6c4-a927179c67c9`::moving from state init -> state preparing >Thread-3812::INFO::2012-12-02 23:46:11,566::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3812::DEBUG::2012-12-02 23:46:11,567::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`522f1ac8-537e-435a-a509-c939750877b0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3812::DEBUG::2012-12-02 23:46:11,567::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3812::DEBUG::2012-12-02 23:46:11,567::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3812::DEBUG::2012-12-02 23:46:11,567::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`522f1ac8-537e-435a-a509-c939750877b0`::Granted request >Thread-3812::DEBUG::2012-12-02 23:46:11,567::task::794::TaskManager.Task::(resourceAcquired) Task=`3b854cc3-3631-426b-b6c4-a927179c67c9`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3812::DEBUG::2012-12-02 23:46:11,567::task::957::TaskManager.Task::(_decref) Task=`3b854cc3-3631-426b-b6c4-a927179c67c9`::ref 1 aborting False >Thread-3812::DEBUG::2012-12-02 23:46:11,574::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3812::DEBUG::2012-12-02 23:46:11,574::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3812::DEBUG::2012-12-02 23:46:11,781::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3812::WARNING::2012-12-02 23:46:11,783::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3812::DEBUG::2012-12-02 23:46:11,783::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3812::WARNING::2012-12-02 23:46:11,814::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3812::DEBUG::2012-12-02 23:46:11,814::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3812::DEBUG::2012-12-02 23:46:11,814::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3812::DEBUG::2012-12-02 23:46:12,020::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3812::WARNING::2012-12-02 23:46:12,022::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3812::DEBUG::2012-12-02 23:46:12,022::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3812::WARNING::2012-12-02 23:46:12,051::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3812::INFO::2012-12-02 23:46:12,052::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3812::DEBUG::2012-12-02 23:46:12,052::task::1151::TaskManager.Task::(prepare) Task=`3b854cc3-3631-426b-b6c4-a927179c67c9`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3812::DEBUG::2012-12-02 23:46:12,052::task::568::TaskManager.Task::(_updateState) Task=`3b854cc3-3631-426b-b6c4-a927179c67c9`::moving from state preparing -> state finished >Thread-3812::DEBUG::2012-12-02 23:46:12,052::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3812::DEBUG::2012-12-02 23:46:12,052::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3812::DEBUG::2012-12-02 23:46:12,052::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3812::DEBUG::2012-12-02 23:46:12,053::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3812::DEBUG::2012-12-02 23:46:12,053::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3812::DEBUG::2012-12-02 23:46:12,053::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3812::DEBUG::2012-12-02 23:46:12,053::task::957::TaskManager.Task::(_decref) Task=`3b854cc3-3631-426b-b6c4-a927179c67c9`::ref 0 aborting False >Thread-3936::DEBUG::2012-12-02 23:46:16,286::task::568::TaskManager.Task::(_updateState) Task=`0969a3dc-0f06-4d19-9994-49688aaaa40d`::moving from state init -> state preparing >Thread-3936::INFO::2012-12-02 23:46:16,286::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-3936::INFO::2012-12-02 23:46:16,286::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00915884971619', 'lastCheck': 1354484775.4762011, 'code': 0, 'valid': True}} >Thread-3936::DEBUG::2012-12-02 23:46:16,286::task::1151::TaskManager.Task::(prepare) Task=`0969a3dc-0f06-4d19-9994-49688aaaa40d`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00915884971619', 'lastCheck': 1354484775.4762011, 'code': 0, 'valid': True}} >Thread-3936::DEBUG::2012-12-02 23:46:16,286::task::568::TaskManager.Task::(_updateState) Task=`0969a3dc-0f06-4d19-9994-49688aaaa40d`::moving from state preparing -> state finished >Thread-3936::DEBUG::2012-12-02 23:46:16,286::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3936::DEBUG::2012-12-02 23:46:16,286::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3936::DEBUG::2012-12-02 23:46:16,287::task::957::TaskManager.Task::(_decref) Task=`0969a3dc-0f06-4d19-9994-49688aaaa40d`::ref 0 aborting False >Thread-3940::DEBUG::2012-12-02 23:46:22,282::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3940::DEBUG::2012-12-02 23:46:22,282::task::568::TaskManager.Task::(_updateState) Task=`d4a7e02a-a474-44fb-a02c-3f594a042cc3`::moving from state init -> state preparing >Thread-3940::INFO::2012-12-02 23:46:22,282::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3940::INFO::2012-12-02 23:46:22,283::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3940::DEBUG::2012-12-02 23:46:22,283::task::1151::TaskManager.Task::(prepare) Task=`d4a7e02a-a474-44fb-a02c-3f594a042cc3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-3940::DEBUG::2012-12-02 23:46:22,283::task::568::TaskManager.Task::(_updateState) Task=`d4a7e02a-a474-44fb-a02c-3f594a042cc3`::moving from state preparing -> state finished >Thread-3940::DEBUG::2012-12-02 23:46:22,283::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-3940::DEBUG::2012-12-02 23:46:22,283::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3940::DEBUG::2012-12-02 23:46:22,283::task::957::TaskManager.Task::(_decref) Task=`d4a7e02a-a474-44fb-a02c-3f594a042cc3`::ref 0 aborting False >Thread-3941::DEBUG::2012-12-02 23:46:22,425::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-3941::DEBUG::2012-12-02 23:46:22,426::task::568::TaskManager.Task::(_updateState) Task=`61ef3696-9257-4192-ac39-5cf342f0b0ff`::moving from state init -> state preparing >Thread-3941::INFO::2012-12-02 23:46:22,426::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-3941::DEBUG::2012-12-02 23:46:22,426::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a746532-26d8-4524-b20e-b1848605a515`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-3941::DEBUG::2012-12-02 23:46:22,426::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-3941::DEBUG::2012-12-02 23:46:22,427::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-3941::DEBUG::2012-12-02 23:46:22,427::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2a746532-26d8-4524-b20e-b1848605a515`::Granted request >Thread-3941::DEBUG::2012-12-02 23:46:22,427::task::794::TaskManager.Task::(resourceAcquired) Task=`61ef3696-9257-4192-ac39-5cf342f0b0ff`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-3941::DEBUG::2012-12-02 23:46:22,427::task::957::TaskManager.Task::(_decref) Task=`61ef3696-9257-4192-ac39-5cf342f0b0ff`::ref 1 aborting False >Thread-3941::DEBUG::2012-12-02 23:46:22,434::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3941::DEBUG::2012-12-02 23:46:22,434::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-3941::DEBUG::2012-12-02 23:46:22,643::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-3941::WARNING::2012-12-02 23:46:22,645::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-3941::DEBUG::2012-12-02 23:46:22,645::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3941::WARNING::2012-12-02 23:46:22,676::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-3941::DEBUG::2012-12-02 23:46:22,676::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-3941::DEBUG::2012-12-02 23:46:22,677::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-3941::DEBUG::2012-12-02 23:46:22,884::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-3941::WARNING::2012-12-02 23:46:22,885::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-3941::DEBUG::2012-12-02 23:46:22,885::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3941::WARNING::2012-12-02 23:46:22,915::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-3941::INFO::2012-12-02 23:46:22,916::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3941::DEBUG::2012-12-02 23:46:22,916::task::1151::TaskManager.Task::(prepare) Task=`61ef3696-9257-4192-ac39-5cf342f0b0ff`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-3941::DEBUG::2012-12-02 23:46:22,916::task::568::TaskManager.Task::(_updateState) Task=`61ef3696-9257-4192-ac39-5cf342f0b0ff`::moving from state preparing -> state finished >Thread-3941::DEBUG::2012-12-02 23:46:22,916::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-3941::DEBUG::2012-12-02 23:46:22,916::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-3941::DEBUG::2012-12-02 23:46:22,917::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-3941::DEBUG::2012-12-02 23:46:22,917::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-3941::DEBUG::2012-12-02 23:46:22,917::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-3941::DEBUG::2012-12-02 23:46:22,917::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-3941::DEBUG::2012-12-02 23:46:22,917::task::957::TaskManager.Task::(_decref) Task=`61ef3696-9257-4192-ac39-5cf342f0b0ff`::ref 0 aborting False >Thread-4066::DEBUG::2012-12-02 23:46:27,355::task::568::TaskManager.Task::(_updateState) Task=`09d77133-5e02-4526-8bf7-a9e39b13d704`::moving from state init -> state preparing >Thread-4066::INFO::2012-12-02 23:46:27,356::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4066::INFO::2012-12-02 23:46:27,356::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00709009170532', 'lastCheck': 1354484785.494755, 'code': 0, 'valid': True}} >Thread-4066::DEBUG::2012-12-02 23:46:27,356::task::1151::TaskManager.Task::(prepare) Task=`09d77133-5e02-4526-8bf7-a9e39b13d704`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00709009170532', 'lastCheck': 1354484785.494755, 'code': 0, 'valid': True}} >Thread-4066::DEBUG::2012-12-02 23:46:27,356::task::568::TaskManager.Task::(_updateState) Task=`09d77133-5e02-4526-8bf7-a9e39b13d704`::moving from state preparing -> state finished >Thread-4066::DEBUG::2012-12-02 23:46:27,356::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4066::DEBUG::2012-12-02 23:46:27,356::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4066::DEBUG::2012-12-02 23:46:27,356::task::957::TaskManager.Task::(_decref) Task=`09d77133-5e02-4526-8bf7-a9e39b13d704`::ref 0 aborting False >Thread-4070::DEBUG::2012-12-02 23:46:33,158::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4070::DEBUG::2012-12-02 23:46:33,159::task::568::TaskManager.Task::(_updateState) Task=`706d5576-af69-474c-964f-9e4d3bcf308b`::moving from state init -> state preparing >Thread-4070::INFO::2012-12-02 23:46:33,159::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4070::INFO::2012-12-02 23:46:33,159::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4070::DEBUG::2012-12-02 23:46:33,159::task::1151::TaskManager.Task::(prepare) Task=`706d5576-af69-474c-964f-9e4d3bcf308b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4070::DEBUG::2012-12-02 23:46:33,159::task::568::TaskManager.Task::(_updateState) Task=`706d5576-af69-474c-964f-9e4d3bcf308b`::moving from state preparing -> state finished >Thread-4070::DEBUG::2012-12-02 23:46:33,159::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4070::DEBUG::2012-12-02 23:46:33,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4070::DEBUG::2012-12-02 23:46:33,160::task::957::TaskManager.Task::(_decref) Task=`706d5576-af69-474c-964f-9e4d3bcf308b`::ref 0 aborting False >Thread-4071::DEBUG::2012-12-02 23:46:33,301::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4071::DEBUG::2012-12-02 23:46:33,301::task::568::TaskManager.Task::(_updateState) Task=`b840d86b-8da7-4a77-a611-8a0873b67f17`::moving from state init -> state preparing >Thread-4071::INFO::2012-12-02 23:46:33,301::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4071::DEBUG::2012-12-02 23:46:33,302::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`791e9f77-150c-4674-8253-fef851d90ec5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4071::DEBUG::2012-12-02 23:46:33,302::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4071::DEBUG::2012-12-02 23:46:33,302::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4071::DEBUG::2012-12-02 23:46:33,302::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`791e9f77-150c-4674-8253-fef851d90ec5`::Granted request >Thread-4071::DEBUG::2012-12-02 23:46:33,302::task::794::TaskManager.Task::(resourceAcquired) Task=`b840d86b-8da7-4a77-a611-8a0873b67f17`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4071::DEBUG::2012-12-02 23:46:33,303::task::957::TaskManager.Task::(_decref) Task=`b840d86b-8da7-4a77-a611-8a0873b67f17`::ref 1 aborting False >Thread-4071::DEBUG::2012-12-02 23:46:33,308::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4071::DEBUG::2012-12-02 23:46:33,308::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4071::DEBUG::2012-12-02 23:46:33,516::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4071::WARNING::2012-12-02 23:46:33,518::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4071::DEBUG::2012-12-02 23:46:33,518::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4071::WARNING::2012-12-02 23:46:33,548::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4071::DEBUG::2012-12-02 23:46:33,549::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4071::DEBUG::2012-12-02 23:46:33,549::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4071::DEBUG::2012-12-02 23:46:33,762::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4071::WARNING::2012-12-02 23:46:33,763::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4071::DEBUG::2012-12-02 23:46:33,764::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4071::WARNING::2012-12-02 23:46:33,793::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4071::INFO::2012-12-02 23:46:33,793::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4071::DEBUG::2012-12-02 23:46:33,794::task::1151::TaskManager.Task::(prepare) Task=`b840d86b-8da7-4a77-a611-8a0873b67f17`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4071::DEBUG::2012-12-02 23:46:33,794::task::568::TaskManager.Task::(_updateState) Task=`b840d86b-8da7-4a77-a611-8a0873b67f17`::moving from state preparing -> state finished >Thread-4071::DEBUG::2012-12-02 23:46:33,794::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4071::DEBUG::2012-12-02 23:46:33,794::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4071::DEBUG::2012-12-02 23:46:33,794::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4071::DEBUG::2012-12-02 23:46:33,794::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4071::DEBUG::2012-12-02 23:46:33,795::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4071::DEBUG::2012-12-02 23:46:33,795::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4071::DEBUG::2012-12-02 23:46:33,795::task::957::TaskManager.Task::(_decref) Task=`b840d86b-8da7-4a77-a611-8a0873b67f17`::ref 0 aborting False >Thread-4196::DEBUG::2012-12-02 23:46:38,339::task::568::TaskManager.Task::(_updateState) Task=`91257ac1-e339-43bc-ab42-7a959cb9cc9f`::moving from state init -> state preparing >Thread-4196::INFO::2012-12-02 23:46:38,339::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4196::INFO::2012-12-02 23:46:38,339::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00804591178894', 'lastCheck': 1354484795.5128591, 'code': 0, 'valid': True}} >Thread-4196::DEBUG::2012-12-02 23:46:38,339::task::1151::TaskManager.Task::(prepare) Task=`91257ac1-e339-43bc-ab42-7a959cb9cc9f`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00804591178894', 'lastCheck': 1354484795.5128591, 'code': 0, 'valid': True}} >Thread-4196::DEBUG::2012-12-02 23:46:38,339::task::568::TaskManager.Task::(_updateState) Task=`91257ac1-e339-43bc-ab42-7a959cb9cc9f`::moving from state preparing -> state finished >Thread-4196::DEBUG::2012-12-02 23:46:38,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4196::DEBUG::2012-12-02 23:46:38,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4196::DEBUG::2012-12-02 23:46:38,340::task::957::TaskManager.Task::(_decref) Task=`91257ac1-e339-43bc-ab42-7a959cb9cc9f`::ref 0 aborting False >Thread-4200::DEBUG::2012-12-02 23:46:44,004::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4200::DEBUG::2012-12-02 23:46:44,004::task::568::TaskManager.Task::(_updateState) Task=`53987e04-c3df-40d5-a6e9-c8d113428e7c`::moving from state init -> state preparing >Thread-4200::INFO::2012-12-02 23:46:44,004::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4200::INFO::2012-12-02 23:46:44,004::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4200::DEBUG::2012-12-02 23:46:44,004::task::1151::TaskManager.Task::(prepare) Task=`53987e04-c3df-40d5-a6e9-c8d113428e7c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4200::DEBUG::2012-12-02 23:46:44,005::task::568::TaskManager.Task::(_updateState) Task=`53987e04-c3df-40d5-a6e9-c8d113428e7c`::moving from state preparing -> state finished >Thread-4200::DEBUG::2012-12-02 23:46:44,005::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4200::DEBUG::2012-12-02 23:46:44,005::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4200::DEBUG::2012-12-02 23:46:44,005::task::957::TaskManager.Task::(_decref) Task=`53987e04-c3df-40d5-a6e9-c8d113428e7c`::ref 0 aborting False >Thread-4201::DEBUG::2012-12-02 23:46:44,148::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4201::DEBUG::2012-12-02 23:46:44,148::task::568::TaskManager.Task::(_updateState) Task=`2a8245c8-e3b4-4610-9545-b29a0f087a72`::moving from state init -> state preparing >Thread-4201::INFO::2012-12-02 23:46:44,148::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4201::DEBUG::2012-12-02 23:46:44,149::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e9ae8756-f974-4f20-860c-2489bafc42e5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4201::DEBUG::2012-12-02 23:46:44,149::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4201::DEBUG::2012-12-02 23:46:44,149::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4201::DEBUG::2012-12-02 23:46:44,149::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e9ae8756-f974-4f20-860c-2489bafc42e5`::Granted request >Thread-4201::DEBUG::2012-12-02 23:46:44,150::task::794::TaskManager.Task::(resourceAcquired) Task=`2a8245c8-e3b4-4610-9545-b29a0f087a72`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4201::DEBUG::2012-12-02 23:46:44,150::task::957::TaskManager.Task::(_decref) Task=`2a8245c8-e3b4-4610-9545-b29a0f087a72`::ref 1 aborting False >Thread-4201::DEBUG::2012-12-02 23:46:44,156::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4201::DEBUG::2012-12-02 23:46:44,156::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4201::DEBUG::2012-12-02 23:46:44,364::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4201::WARNING::2012-12-02 23:46:44,366::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4201::DEBUG::2012-12-02 23:46:44,366::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4201::WARNING::2012-12-02 23:46:44,402::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4201::DEBUG::2012-12-02 23:46:44,402::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4201::DEBUG::2012-12-02 23:46:44,402::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4201::DEBUG::2012-12-02 23:46:44,608::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4201::WARNING::2012-12-02 23:46:44,610::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4201::DEBUG::2012-12-02 23:46:44,610::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4201::WARNING::2012-12-02 23:46:44,641::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4201::INFO::2012-12-02 23:46:44,641::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4201::DEBUG::2012-12-02 23:46:44,641::task::1151::TaskManager.Task::(prepare) Task=`2a8245c8-e3b4-4610-9545-b29a0f087a72`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4201::DEBUG::2012-12-02 23:46:44,642::task::568::TaskManager.Task::(_updateState) Task=`2a8245c8-e3b4-4610-9545-b29a0f087a72`::moving from state preparing -> state finished >Thread-4201::DEBUG::2012-12-02 23:46:44,642::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4201::DEBUG::2012-12-02 23:46:44,642::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4201::DEBUG::2012-12-02 23:46:44,642::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4201::DEBUG::2012-12-02 23:46:44,642::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4201::DEBUG::2012-12-02 23:46:44,642::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4201::DEBUG::2012-12-02 23:46:44,643::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4201::DEBUG::2012-12-02 23:46:44,643::task::957::TaskManager.Task::(_decref) Task=`2a8245c8-e3b4-4610-9545-b29a0f087a72`::ref 0 aborting False >Thread-4326::DEBUG::2012-12-02 23:46:49,291::task::568::TaskManager.Task::(_updateState) Task=`644cce51-24b3-49e7-b198-3d64d0218e80`::moving from state init -> state preparing >Thread-4326::INFO::2012-12-02 23:46:49,291::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4326::INFO::2012-12-02 23:46:49,292::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00864386558533', 'lastCheck': 1354484805.5327151, 'code': 0, 'valid': True}} >Thread-4326::DEBUG::2012-12-02 23:46:49,292::task::1151::TaskManager.Task::(prepare) Task=`644cce51-24b3-49e7-b198-3d64d0218e80`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00864386558533', 'lastCheck': 1354484805.5327151, 'code': 0, 'valid': True}} >Thread-4326::DEBUG::2012-12-02 23:46:49,292::task::568::TaskManager.Task::(_updateState) Task=`644cce51-24b3-49e7-b198-3d64d0218e80`::moving from state preparing -> state finished >Thread-4326::DEBUG::2012-12-02 23:46:49,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4326::DEBUG::2012-12-02 23:46:49,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4326::DEBUG::2012-12-02 23:46:49,292::task::957::TaskManager.Task::(_decref) Task=`644cce51-24b3-49e7-b198-3d64d0218e80`::ref 0 aborting False >Thread-4330::DEBUG::2012-12-02 23:46:54,872::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4330::DEBUG::2012-12-02 23:46:54,873::task::568::TaskManager.Task::(_updateState) Task=`7bb41ee6-d9dc-4231-bddc-27169a66e39d`::moving from state init -> state preparing >Thread-4330::INFO::2012-12-02 23:46:54,873::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4330::INFO::2012-12-02 23:46:54,873::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4330::DEBUG::2012-12-02 23:46:54,873::task::1151::TaskManager.Task::(prepare) Task=`7bb41ee6-d9dc-4231-bddc-27169a66e39d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4330::DEBUG::2012-12-02 23:46:54,874::task::568::TaskManager.Task::(_updateState) Task=`7bb41ee6-d9dc-4231-bddc-27169a66e39d`::moving from state preparing -> state finished >Thread-4330::DEBUG::2012-12-02 23:46:54,874::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4330::DEBUG::2012-12-02 23:46:54,874::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4330::DEBUG::2012-12-02 23:46:54,874::task::957::TaskManager.Task::(_decref) Task=`7bb41ee6-d9dc-4231-bddc-27169a66e39d`::ref 0 aborting False >Thread-4331::DEBUG::2012-12-02 23:46:55,013::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4331::DEBUG::2012-12-02 23:46:55,013::task::568::TaskManager.Task::(_updateState) Task=`fcce38b6-20b0-42cd-9e28-868855da4526`::moving from state init -> state preparing >Thread-4331::INFO::2012-12-02 23:46:55,013::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4331::DEBUG::2012-12-02 23:46:55,013::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e41e8fd1-cf9d-466b-a313-143f81d9f06c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4331::DEBUG::2012-12-02 23:46:55,013::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4331::DEBUG::2012-12-02 23:46:55,014::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4331::DEBUG::2012-12-02 23:46:55,014::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e41e8fd1-cf9d-466b-a313-143f81d9f06c`::Granted request >Thread-4331::DEBUG::2012-12-02 23:46:55,014::task::794::TaskManager.Task::(resourceAcquired) Task=`fcce38b6-20b0-42cd-9e28-868855da4526`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4331::DEBUG::2012-12-02 23:46:55,014::task::957::TaskManager.Task::(_decref) Task=`fcce38b6-20b0-42cd-9e28-868855da4526`::ref 1 aborting False >Thread-4331::DEBUG::2012-12-02 23:46:55,019::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4331::DEBUG::2012-12-02 23:46:55,019::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4331::DEBUG::2012-12-02 23:46:55,225::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4331::WARNING::2012-12-02 23:46:55,226::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4331::DEBUG::2012-12-02 23:46:55,227::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4331::WARNING::2012-12-02 23:46:55,256::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4331::DEBUG::2012-12-02 23:46:55,256::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4331::DEBUG::2012-12-02 23:46:55,257::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4331::DEBUG::2012-12-02 23:46:55,465::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4331::WARNING::2012-12-02 23:46:55,466::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4331::DEBUG::2012-12-02 23:46:55,466::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4331::WARNING::2012-12-02 23:46:55,495::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4331::INFO::2012-12-02 23:46:55,495::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4331::DEBUG::2012-12-02 23:46:55,495::task::1151::TaskManager.Task::(prepare) Task=`fcce38b6-20b0-42cd-9e28-868855da4526`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4331::DEBUG::2012-12-02 23:46:55,495::task::568::TaskManager.Task::(_updateState) Task=`fcce38b6-20b0-42cd-9e28-868855da4526`::moving from state preparing -> state finished >Thread-4331::DEBUG::2012-12-02 23:46:55,496::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4331::DEBUG::2012-12-02 23:46:55,496::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4331::DEBUG::2012-12-02 23:46:55,496::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4331::DEBUG::2012-12-02 23:46:55,496::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4331::DEBUG::2012-12-02 23:46:55,496::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4331::DEBUG::2012-12-02 23:46:55,496::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4331::DEBUG::2012-12-02 23:46:55,497::task::957::TaskManager.Task::(_decref) Task=`fcce38b6-20b0-42cd-9e28-868855da4526`::ref 0 aborting False >Thread-4456::DEBUG::2012-12-02 23:47:00,283::task::568::TaskManager.Task::(_updateState) Task=`7b359f0e-d4ea-4ebe-aba9-704dfc262f27`::moving from state init -> state preparing >Thread-4456::INFO::2012-12-02 23:47:00,283::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4456::INFO::2012-12-02 23:47:00,283::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00691509246826', 'lastCheck': 1354484815.549911, 'code': 0, 'valid': True}} >Thread-4456::DEBUG::2012-12-02 23:47:00,284::task::1151::TaskManager.Task::(prepare) Task=`7b359f0e-d4ea-4ebe-aba9-704dfc262f27`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00691509246826', 'lastCheck': 1354484815.549911, 'code': 0, 'valid': True}} >Thread-4456::DEBUG::2012-12-02 23:47:00,284::task::568::TaskManager.Task::(_updateState) Task=`7b359f0e-d4ea-4ebe-aba9-704dfc262f27`::moving from state preparing -> state finished >Thread-4456::DEBUG::2012-12-02 23:47:00,284::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4456::DEBUG::2012-12-02 23:47:00,284::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4456::DEBUG::2012-12-02 23:47:00,284::task::957::TaskManager.Task::(_decref) Task=`7b359f0e-d4ea-4ebe-aba9-704dfc262f27`::ref 0 aborting False >Thread-4460::DEBUG::2012-12-02 23:47:05,752::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4460::DEBUG::2012-12-02 23:47:05,752::task::568::TaskManager.Task::(_updateState) Task=`167a22af-9a8c-4a80-a216-09e3b21393fb`::moving from state init -> state preparing >Thread-4460::INFO::2012-12-02 23:47:05,752::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4460::INFO::2012-12-02 23:47:05,753::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4460::DEBUG::2012-12-02 23:47:05,753::task::1151::TaskManager.Task::(prepare) Task=`167a22af-9a8c-4a80-a216-09e3b21393fb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4460::DEBUG::2012-12-02 23:47:05,753::task::568::TaskManager.Task::(_updateState) Task=`167a22af-9a8c-4a80-a216-09e3b21393fb`::moving from state preparing -> state finished >Thread-4460::DEBUG::2012-12-02 23:47:05,753::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4460::DEBUG::2012-12-02 23:47:05,753::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4460::DEBUG::2012-12-02 23:47:05,753::task::957::TaskManager.Task::(_decref) Task=`167a22af-9a8c-4a80-a216-09e3b21393fb`::ref 0 aborting False >Thread-4461::DEBUG::2012-12-02 23:47:05,892::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4461::DEBUG::2012-12-02 23:47:05,892::task::568::TaskManager.Task::(_updateState) Task=`0ccf2e89-63a7-43bf-b2ef-3420c56be271`::moving from state init -> state preparing >Thread-4461::INFO::2012-12-02 23:47:05,892::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4461::DEBUG::2012-12-02 23:47:05,893::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`52204862-2b68-4627-8afc-e930f1c903d3`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4461::DEBUG::2012-12-02 23:47:05,893::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4461::DEBUG::2012-12-02 23:47:05,893::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4461::DEBUG::2012-12-02 23:47:05,893::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`52204862-2b68-4627-8afc-e930f1c903d3`::Granted request >Thread-4461::DEBUG::2012-12-02 23:47:05,893::task::794::TaskManager.Task::(resourceAcquired) Task=`0ccf2e89-63a7-43bf-b2ef-3420c56be271`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4461::DEBUG::2012-12-02 23:47:05,893::task::957::TaskManager.Task::(_decref) Task=`0ccf2e89-63a7-43bf-b2ef-3420c56be271`::ref 1 aborting False >Thread-4461::DEBUG::2012-12-02 23:47:05,898::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4461::DEBUG::2012-12-02 23:47:05,899::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4461::DEBUG::2012-12-02 23:47:06,105::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4461::WARNING::2012-12-02 23:47:06,106::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4461::DEBUG::2012-12-02 23:47:06,106::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4461::WARNING::2012-12-02 23:47:06,135::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4461::DEBUG::2012-12-02 23:47:06,136::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4461::DEBUG::2012-12-02 23:47:06,136::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4461::DEBUG::2012-12-02 23:47:06,344::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4461::WARNING::2012-12-02 23:47:06,345::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4461::DEBUG::2012-12-02 23:47:06,345::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4461::WARNING::2012-12-02 23:47:06,374::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4461::INFO::2012-12-02 23:47:06,374::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4461::DEBUG::2012-12-02 23:47:06,374::task::1151::TaskManager.Task::(prepare) Task=`0ccf2e89-63a7-43bf-b2ef-3420c56be271`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4461::DEBUG::2012-12-02 23:47:06,375::task::568::TaskManager.Task::(_updateState) Task=`0ccf2e89-63a7-43bf-b2ef-3420c56be271`::moving from state preparing -> state finished >Thread-4461::DEBUG::2012-12-02 23:47:06,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4461::DEBUG::2012-12-02 23:47:06,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4461::DEBUG::2012-12-02 23:47:06,375::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4461::DEBUG::2012-12-02 23:47:06,375::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4461::DEBUG::2012-12-02 23:47:06,375::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4461::DEBUG::2012-12-02 23:47:06,376::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4461::DEBUG::2012-12-02 23:47:06,376::task::957::TaskManager.Task::(_decref) Task=`0ccf2e89-63a7-43bf-b2ef-3420c56be271`::ref 0 aborting False >Thread-4586::DEBUG::2012-12-02 23:47:11,240::task::568::TaskManager.Task::(_updateState) Task=`f9c7dc87-32cb-461a-b568-87b848975b8a`::moving from state init -> state preparing >Thread-4586::INFO::2012-12-02 23:47:11,241::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4586::INFO::2012-12-02 23:47:11,241::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00736403465271', 'lastCheck': 1354484825.567349, 'code': 0, 'valid': True}} >Thread-4586::DEBUG::2012-12-02 23:47:11,241::task::1151::TaskManager.Task::(prepare) Task=`f9c7dc87-32cb-461a-b568-87b848975b8a`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00736403465271', 'lastCheck': 1354484825.567349, 'code': 0, 'valid': True}} >Thread-4586::DEBUG::2012-12-02 23:47:11,241::task::568::TaskManager.Task::(_updateState) Task=`f9c7dc87-32cb-461a-b568-87b848975b8a`::moving from state preparing -> state finished >Thread-4586::DEBUG::2012-12-02 23:47:11,242::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4586::DEBUG::2012-12-02 23:47:11,242::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4586::DEBUG::2012-12-02 23:47:11,242::task::957::TaskManager.Task::(_decref) Task=`f9c7dc87-32cb-461a-b568-87b848975b8a`::ref 0 aborting False >Thread-2443::DEBUG::2012-12-02 23:47:15,567::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 5971512c-ee51-475e-b956-3872c38ce193 >Thread-2443::DEBUG::2012-12-02 23:47:15,567::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2443::DEBUG::2012-12-02 23:47:15,568::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-2443::DEBUG::2012-12-02 23:47:15,776::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-2443::WARNING::2012-12-02 23:47:15,778::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-2443::DEBUG::2012-12-02 23:47:15,778::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2443::DEBUG::2012-12-02 23:47:15,813::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-2443::DEBUG::2012-12-02 23:47:15,814::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-2443::DEBUG::2012-12-02 23:47:15,823::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >Thread-2443::DEBUG::2012-12-02 23:47:15,826::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-2443::WARNING::2012-12-02 23:47:15,826::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >Thread-2443::WARNING::2012-12-02 23:47:15,826::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >Thread-4651::DEBUG::2012-12-02 23:47:16,591::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4651::DEBUG::2012-12-02 23:47:16,592::task::568::TaskManager.Task::(_updateState) Task=`43e550fd-83f8-423c-ba7e-620f3f6da4bf`::moving from state init -> state preparing >Thread-4651::INFO::2012-12-02 23:47:16,592::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4651::INFO::2012-12-02 23:47:16,592::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4651::DEBUG::2012-12-02 23:47:16,593::task::1151::TaskManager.Task::(prepare) Task=`43e550fd-83f8-423c-ba7e-620f3f6da4bf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4651::DEBUG::2012-12-02 23:47:16,593::task::568::TaskManager.Task::(_updateState) Task=`43e550fd-83f8-423c-ba7e-620f3f6da4bf`::moving from state preparing -> state finished >Thread-4651::DEBUG::2012-12-02 23:47:16,593::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4651::DEBUG::2012-12-02 23:47:16,593::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4651::DEBUG::2012-12-02 23:47:16,593::task::957::TaskManager.Task::(_decref) Task=`43e550fd-83f8-423c-ba7e-620f3f6da4bf`::ref 0 aborting False >Thread-4652::DEBUG::2012-12-02 23:47:16,732::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4652::DEBUG::2012-12-02 23:47:16,732::task::568::TaskManager.Task::(_updateState) Task=`8269bfc0-4554-4241-837a-441a0e9e7471`::moving from state init -> state preparing >Thread-4652::INFO::2012-12-02 23:47:16,732::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4652::DEBUG::2012-12-02 23:47:16,732::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`07de40d3-78c7-4bac-87eb-9d5fe5378e6f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4652::DEBUG::2012-12-02 23:47:16,733::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4652::DEBUG::2012-12-02 23:47:16,733::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4652::DEBUG::2012-12-02 23:47:16,733::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`07de40d3-78c7-4bac-87eb-9d5fe5378e6f`::Granted request >Thread-4652::DEBUG::2012-12-02 23:47:16,733::task::794::TaskManager.Task::(resourceAcquired) Task=`8269bfc0-4554-4241-837a-441a0e9e7471`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4652::DEBUG::2012-12-02 23:47:16,733::task::957::TaskManager.Task::(_decref) Task=`8269bfc0-4554-4241-837a-441a0e9e7471`::ref 1 aborting False >Thread-4652::DEBUG::2012-12-02 23:47:16,738::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4652::DEBUG::2012-12-02 23:47:16,738::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4652::DEBUG::2012-12-02 23:47:16,945::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4652::WARNING::2012-12-02 23:47:16,946::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4652::DEBUG::2012-12-02 23:47:16,947::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4652::WARNING::2012-12-02 23:47:16,975::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4652::DEBUG::2012-12-02 23:47:16,976::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4652::DEBUG::2012-12-02 23:47:16,976::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4652::DEBUG::2012-12-02 23:47:17,184::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4652::WARNING::2012-12-02 23:47:17,185::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4652::DEBUG::2012-12-02 23:47:17,185::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4652::WARNING::2012-12-02 23:47:17,215::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4652::INFO::2012-12-02 23:47:17,215::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4652::DEBUG::2012-12-02 23:47:17,215::task::1151::TaskManager.Task::(prepare) Task=`8269bfc0-4554-4241-837a-441a0e9e7471`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4652::DEBUG::2012-12-02 23:47:17,215::task::568::TaskManager.Task::(_updateState) Task=`8269bfc0-4554-4241-837a-441a0e9e7471`::moving from state preparing -> state finished >Thread-4652::DEBUG::2012-12-02 23:47:17,216::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4652::DEBUG::2012-12-02 23:47:17,216::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4652::DEBUG::2012-12-02 23:47:17,216::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4652::DEBUG::2012-12-02 23:47:17,216::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4652::DEBUG::2012-12-02 23:47:17,216::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4652::DEBUG::2012-12-02 23:47:17,216::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4652::DEBUG::2012-12-02 23:47:17,217::task::957::TaskManager.Task::(_decref) Task=`8269bfc0-4554-4241-837a-441a0e9e7471`::ref 0 aborting False >Thread-4777::DEBUG::2012-12-02 23:47:22,185::task::568::TaskManager.Task::(_updateState) Task=`c576c8b3-719d-41f2-939c-d55c970827d8`::moving from state init -> state preparing >Thread-4777::INFO::2012-12-02 23:47:22,186::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4777::INFO::2012-12-02 23:47:22,186::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00656318664551', 'lastCheck': 1354484835.8453751, 'code': 0, 'valid': True}} >Thread-4777::DEBUG::2012-12-02 23:47:22,186::task::1151::TaskManager.Task::(prepare) Task=`c576c8b3-719d-41f2-939c-d55c970827d8`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00656318664551', 'lastCheck': 1354484835.8453751, 'code': 0, 'valid': True}} >Thread-4777::DEBUG::2012-12-02 23:47:22,187::task::568::TaskManager.Task::(_updateState) Task=`c576c8b3-719d-41f2-939c-d55c970827d8`::moving from state preparing -> state finished >Thread-4777::DEBUG::2012-12-02 23:47:22,187::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4777::DEBUG::2012-12-02 23:47:22,187::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4777::DEBUG::2012-12-02 23:47:22,187::task::957::TaskManager.Task::(_decref) Task=`c576c8b3-719d-41f2-939c-d55c970827d8`::ref 0 aborting False >Thread-4781::DEBUG::2012-12-02 23:47:27,477::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4781::DEBUG::2012-12-02 23:47:27,477::task::568::TaskManager.Task::(_updateState) Task=`22bbfe5a-4abc-4bb6-afd2-0fa0deb23dd9`::moving from state init -> state preparing >Thread-4781::INFO::2012-12-02 23:47:27,477::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4781::INFO::2012-12-02 23:47:27,477::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4781::DEBUG::2012-12-02 23:47:27,477::task::1151::TaskManager.Task::(prepare) Task=`22bbfe5a-4abc-4bb6-afd2-0fa0deb23dd9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4781::DEBUG::2012-12-02 23:47:27,478::task::568::TaskManager.Task::(_updateState) Task=`22bbfe5a-4abc-4bb6-afd2-0fa0deb23dd9`::moving from state preparing -> state finished >Thread-4781::DEBUG::2012-12-02 23:47:27,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4781::DEBUG::2012-12-02 23:47:27,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4781::DEBUG::2012-12-02 23:47:27,478::task::957::TaskManager.Task::(_decref) Task=`22bbfe5a-4abc-4bb6-afd2-0fa0deb23dd9`::ref 0 aborting False >Thread-4782::DEBUG::2012-12-02 23:47:27,612::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4782::DEBUG::2012-12-02 23:47:27,613::task::568::TaskManager.Task::(_updateState) Task=`d281073c-a5fd-4e08-8875-137e5f2ed74c`::moving from state init -> state preparing >Thread-4782::INFO::2012-12-02 23:47:27,613::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4782::DEBUG::2012-12-02 23:47:27,613::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2fae6452-cbe1-464d-9917-d56ffc99e25c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4782::DEBUG::2012-12-02 23:47:27,613::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4782::DEBUG::2012-12-02 23:47:27,613::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4782::DEBUG::2012-12-02 23:47:27,614::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`2fae6452-cbe1-464d-9917-d56ffc99e25c`::Granted request >Thread-4782::DEBUG::2012-12-02 23:47:27,614::task::794::TaskManager.Task::(resourceAcquired) Task=`d281073c-a5fd-4e08-8875-137e5f2ed74c`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4782::DEBUG::2012-12-02 23:47:27,614::task::957::TaskManager.Task::(_decref) Task=`d281073c-a5fd-4e08-8875-137e5f2ed74c`::ref 1 aborting False >Thread-4782::DEBUG::2012-12-02 23:47:27,619::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4782::DEBUG::2012-12-02 23:47:27,619::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4782::DEBUG::2012-12-02 23:47:27,825::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4782::WARNING::2012-12-02 23:47:27,826::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4782::DEBUG::2012-12-02 23:47:27,826::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4782::WARNING::2012-12-02 23:47:27,855::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4782::DEBUG::2012-12-02 23:47:27,856::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4782::DEBUG::2012-12-02 23:47:27,856::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4782::DEBUG::2012-12-02 23:47:28,063::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4782::WARNING::2012-12-02 23:47:28,065::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4782::DEBUG::2012-12-02 23:47:28,065::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4782::WARNING::2012-12-02 23:47:28,094::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4782::INFO::2012-12-02 23:47:28,095::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4782::DEBUG::2012-12-02 23:47:28,095::task::1151::TaskManager.Task::(prepare) Task=`d281073c-a5fd-4e08-8875-137e5f2ed74c`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4782::DEBUG::2012-12-02 23:47:28,095::task::568::TaskManager.Task::(_updateState) Task=`d281073c-a5fd-4e08-8875-137e5f2ed74c`::moving from state preparing -> state finished >Thread-4782::DEBUG::2012-12-02 23:47:28,095::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4782::DEBUG::2012-12-02 23:47:28,095::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4782::DEBUG::2012-12-02 23:47:28,096::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4782::DEBUG::2012-12-02 23:47:28,096::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4782::DEBUG::2012-12-02 23:47:28,096::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4782::DEBUG::2012-12-02 23:47:28,096::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4782::DEBUG::2012-12-02 23:47:28,096::task::957::TaskManager.Task::(_decref) Task=`d281073c-a5fd-4e08-8875-137e5f2ed74c`::ref 0 aborting False >Thread-4907::DEBUG::2012-12-02 23:47:33,199::task::568::TaskManager.Task::(_updateState) Task=`55819ffc-8d98-4aa1-a08e-c22190ba8e08`::moving from state init -> state preparing >Thread-4907::INFO::2012-12-02 23:47:33,200::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-4907::INFO::2012-12-02 23:47:33,200::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00756096839905', 'lastCheck': 1354484845.8647649, 'code': 0, 'valid': True}} >Thread-4907::DEBUG::2012-12-02 23:47:33,200::task::1151::TaskManager.Task::(prepare) Task=`55819ffc-8d98-4aa1-a08e-c22190ba8e08`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00756096839905', 'lastCheck': 1354484845.8647649, 'code': 0, 'valid': True}} >Thread-4907::DEBUG::2012-12-02 23:47:33,200::task::568::TaskManager.Task::(_updateState) Task=`55819ffc-8d98-4aa1-a08e-c22190ba8e08`::moving from state preparing -> state finished >Thread-4907::DEBUG::2012-12-02 23:47:33,201::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4907::DEBUG::2012-12-02 23:47:33,201::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4907::DEBUG::2012-12-02 23:47:33,201::task::957::TaskManager.Task::(_decref) Task=`55819ffc-8d98-4aa1-a08e-c22190ba8e08`::ref 0 aborting False >Thread-4911::DEBUG::2012-12-02 23:47:38,325::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4911::DEBUG::2012-12-02 23:47:38,325::task::568::TaskManager.Task::(_updateState) Task=`776ac3b8-838e-4964-bf33-d1a82baa5a02`::moving from state init -> state preparing >Thread-4911::INFO::2012-12-02 23:47:38,325::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4911::INFO::2012-12-02 23:47:38,325::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4911::DEBUG::2012-12-02 23:47:38,326::task::1151::TaskManager.Task::(prepare) Task=`776ac3b8-838e-4964-bf33-d1a82baa5a02`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-4911::DEBUG::2012-12-02 23:47:38,326::task::568::TaskManager.Task::(_updateState) Task=`776ac3b8-838e-4964-bf33-d1a82baa5a02`::moving from state preparing -> state finished >Thread-4911::DEBUG::2012-12-02 23:47:38,326::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-4911::DEBUG::2012-12-02 23:47:38,326::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4911::DEBUG::2012-12-02 23:47:38,326::task::957::TaskManager.Task::(_decref) Task=`776ac3b8-838e-4964-bf33-d1a82baa5a02`::ref 0 aborting False >Thread-4912::DEBUG::2012-12-02 23:47:38,465::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-4912::DEBUG::2012-12-02 23:47:38,465::task::568::TaskManager.Task::(_updateState) Task=`aad7bf42-0968-4ef2-9c0f-63f6178e05e4`::moving from state init -> state preparing >Thread-4912::INFO::2012-12-02 23:47:38,465::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-4912::DEBUG::2012-12-02 23:47:38,466::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1bab7033-9e9e-44b5-8d09-0c2ccf12f0ec`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-4912::DEBUG::2012-12-02 23:47:38,466::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-4912::DEBUG::2012-12-02 23:47:38,466::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-4912::DEBUG::2012-12-02 23:47:38,466::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1bab7033-9e9e-44b5-8d09-0c2ccf12f0ec`::Granted request >Thread-4912::DEBUG::2012-12-02 23:47:38,466::task::794::TaskManager.Task::(resourceAcquired) Task=`aad7bf42-0968-4ef2-9c0f-63f6178e05e4`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-4912::DEBUG::2012-12-02 23:47:38,466::task::957::TaskManager.Task::(_decref) Task=`aad7bf42-0968-4ef2-9c0f-63f6178e05e4`::ref 1 aborting False >Thread-4912::DEBUG::2012-12-02 23:47:38,473::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4912::DEBUG::2012-12-02 23:47:38,473::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-4912::DEBUG::2012-12-02 23:47:38,682::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-4912::WARNING::2012-12-02 23:47:38,683::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-4912::DEBUG::2012-12-02 23:47:38,683::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4912::WARNING::2012-12-02 23:47:38,713::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-4912::DEBUG::2012-12-02 23:47:38,714::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-4912::DEBUG::2012-12-02 23:47:38,714::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-4912::DEBUG::2012-12-02 23:47:38,919::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-4912::WARNING::2012-12-02 23:47:38,920::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-4912::DEBUG::2012-12-02 23:47:38,920::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4912::WARNING::2012-12-02 23:47:38,950::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-4912::INFO::2012-12-02 23:47:38,950::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4912::DEBUG::2012-12-02 23:47:38,950::task::1151::TaskManager.Task::(prepare) Task=`aad7bf42-0968-4ef2-9c0f-63f6178e05e4`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-4912::DEBUG::2012-12-02 23:47:38,951::task::568::TaskManager.Task::(_updateState) Task=`aad7bf42-0968-4ef2-9c0f-63f6178e05e4`::moving from state preparing -> state finished >Thread-4912::DEBUG::2012-12-02 23:47:38,951::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-4912::DEBUG::2012-12-02 23:47:38,951::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4912::DEBUG::2012-12-02 23:47:38,951::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-4912::DEBUG::2012-12-02 23:47:38,951::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-4912::DEBUG::2012-12-02 23:47:38,951::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-4912::DEBUG::2012-12-02 23:47:38,952::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-4912::DEBUG::2012-12-02 23:47:38,952::task::957::TaskManager.Task::(_decref) Task=`aad7bf42-0968-4ef2-9c0f-63f6178e05e4`::ref 0 aborting False >Thread-5037::DEBUG::2012-12-02 23:47:44,165::task::568::TaskManager.Task::(_updateState) Task=`d2141aa4-52c1-4fd5-a979-53fae7648b46`::moving from state init -> state preparing >Thread-5037::INFO::2012-12-02 23:47:44,165::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5037::INFO::2012-12-02 23:47:44,165::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00774192810059', 'lastCheck': 1354484855.8828509, 'code': 0, 'valid': True}} >Thread-5037::DEBUG::2012-12-02 23:47:44,165::task::1151::TaskManager.Task::(prepare) Task=`d2141aa4-52c1-4fd5-a979-53fae7648b46`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00774192810059', 'lastCheck': 1354484855.8828509, 'code': 0, 'valid': True}} >Thread-5037::DEBUG::2012-12-02 23:47:44,165::task::568::TaskManager.Task::(_updateState) Task=`d2141aa4-52c1-4fd5-a979-53fae7648b46`::moving from state preparing -> state finished >Thread-5037::DEBUG::2012-12-02 23:47:44,166::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5037::DEBUG::2012-12-02 23:47:44,166::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5037::DEBUG::2012-12-02 23:47:44,166::task::957::TaskManager.Task::(_decref) Task=`d2141aa4-52c1-4fd5-a979-53fae7648b46`::ref 0 aborting False >Thread-5041::DEBUG::2012-12-02 23:47:49,196::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5041::DEBUG::2012-12-02 23:47:49,196::task::568::TaskManager.Task::(_updateState) Task=`8e5d12b3-97ba-4c05-961e-d4d424dc3075`::moving from state init -> state preparing >Thread-5041::INFO::2012-12-02 23:47:49,196::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5041::INFO::2012-12-02 23:47:49,197::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5041::DEBUG::2012-12-02 23:47:49,197::task::1151::TaskManager.Task::(prepare) Task=`8e5d12b3-97ba-4c05-961e-d4d424dc3075`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5041::DEBUG::2012-12-02 23:47:49,197::task::568::TaskManager.Task::(_updateState) Task=`8e5d12b3-97ba-4c05-961e-d4d424dc3075`::moving from state preparing -> state finished >Thread-5041::DEBUG::2012-12-02 23:47:49,197::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5041::DEBUG::2012-12-02 23:47:49,197::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5041::DEBUG::2012-12-02 23:47:49,197::task::957::TaskManager.Task::(_decref) Task=`8e5d12b3-97ba-4c05-961e-d4d424dc3075`::ref 0 aborting False >Thread-5042::DEBUG::2012-12-02 23:47:49,357::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5042::DEBUG::2012-12-02 23:47:49,358::task::568::TaskManager.Task::(_updateState) Task=`0f78cc0a-e190-440e-89db-29c358f61222`::moving from state init -> state preparing >Thread-5042::INFO::2012-12-02 23:47:49,358::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5042::DEBUG::2012-12-02 23:47:49,358::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`79144259-21e0-45d3-844a-4e4f7eedce06`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5042::DEBUG::2012-12-02 23:47:49,358::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5042::DEBUG::2012-12-02 23:47:49,359::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5042::DEBUG::2012-12-02 23:47:49,359::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`79144259-21e0-45d3-844a-4e4f7eedce06`::Granted request >Thread-5042::DEBUG::2012-12-02 23:47:49,359::task::794::TaskManager.Task::(resourceAcquired) Task=`0f78cc0a-e190-440e-89db-29c358f61222`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5042::DEBUG::2012-12-02 23:47:49,359::task::957::TaskManager.Task::(_decref) Task=`0f78cc0a-e190-440e-89db-29c358f61222`::ref 1 aborting False >Thread-5042::DEBUG::2012-12-02 23:47:49,365::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5042::DEBUG::2012-12-02 23:47:49,366::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5042::DEBUG::2012-12-02 23:47:49,570::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5042::WARNING::2012-12-02 23:47:49,572::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5042::DEBUG::2012-12-02 23:47:49,572::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5042::WARNING::2012-12-02 23:47:49,617::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5042::DEBUG::2012-12-02 23:47:49,618::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5042::DEBUG::2012-12-02 23:47:49,618::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5042::DEBUG::2012-12-02 23:47:49,827::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5042::WARNING::2012-12-02 23:47:49,828::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5042::DEBUG::2012-12-02 23:47:49,828::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5042::WARNING::2012-12-02 23:47:49,857::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5042::INFO::2012-12-02 23:47:49,858::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5042::DEBUG::2012-12-02 23:47:49,858::task::1151::TaskManager.Task::(prepare) Task=`0f78cc0a-e190-440e-89db-29c358f61222`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5042::DEBUG::2012-12-02 23:47:49,858::task::568::TaskManager.Task::(_updateState) Task=`0f78cc0a-e190-440e-89db-29c358f61222`::moving from state preparing -> state finished >Thread-5042::DEBUG::2012-12-02 23:47:49,858::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5042::DEBUG::2012-12-02 23:47:49,858::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5042::DEBUG::2012-12-02 23:47:49,859::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5042::DEBUG::2012-12-02 23:47:49,859::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5042::DEBUG::2012-12-02 23:47:49,859::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5042::DEBUG::2012-12-02 23:47:49,859::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5042::DEBUG::2012-12-02 23:47:49,859::task::957::TaskManager.Task::(_decref) Task=`0f78cc0a-e190-440e-89db-29c358f61222`::ref 0 aborting False >Thread-5167::DEBUG::2012-12-02 23:47:55,143::task::568::TaskManager.Task::(_updateState) Task=`d478e1ed-05e3-4b64-b71b-7125ad7978af`::moving from state init -> state preparing >Thread-5167::INFO::2012-12-02 23:47:55,143::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5167::INFO::2012-12-02 23:47:55,143::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00859189033508', 'lastCheck': 1354484865.9024069, 'code': 0, 'valid': True}} >Thread-5167::DEBUG::2012-12-02 23:47:55,143::task::1151::TaskManager.Task::(prepare) Task=`d478e1ed-05e3-4b64-b71b-7125ad7978af`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00859189033508', 'lastCheck': 1354484865.9024069, 'code': 0, 'valid': True}} >Thread-5167::DEBUG::2012-12-02 23:47:55,143::task::568::TaskManager.Task::(_updateState) Task=`d478e1ed-05e3-4b64-b71b-7125ad7978af`::moving from state preparing -> state finished >Thread-5167::DEBUG::2012-12-02 23:47:55,143::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5167::DEBUG::2012-12-02 23:47:55,144::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5167::DEBUG::2012-12-02 23:47:55,144::task::957::TaskManager.Task::(_decref) Task=`d478e1ed-05e3-4b64-b71b-7125ad7978af`::ref 0 aborting False >Thread-5171::DEBUG::2012-12-02 23:48:00,093::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5171::DEBUG::2012-12-02 23:48:00,093::task::568::TaskManager.Task::(_updateState) Task=`c0d2d98b-795d-40f5-a022-82aacb2ce93a`::moving from state init -> state preparing >Thread-5171::INFO::2012-12-02 23:48:00,093::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5171::INFO::2012-12-02 23:48:00,094::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5171::DEBUG::2012-12-02 23:48:00,094::task::1151::TaskManager.Task::(prepare) Task=`c0d2d98b-795d-40f5-a022-82aacb2ce93a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5171::DEBUG::2012-12-02 23:48:00,094::task::568::TaskManager.Task::(_updateState) Task=`c0d2d98b-795d-40f5-a022-82aacb2ce93a`::moving from state preparing -> state finished >Thread-5171::DEBUG::2012-12-02 23:48:00,094::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5171::DEBUG::2012-12-02 23:48:00,094::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5171::DEBUG::2012-12-02 23:48:00,094::task::957::TaskManager.Task::(_decref) Task=`c0d2d98b-795d-40f5-a022-82aacb2ce93a`::ref 0 aborting False >Thread-5172::DEBUG::2012-12-02 23:48:00,239::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5172::DEBUG::2012-12-02 23:48:00,240::task::568::TaskManager.Task::(_updateState) Task=`382d09f9-91df-46c1-9ea9-c04d7cea9147`::moving from state init -> state preparing >Thread-5172::INFO::2012-12-02 23:48:00,240::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5172::DEBUG::2012-12-02 23:48:00,240::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ed94e745-da26-4d64-ba6d-01560841756a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5172::DEBUG::2012-12-02 23:48:00,240::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5172::DEBUG::2012-12-02 23:48:00,241::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5172::DEBUG::2012-12-02 23:48:00,241::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ed94e745-da26-4d64-ba6d-01560841756a`::Granted request >Thread-5172::DEBUG::2012-12-02 23:48:00,241::task::794::TaskManager.Task::(resourceAcquired) Task=`382d09f9-91df-46c1-9ea9-c04d7cea9147`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5172::DEBUG::2012-12-02 23:48:00,241::task::957::TaskManager.Task::(_decref) Task=`382d09f9-91df-46c1-9ea9-c04d7cea9147`::ref 1 aborting False >Thread-5172::DEBUG::2012-12-02 23:48:00,247::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5172::DEBUG::2012-12-02 23:48:00,247::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5172::DEBUG::2012-12-02 23:48:00,455::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5172::WARNING::2012-12-02 23:48:00,456::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5172::DEBUG::2012-12-02 23:48:00,456::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5172::WARNING::2012-12-02 23:48:00,487::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5172::DEBUG::2012-12-02 23:48:00,487::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5172::DEBUG::2012-12-02 23:48:00,488::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5172::DEBUG::2012-12-02 23:48:00,696::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5172::WARNING::2012-12-02 23:48:00,698::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5172::DEBUG::2012-12-02 23:48:00,698::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5172::WARNING::2012-12-02 23:48:00,727::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5172::INFO::2012-12-02 23:48:00,728::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5172::DEBUG::2012-12-02 23:48:00,728::task::1151::TaskManager.Task::(prepare) Task=`382d09f9-91df-46c1-9ea9-c04d7cea9147`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5172::DEBUG::2012-12-02 23:48:00,728::task::568::TaskManager.Task::(_updateState) Task=`382d09f9-91df-46c1-9ea9-c04d7cea9147`::moving from state preparing -> state finished >Thread-5172::DEBUG::2012-12-02 23:48:00,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5172::DEBUG::2012-12-02 23:48:00,728::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5172::DEBUG::2012-12-02 23:48:00,728::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5172::DEBUG::2012-12-02 23:48:00,729::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5172::DEBUG::2012-12-02 23:48:00,729::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5172::DEBUG::2012-12-02 23:48:00,729::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5172::DEBUG::2012-12-02 23:48:00,729::task::957::TaskManager.Task::(_decref) Task=`382d09f9-91df-46c1-9ea9-c04d7cea9147`::ref 0 aborting False >Thread-5297::DEBUG::2012-12-02 23:48:06,066::task::568::TaskManager.Task::(_updateState) Task=`ec836eb9-3a6e-4af8-bac1-259d335f3666`::moving from state init -> state preparing >Thread-5297::INFO::2012-12-02 23:48:06,067::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5297::INFO::2012-12-02 23:48:06,067::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00866103172302', 'lastCheck': 1354484885.940244, 'code': 0, 'valid': True}} >Thread-5297::DEBUG::2012-12-02 23:48:06,067::task::1151::TaskManager.Task::(prepare) Task=`ec836eb9-3a6e-4af8-bac1-259d335f3666`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00866103172302', 'lastCheck': 1354484885.940244, 'code': 0, 'valid': True}} >Thread-5297::DEBUG::2012-12-02 23:48:06,067::task::568::TaskManager.Task::(_updateState) Task=`ec836eb9-3a6e-4af8-bac1-259d335f3666`::moving from state preparing -> state finished >Thread-5297::DEBUG::2012-12-02 23:48:06,067::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5297::DEBUG::2012-12-02 23:48:06,067::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5297::DEBUG::2012-12-02 23:48:06,067::task::957::TaskManager.Task::(_decref) Task=`ec836eb9-3a6e-4af8-bac1-259d335f3666`::ref 0 aborting False >Thread-5301::DEBUG::2012-12-02 23:48:10,955::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5301::DEBUG::2012-12-02 23:48:10,955::task::568::TaskManager.Task::(_updateState) Task=`b132548a-3914-4fdb-ac0c-b54842fb0a2e`::moving from state init -> state preparing >Thread-5301::INFO::2012-12-02 23:48:10,956::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5301::INFO::2012-12-02 23:48:10,956::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5301::DEBUG::2012-12-02 23:48:10,956::task::1151::TaskManager.Task::(prepare) Task=`b132548a-3914-4fdb-ac0c-b54842fb0a2e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5301::DEBUG::2012-12-02 23:48:10,956::task::568::TaskManager.Task::(_updateState) Task=`b132548a-3914-4fdb-ac0c-b54842fb0a2e`::moving from state preparing -> state finished >Thread-5301::DEBUG::2012-12-02 23:48:10,956::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5301::DEBUG::2012-12-02 23:48:10,956::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5301::DEBUG::2012-12-02 23:48:10,956::task::957::TaskManager.Task::(_decref) Task=`b132548a-3914-4fdb-ac0c-b54842fb0a2e`::ref 0 aborting False >Thread-5302::DEBUG::2012-12-02 23:48:11,099::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5302::DEBUG::2012-12-02 23:48:11,099::task::568::TaskManager.Task::(_updateState) Task=`b99281d8-a5fc-4e9d-9320-ae743ca7b944`::moving from state init -> state preparing >Thread-5302::INFO::2012-12-02 23:48:11,100::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5302::DEBUG::2012-12-02 23:48:11,100::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ae4f1f01-4689-4b3f-ae88-a54d5295190d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5302::DEBUG::2012-12-02 23:48:11,100::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5302::DEBUG::2012-12-02 23:48:11,100::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5302::DEBUG::2012-12-02 23:48:11,100::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ae4f1f01-4689-4b3f-ae88-a54d5295190d`::Granted request >Thread-5302::DEBUG::2012-12-02 23:48:11,101::task::794::TaskManager.Task::(resourceAcquired) Task=`b99281d8-a5fc-4e9d-9320-ae743ca7b944`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5302::DEBUG::2012-12-02 23:48:11,101::task::957::TaskManager.Task::(_decref) Task=`b99281d8-a5fc-4e9d-9320-ae743ca7b944`::ref 1 aborting False >Thread-5302::DEBUG::2012-12-02 23:48:11,107::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5302::DEBUG::2012-12-02 23:48:11,107::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5302::DEBUG::2012-12-02 23:48:11,313::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5302::WARNING::2012-12-02 23:48:11,314::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5302::DEBUG::2012-12-02 23:48:11,314::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5302::WARNING::2012-12-02 23:48:11,344::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5302::DEBUG::2012-12-02 23:48:11,344::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5302::DEBUG::2012-12-02 23:48:11,344::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5302::DEBUG::2012-12-02 23:48:11,552::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5302::WARNING::2012-12-02 23:48:11,554::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5302::DEBUG::2012-12-02 23:48:11,554::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5302::WARNING::2012-12-02 23:48:11,583::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5302::INFO::2012-12-02 23:48:11,584::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5302::DEBUG::2012-12-02 23:48:11,584::task::1151::TaskManager.Task::(prepare) Task=`b99281d8-a5fc-4e9d-9320-ae743ca7b944`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5302::DEBUG::2012-12-02 23:48:11,584::task::568::TaskManager.Task::(_updateState) Task=`b99281d8-a5fc-4e9d-9320-ae743ca7b944`::moving from state preparing -> state finished >Thread-5302::DEBUG::2012-12-02 23:48:11,584::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5302::DEBUG::2012-12-02 23:48:11,584::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5302::DEBUG::2012-12-02 23:48:11,584::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5302::DEBUG::2012-12-02 23:48:11,585::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5302::DEBUG::2012-12-02 23:48:11,585::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5302::DEBUG::2012-12-02 23:48:11,585::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5302::DEBUG::2012-12-02 23:48:11,585::task::957::TaskManager.Task::(_decref) Task=`b99281d8-a5fc-4e9d-9320-ae743ca7b944`::ref 0 aborting False >Thread-5427::DEBUG::2012-12-02 23:48:17,010::task::568::TaskManager.Task::(_updateState) Task=`18f166f1-3321-4019-bac3-5b25fbaaf7c7`::moving from state init -> state preparing >Thread-5427::INFO::2012-12-02 23:48:17,010::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5427::INFO::2012-12-02 23:48:17,010::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0151488780975', 'lastCheck': 1354484895.970413, 'code': 0, 'valid': True}} >Thread-5427::DEBUG::2012-12-02 23:48:17,011::task::1151::TaskManager.Task::(prepare) Task=`18f166f1-3321-4019-bac3-5b25fbaaf7c7`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0151488780975', 'lastCheck': 1354484895.970413, 'code': 0, 'valid': True}} >Thread-5427::DEBUG::2012-12-02 23:48:17,011::task::568::TaskManager.Task::(_updateState) Task=`18f166f1-3321-4019-bac3-5b25fbaaf7c7`::moving from state preparing -> state finished >Thread-5427::DEBUG::2012-12-02 23:48:17,011::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5427::DEBUG::2012-12-02 23:48:17,011::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5427::DEBUG::2012-12-02 23:48:17,011::task::957::TaskManager.Task::(_decref) Task=`18f166f1-3321-4019-bac3-5b25fbaaf7c7`::ref 0 aborting False >Thread-5431::DEBUG::2012-12-02 23:48:21,830::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5431::DEBUG::2012-12-02 23:48:21,831::task::568::TaskManager.Task::(_updateState) Task=`d8bf9826-6cd5-4c07-a797-34d69e8d6a44`::moving from state init -> state preparing >Thread-5431::INFO::2012-12-02 23:48:21,831::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5431::INFO::2012-12-02 23:48:21,831::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5431::DEBUG::2012-12-02 23:48:21,831::task::1151::TaskManager.Task::(prepare) Task=`d8bf9826-6cd5-4c07-a797-34d69e8d6a44`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5431::DEBUG::2012-12-02 23:48:21,831::task::568::TaskManager.Task::(_updateState) Task=`d8bf9826-6cd5-4c07-a797-34d69e8d6a44`::moving from state preparing -> state finished >Thread-5431::DEBUG::2012-12-02 23:48:21,831::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5431::DEBUG::2012-12-02 23:48:21,832::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5431::DEBUG::2012-12-02 23:48:21,832::task::957::TaskManager.Task::(_decref) Task=`d8bf9826-6cd5-4c07-a797-34d69e8d6a44`::ref 0 aborting False >Thread-5432::DEBUG::2012-12-02 23:48:21,991::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5432::DEBUG::2012-12-02 23:48:21,991::task::568::TaskManager.Task::(_updateState) Task=`beb22c6b-75fc-48af-bcfc-dc19c9cc53ad`::moving from state init -> state preparing >Thread-5432::INFO::2012-12-02 23:48:21,992::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5432::DEBUG::2012-12-02 23:48:21,992::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b0e57cde-f2d9-4447-9e3a-61a42f0b8195`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5432::DEBUG::2012-12-02 23:48:21,992::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5432::DEBUG::2012-12-02 23:48:21,992::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5432::DEBUG::2012-12-02 23:48:21,992::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b0e57cde-f2d9-4447-9e3a-61a42f0b8195`::Granted request >Thread-5432::DEBUG::2012-12-02 23:48:21,993::task::794::TaskManager.Task::(resourceAcquired) Task=`beb22c6b-75fc-48af-bcfc-dc19c9cc53ad`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5432::DEBUG::2012-12-02 23:48:21,993::task::957::TaskManager.Task::(_decref) Task=`beb22c6b-75fc-48af-bcfc-dc19c9cc53ad`::ref 1 aborting False >Thread-5432::DEBUG::2012-12-02 23:48:21,998::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5432::DEBUG::2012-12-02 23:48:21,998::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5432::DEBUG::2012-12-02 23:48:22,207::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5432::WARNING::2012-12-02 23:48:22,208::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5432::DEBUG::2012-12-02 23:48:22,208::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5432::WARNING::2012-12-02 23:48:22,238::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5432::DEBUG::2012-12-02 23:48:22,238::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5432::DEBUG::2012-12-02 23:48:22,239::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5432::DEBUG::2012-12-02 23:48:22,447::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5432::WARNING::2012-12-02 23:48:22,448::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5432::DEBUG::2012-12-02 23:48:22,448::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5432::WARNING::2012-12-02 23:48:22,476::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5432::INFO::2012-12-02 23:48:22,477::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5432::DEBUG::2012-12-02 23:48:22,477::task::1151::TaskManager.Task::(prepare) Task=`beb22c6b-75fc-48af-bcfc-dc19c9cc53ad`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5432::DEBUG::2012-12-02 23:48:22,477::task::568::TaskManager.Task::(_updateState) Task=`beb22c6b-75fc-48af-bcfc-dc19c9cc53ad`::moving from state preparing -> state finished >Thread-5432::DEBUG::2012-12-02 23:48:22,477::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5432::DEBUG::2012-12-02 23:48:22,477::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5432::DEBUG::2012-12-02 23:48:22,478::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5432::DEBUG::2012-12-02 23:48:22,478::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5432::DEBUG::2012-12-02 23:48:22,478::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5432::DEBUG::2012-12-02 23:48:22,478::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5432::DEBUG::2012-12-02 23:48:22,478::task::957::TaskManager.Task::(_decref) Task=`beb22c6b-75fc-48af-bcfc-dc19c9cc53ad`::ref 0 aborting False >Thread-5557::DEBUG::2012-12-02 23:48:27,965::task::568::TaskManager.Task::(_updateState) Task=`358b4229-06ce-4751-a9af-1a6fe8aeed56`::moving from state init -> state preparing >Thread-5557::INFO::2012-12-02 23:48:27,965::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5557::INFO::2012-12-02 23:48:27,966::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00627899169922', 'lastCheck': 1354484905.9898541, 'code': 0, 'valid': True}} >Thread-5557::DEBUG::2012-12-02 23:48:27,966::task::1151::TaskManager.Task::(prepare) Task=`358b4229-06ce-4751-a9af-1a6fe8aeed56`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00627899169922', 'lastCheck': 1354484905.9898541, 'code': 0, 'valid': True}} >Thread-5557::DEBUG::2012-12-02 23:48:27,966::task::568::TaskManager.Task::(_updateState) Task=`358b4229-06ce-4751-a9af-1a6fe8aeed56`::moving from state preparing -> state finished >Thread-5557::DEBUG::2012-12-02 23:48:27,966::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5557::DEBUG::2012-12-02 23:48:27,966::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5557::DEBUG::2012-12-02 23:48:27,966::task::957::TaskManager.Task::(_decref) Task=`358b4229-06ce-4751-a9af-1a6fe8aeed56`::ref 0 aborting False >Thread-5561::DEBUG::2012-12-02 23:48:32,708::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5561::DEBUG::2012-12-02 23:48:32,708::task::568::TaskManager.Task::(_updateState) Task=`575fc03b-1770-47fe-afb4-af17e31862b5`::moving from state init -> state preparing >Thread-5561::INFO::2012-12-02 23:48:32,708::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5561::INFO::2012-12-02 23:48:32,709::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5561::DEBUG::2012-12-02 23:48:32,709::task::1151::TaskManager.Task::(prepare) Task=`575fc03b-1770-47fe-afb4-af17e31862b5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5561::DEBUG::2012-12-02 23:48:32,709::task::568::TaskManager.Task::(_updateState) Task=`575fc03b-1770-47fe-afb4-af17e31862b5`::moving from state preparing -> state finished >Thread-5561::DEBUG::2012-12-02 23:48:32,709::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5561::DEBUG::2012-12-02 23:48:32,709::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5561::DEBUG::2012-12-02 23:48:32,709::task::957::TaskManager.Task::(_decref) Task=`575fc03b-1770-47fe-afb4-af17e31862b5`::ref 0 aborting False >Thread-5562::DEBUG::2012-12-02 23:48:32,849::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5562::DEBUG::2012-12-02 23:48:32,849::task::568::TaskManager.Task::(_updateState) Task=`6df56a19-9f1d-40a3-babc-d4035c594c1d`::moving from state init -> state preparing >Thread-5562::INFO::2012-12-02 23:48:32,849::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5562::DEBUG::2012-12-02 23:48:32,850::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6700da9d-1bd5-4636-a5f8-ebd4d81d9c82`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5562::DEBUG::2012-12-02 23:48:32,850::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5562::DEBUG::2012-12-02 23:48:32,850::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5562::DEBUG::2012-12-02 23:48:32,850::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6700da9d-1bd5-4636-a5f8-ebd4d81d9c82`::Granted request >Thread-5562::DEBUG::2012-12-02 23:48:32,850::task::794::TaskManager.Task::(resourceAcquired) Task=`6df56a19-9f1d-40a3-babc-d4035c594c1d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5562::DEBUG::2012-12-02 23:48:32,850::task::957::TaskManager.Task::(_decref) Task=`6df56a19-9f1d-40a3-babc-d4035c594c1d`::ref 1 aborting False >Thread-5562::DEBUG::2012-12-02 23:48:32,856::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5562::DEBUG::2012-12-02 23:48:32,857::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5562::DEBUG::2012-12-02 23:48:33,062::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5562::WARNING::2012-12-02 23:48:33,064::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5562::DEBUG::2012-12-02 23:48:33,064::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5562::WARNING::2012-12-02 23:48:33,095::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5562::DEBUG::2012-12-02 23:48:33,095::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5562::DEBUG::2012-12-02 23:48:33,095::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5562::DEBUG::2012-12-02 23:48:33,301::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5562::WARNING::2012-12-02 23:48:33,302::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5562::DEBUG::2012-12-02 23:48:33,303::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5562::WARNING::2012-12-02 23:48:33,332::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5562::INFO::2012-12-02 23:48:33,333::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5562::DEBUG::2012-12-02 23:48:33,333::task::1151::TaskManager.Task::(prepare) Task=`6df56a19-9f1d-40a3-babc-d4035c594c1d`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5562::DEBUG::2012-12-02 23:48:33,333::task::568::TaskManager.Task::(_updateState) Task=`6df56a19-9f1d-40a3-babc-d4035c594c1d`::moving from state preparing -> state finished >Thread-5562::DEBUG::2012-12-02 23:48:33,333::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5562::DEBUG::2012-12-02 23:48:33,333::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5562::DEBUG::2012-12-02 23:48:33,333::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5562::DEBUG::2012-12-02 23:48:33,334::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5562::DEBUG::2012-12-02 23:48:33,334::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5562::DEBUG::2012-12-02 23:48:33,334::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5562::DEBUG::2012-12-02 23:48:33,334::task::957::TaskManager.Task::(_decref) Task=`6df56a19-9f1d-40a3-babc-d4035c594c1d`::ref 0 aborting False >Thread-5687::DEBUG::2012-12-02 23:48:38,982::task::568::TaskManager.Task::(_updateState) Task=`58247cd5-a4be-47c9-b79f-5482f317fecb`::moving from state init -> state preparing >Thread-5687::INFO::2012-12-02 23:48:38,982::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5687::INFO::2012-12-02 23:48:38,983::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00743007659912', 'lastCheck': 1354484916.0088201, 'code': 0, 'valid': True}} >Thread-5687::DEBUG::2012-12-02 23:48:38,983::task::1151::TaskManager.Task::(prepare) Task=`58247cd5-a4be-47c9-b79f-5482f317fecb`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00743007659912', 'lastCheck': 1354484916.0088201, 'code': 0, 'valid': True}} >Thread-5687::DEBUG::2012-12-02 23:48:38,983::task::568::TaskManager.Task::(_updateState) Task=`58247cd5-a4be-47c9-b79f-5482f317fecb`::moving from state preparing -> state finished >Thread-5687::DEBUG::2012-12-02 23:48:38,983::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5687::DEBUG::2012-12-02 23:48:38,983::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5687::DEBUG::2012-12-02 23:48:38,983::task::957::TaskManager.Task::(_decref) Task=`58247cd5-a4be-47c9-b79f-5482f317fecb`::ref 0 aborting False >Thread-5691::DEBUG::2012-12-02 23:48:43,561::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5691::DEBUG::2012-12-02 23:48:43,562::task::568::TaskManager.Task::(_updateState) Task=`262b7352-9e84-4c26-b6aa-853c6dcf1429`::moving from state init -> state preparing >Thread-5691::INFO::2012-12-02 23:48:43,562::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5691::INFO::2012-12-02 23:48:43,562::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5691::DEBUG::2012-12-02 23:48:43,562::task::1151::TaskManager.Task::(prepare) Task=`262b7352-9e84-4c26-b6aa-853c6dcf1429`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5691::DEBUG::2012-12-02 23:48:43,562::task::568::TaskManager.Task::(_updateState) Task=`262b7352-9e84-4c26-b6aa-853c6dcf1429`::moving from state preparing -> state finished >Thread-5691::DEBUG::2012-12-02 23:48:43,562::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5691::DEBUG::2012-12-02 23:48:43,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5691::DEBUG::2012-12-02 23:48:43,563::task::957::TaskManager.Task::(_decref) Task=`262b7352-9e84-4c26-b6aa-853c6dcf1429`::ref 0 aborting False >Thread-5692::DEBUG::2012-12-02 23:48:43,701::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5692::DEBUG::2012-12-02 23:48:43,702::task::568::TaskManager.Task::(_updateState) Task=`5625b91c-0212-4527-a5c2-8687e73f6cfe`::moving from state init -> state preparing >Thread-5692::INFO::2012-12-02 23:48:43,702::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5692::DEBUG::2012-12-02 23:48:43,702::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a5818fcd-b270-410f-8e8b-fb5e639aee54`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5692::DEBUG::2012-12-02 23:48:43,702::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5692::DEBUG::2012-12-02 23:48:43,703::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5692::DEBUG::2012-12-02 23:48:43,703::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a5818fcd-b270-410f-8e8b-fb5e639aee54`::Granted request >Thread-5692::DEBUG::2012-12-02 23:48:43,703::task::794::TaskManager.Task::(resourceAcquired) Task=`5625b91c-0212-4527-a5c2-8687e73f6cfe`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5692::DEBUG::2012-12-02 23:48:43,703::task::957::TaskManager.Task::(_decref) Task=`5625b91c-0212-4527-a5c2-8687e73f6cfe`::ref 1 aborting False >Thread-5692::DEBUG::2012-12-02 23:48:43,708::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5692::DEBUG::2012-12-02 23:48:43,708::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5692::DEBUG::2012-12-02 23:48:43,912::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5692::WARNING::2012-12-02 23:48:43,913::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5692::DEBUG::2012-12-02 23:48:43,914::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5692::WARNING::2012-12-02 23:48:43,944::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5692::DEBUG::2012-12-02 23:48:43,944::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5692::DEBUG::2012-12-02 23:48:43,945::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5692::DEBUG::2012-12-02 23:48:44,151::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5692::WARNING::2012-12-02 23:48:44,153::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5692::DEBUG::2012-12-02 23:48:44,153::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5692::WARNING::2012-12-02 23:48:44,181::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5692::INFO::2012-12-02 23:48:44,181::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5692::DEBUG::2012-12-02 23:48:44,181::task::1151::TaskManager.Task::(prepare) Task=`5625b91c-0212-4527-a5c2-8687e73f6cfe`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5692::DEBUG::2012-12-02 23:48:44,182::task::568::TaskManager.Task::(_updateState) Task=`5625b91c-0212-4527-a5c2-8687e73f6cfe`::moving from state preparing -> state finished >Thread-5692::DEBUG::2012-12-02 23:48:44,182::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5692::DEBUG::2012-12-02 23:48:44,182::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5692::DEBUG::2012-12-02 23:48:44,182::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5692::DEBUG::2012-12-02 23:48:44,182::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5692::DEBUG::2012-12-02 23:48:44,182::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5692::DEBUG::2012-12-02 23:48:44,182::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5692::DEBUG::2012-12-02 23:48:44,183::task::957::TaskManager.Task::(_decref) Task=`5625b91c-0212-4527-a5c2-8687e73f6cfe`::ref 0 aborting False >Thread-5817::DEBUG::2012-12-02 23:48:49,971::task::568::TaskManager.Task::(_updateState) Task=`6a63b2f6-0522-4dad-96f6-6d9b38518995`::moving from state init -> state preparing >Thread-5817::INFO::2012-12-02 23:48:49,971::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5817::INFO::2012-12-02 23:48:49,971::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00941705703735', 'lastCheck': 1354484926.0293529, 'code': 0, 'valid': True}} >Thread-5817::DEBUG::2012-12-02 23:48:49,971::task::1151::TaskManager.Task::(prepare) Task=`6a63b2f6-0522-4dad-96f6-6d9b38518995`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00941705703735', 'lastCheck': 1354484926.0293529, 'code': 0, 'valid': True}} >Thread-5817::DEBUG::2012-12-02 23:48:49,971::task::568::TaskManager.Task::(_updateState) Task=`6a63b2f6-0522-4dad-96f6-6d9b38518995`::moving from state preparing -> state finished >Thread-5817::DEBUG::2012-12-02 23:48:49,971::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5817::DEBUG::2012-12-02 23:48:49,971::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5817::DEBUG::2012-12-02 23:48:49,972::task::957::TaskManager.Task::(_decref) Task=`6a63b2f6-0522-4dad-96f6-6d9b38518995`::ref 0 aborting False >Thread-5820::DEBUG::2012-12-02 23:48:54,397::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5820::DEBUG::2012-12-02 23:48:54,398::task::568::TaskManager.Task::(_updateState) Task=`50440333-f0cc-4924-a9d4-8a5dc0f09517`::moving from state init -> state preparing >Thread-5820::INFO::2012-12-02 23:48:54,398::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5820::INFO::2012-12-02 23:48:54,398::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5820::DEBUG::2012-12-02 23:48:54,398::task::1151::TaskManager.Task::(prepare) Task=`50440333-f0cc-4924-a9d4-8a5dc0f09517`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5820::DEBUG::2012-12-02 23:48:54,399::task::568::TaskManager.Task::(_updateState) Task=`50440333-f0cc-4924-a9d4-8a5dc0f09517`::moving from state preparing -> state finished >Thread-5820::DEBUG::2012-12-02 23:48:54,399::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5820::DEBUG::2012-12-02 23:48:54,399::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5820::DEBUG::2012-12-02 23:48:54,399::task::957::TaskManager.Task::(_decref) Task=`50440333-f0cc-4924-a9d4-8a5dc0f09517`::ref 0 aborting False >Thread-5822::DEBUG::2012-12-02 23:48:54,539::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5822::DEBUG::2012-12-02 23:48:54,539::task::568::TaskManager.Task::(_updateState) Task=`9b1a2f99-a125-4395-8fa2-aa70449da65d`::moving from state init -> state preparing >Thread-5822::INFO::2012-12-02 23:48:54,540::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5822::DEBUG::2012-12-02 23:48:54,540::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`18caa84d-3933-438e-aa14-0b87ef1a0503`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5822::DEBUG::2012-12-02 23:48:54,540::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5822::DEBUG::2012-12-02 23:48:54,540::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5822::DEBUG::2012-12-02 23:48:54,540::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`18caa84d-3933-438e-aa14-0b87ef1a0503`::Granted request >Thread-5822::DEBUG::2012-12-02 23:48:54,541::task::794::TaskManager.Task::(resourceAcquired) Task=`9b1a2f99-a125-4395-8fa2-aa70449da65d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5822::DEBUG::2012-12-02 23:48:54,541::task::957::TaskManager.Task::(_decref) Task=`9b1a2f99-a125-4395-8fa2-aa70449da65d`::ref 1 aborting False >Thread-5822::DEBUG::2012-12-02 23:48:54,547::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5822::DEBUG::2012-12-02 23:48:54,547::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5822::DEBUG::2012-12-02 23:48:54,750::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5822::WARNING::2012-12-02 23:48:54,751::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5822::DEBUG::2012-12-02 23:48:54,752::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5822::WARNING::2012-12-02 23:48:54,788::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5822::DEBUG::2012-12-02 23:48:54,788::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5822::DEBUG::2012-12-02 23:48:54,789::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5822::DEBUG::2012-12-02 23:48:54,995::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5822::WARNING::2012-12-02 23:48:54,997::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5822::DEBUG::2012-12-02 23:48:54,997::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5822::WARNING::2012-12-02 23:48:55,026::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5822::INFO::2012-12-02 23:48:55,027::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5822::DEBUG::2012-12-02 23:48:55,027::task::1151::TaskManager.Task::(prepare) Task=`9b1a2f99-a125-4395-8fa2-aa70449da65d`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5822::DEBUG::2012-12-02 23:48:55,027::task::568::TaskManager.Task::(_updateState) Task=`9b1a2f99-a125-4395-8fa2-aa70449da65d`::moving from state preparing -> state finished >Thread-5822::DEBUG::2012-12-02 23:48:55,027::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5822::DEBUG::2012-12-02 23:48:55,027::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5822::DEBUG::2012-12-02 23:48:55,027::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5822::DEBUG::2012-12-02 23:48:55,028::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5822::DEBUG::2012-12-02 23:48:55,028::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5822::DEBUG::2012-12-02 23:48:55,028::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5822::DEBUG::2012-12-02 23:48:55,028::task::957::TaskManager.Task::(_decref) Task=`9b1a2f99-a125-4395-8fa2-aa70449da65d`::ref 0 aborting False >Thread-5947::DEBUG::2012-12-02 23:49:00,955::task::568::TaskManager.Task::(_updateState) Task=`0b5aa336-4037-4dce-a79c-2b5ca2cd4688`::moving from state init -> state preparing >Thread-5947::INFO::2012-12-02 23:49:00,955::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-5947::INFO::2012-12-02 23:49:00,955::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00667881965637', 'lastCheck': 1354484936.047713, 'code': 0, 'valid': True}} >Thread-5947::DEBUG::2012-12-02 23:49:00,956::task::1151::TaskManager.Task::(prepare) Task=`0b5aa336-4037-4dce-a79c-2b5ca2cd4688`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00667881965637', 'lastCheck': 1354484936.047713, 'code': 0, 'valid': True}} >Thread-5947::DEBUG::2012-12-02 23:49:00,956::task::568::TaskManager.Task::(_updateState) Task=`0b5aa336-4037-4dce-a79c-2b5ca2cd4688`::moving from state preparing -> state finished >Thread-5947::DEBUG::2012-12-02 23:49:00,956::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5947::DEBUG::2012-12-02 23:49:00,956::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5947::DEBUG::2012-12-02 23:49:00,956::task::957::TaskManager.Task::(_decref) Task=`0b5aa336-4037-4dce-a79c-2b5ca2cd4688`::ref 0 aborting False >Thread-5950::DEBUG::2012-12-02 23:49:05,286::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5950::DEBUG::2012-12-02 23:49:05,286::task::568::TaskManager.Task::(_updateState) Task=`29017b78-2a0d-47ab-9501-b516ec3b9c12`::moving from state init -> state preparing >Thread-5950::INFO::2012-12-02 23:49:05,286::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5950::INFO::2012-12-02 23:49:05,286::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5950::DEBUG::2012-12-02 23:49:05,286::task::1151::TaskManager.Task::(prepare) Task=`29017b78-2a0d-47ab-9501-b516ec3b9c12`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-5950::DEBUG::2012-12-02 23:49:05,287::task::568::TaskManager.Task::(_updateState) Task=`29017b78-2a0d-47ab-9501-b516ec3b9c12`::moving from state preparing -> state finished >Thread-5950::DEBUG::2012-12-02 23:49:05,287::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-5950::DEBUG::2012-12-02 23:49:05,287::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5950::DEBUG::2012-12-02 23:49:05,287::task::957::TaskManager.Task::(_decref) Task=`29017b78-2a0d-47ab-9501-b516ec3b9c12`::ref 0 aborting False >Thread-5951::DEBUG::2012-12-02 23:49:05,422::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-5951::DEBUG::2012-12-02 23:49:05,423::task::568::TaskManager.Task::(_updateState) Task=`1fcd75b5-d9d8-4165-acc7-5267fb385e40`::moving from state init -> state preparing >Thread-5951::INFO::2012-12-02 23:49:05,423::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-5951::DEBUG::2012-12-02 23:49:05,423::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`305df943-e9e7-4c2f-80bf-c874c5e3b382`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-5951::DEBUG::2012-12-02 23:49:05,423::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-5951::DEBUG::2012-12-02 23:49:05,424::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-5951::DEBUG::2012-12-02 23:49:05,424::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`305df943-e9e7-4c2f-80bf-c874c5e3b382`::Granted request >Thread-5951::DEBUG::2012-12-02 23:49:05,424::task::794::TaskManager.Task::(resourceAcquired) Task=`1fcd75b5-d9d8-4165-acc7-5267fb385e40`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-5951::DEBUG::2012-12-02 23:49:05,424::task::957::TaskManager.Task::(_decref) Task=`1fcd75b5-d9d8-4165-acc7-5267fb385e40`::ref 1 aborting False >Thread-5951::DEBUG::2012-12-02 23:49:05,429::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5951::DEBUG::2012-12-02 23:49:05,429::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-5951::DEBUG::2012-12-02 23:49:05,633::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-5951::WARNING::2012-12-02 23:49:05,634::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-5951::DEBUG::2012-12-02 23:49:05,634::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5951::WARNING::2012-12-02 23:49:05,663::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-5951::DEBUG::2012-12-02 23:49:05,664::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-5951::DEBUG::2012-12-02 23:49:05,664::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-5951::DEBUG::2012-12-02 23:49:05,868::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-5951::WARNING::2012-12-02 23:49:05,869::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-5951::DEBUG::2012-12-02 23:49:05,869::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5951::WARNING::2012-12-02 23:49:05,897::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-5951::INFO::2012-12-02 23:49:05,897::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5951::DEBUG::2012-12-02 23:49:05,897::task::1151::TaskManager.Task::(prepare) Task=`1fcd75b5-d9d8-4165-acc7-5267fb385e40`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-5951::DEBUG::2012-12-02 23:49:05,898::task::568::TaskManager.Task::(_updateState) Task=`1fcd75b5-d9d8-4165-acc7-5267fb385e40`::moving from state preparing -> state finished >Thread-5951::DEBUG::2012-12-02 23:49:05,898::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-5951::DEBUG::2012-12-02 23:49:05,898::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-5951::DEBUG::2012-12-02 23:49:05,898::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-5951::DEBUG::2012-12-02 23:49:05,898::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-5951::DEBUG::2012-12-02 23:49:05,898::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-5951::DEBUG::2012-12-02 23:49:05,898::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-5951::DEBUG::2012-12-02 23:49:05,899::task::957::TaskManager.Task::(_decref) Task=`1fcd75b5-d9d8-4165-acc7-5267fb385e40`::ref 0 aborting False >Thread-6076::DEBUG::2012-12-02 23:49:09,003::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6076::DEBUG::2012-12-02 23:49:09,003::task::568::TaskManager.Task::(_updateState) Task=`c70c0533-62e0-49cf-bdf1-334c19f204ff`::moving from state init -> state preparing >Thread-6076::INFO::2012-12-02 23:49:09,003::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-6076::INFO::2012-12-02 23:49:09,003::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-6076::DEBUG::2012-12-02 23:49:09,003::task::1151::TaskManager.Task::(prepare) Task=`c70c0533-62e0-49cf-bdf1-334c19f204ff`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-6076::DEBUG::2012-12-02 23:49:09,004::task::568::TaskManager.Task::(_updateState) Task=`c70c0533-62e0-49cf-bdf1-334c19f204ff`::moving from state preparing -> state finished >Thread-6076::DEBUG::2012-12-02 23:49:09,004::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6076::DEBUG::2012-12-02 23:49:09,004::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6076::DEBUG::2012-12-02 23:49:09,004::task::957::TaskManager.Task::(_decref) Task=`c70c0533-62e0-49cf-bdf1-334c19f204ff`::ref 0 aborting False >Thread-6077::DEBUG::2012-12-02 23:49:09,156::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6077::DEBUG::2012-12-02 23:49:09,156::task::568::TaskManager.Task::(_updateState) Task=`89cf7bc1-983f-4023-b7f4-de7382510895`::moving from state init -> state preparing >Thread-6077::INFO::2012-12-02 23:49:09,156::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-6077::DEBUG::2012-12-02 23:49:09,163::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348' (cwd None) >Thread-6077::DEBUG::2012-12-02 23:49:09,292::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-6077::DEBUG::2012-12-02 23:49:09,292::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-6077::DEBUG::2012-12-02 23:49:09,292::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-6077::DEBUG::2012-12-02 23:49:09,292::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-6077::DEBUG::2012-12-02 23:49:09,292::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-6077::DEBUG::2012-12-02 23:49:09,293::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-6077::INFO::2012-12-02 23:49:09,293::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-6077::DEBUG::2012-12-02 23:49:09,293::task::1151::TaskManager.Task::(prepare) Task=`89cf7bc1-983f-4023-b7f4-de7382510895`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-6077::DEBUG::2012-12-02 23:49:09,293::task::568::TaskManager.Task::(_updateState) Task=`89cf7bc1-983f-4023-b7f4-de7382510895`::moving from state preparing -> state finished >Thread-6077::DEBUG::2012-12-02 23:49:09,293::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6077::DEBUG::2012-12-02 23:49:09,293::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6077::DEBUG::2012-12-02 23:49:09,294::task::957::TaskManager.Task::(_decref) Task=`89cf7bc1-983f-4023-b7f4-de7382510895`::ref 0 aborting False >Thread-6078::DEBUG::2012-12-02 23:49:09,444::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6078::DEBUG::2012-12-02 23:49:09,444::task::568::TaskManager.Task::(_updateState) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::moving from state init -> state preparing >Thread-6078::INFO::2012-12-02 23:49:09,444::logUtils::37::dispatcher::(wrapper) Run and protect: attachStorageDomain(sdUUID='a00efcc1-5685-4dbe-ae70-440e9f403b76', spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6078::DEBUG::2012-12-02 23:49:09,445::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8b56b582-3947-4de8-87d1-20c7eb46c956`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6078::DEBUG::2012-12-02 23:49:09,445::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-6078::DEBUG::2012-12-02 23:49:09,445::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-6078::DEBUG::2012-12-02 23:49:09,445::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8b56b582-3947-4de8-87d1-20c7eb46c956`::Granted request >Thread-6078::DEBUG::2012-12-02 23:49:09,445::task::794::TaskManager.Task::(resourceAcquired) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-6078::DEBUG::2012-12-02 23:49:09,445::task::957::TaskManager.Task::(_decref) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::ref 1 aborting False >Thread-6078::DEBUG::2012-12-02 23:49:09,446::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76`ReqID=`ebfeb9f4-df9a-4dc6-9b2a-6242bcbee52f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6078::DEBUG::2012-12-02 23:49:09,446::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' for lock type 'exclusive' >Thread-6078::DEBUG::2012-12-02 23:49:09,446::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' is free. Now locking as 'exclusive' (1 active user) >Thread-6078::DEBUG::2012-12-02 23:49:09,446::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76`ReqID=`ebfeb9f4-df9a-4dc6-9b2a-6242bcbee52f`::Granted request >Thread-6078::DEBUG::2012-12-02 23:49:09,446::task::794::TaskManager.Task::(resourceAcquired) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::_resourcesAcquired: Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76 (exclusive) >Thread-6078::DEBUG::2012-12-02 23:49:09,447::task::957::TaskManager.Task::(_decref) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::ref 1 aborting False >Thread-6078::INFO::2012-12-02 23:49:09,447::sp::912::Storage.StoragePool::(attachSD) sdUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76 spUUID=479cf195-bb99-45c5-b400-433d810eaa48 >Thread-6078::DEBUG::2012-12-02 23:49:09,453::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-6078::DEBUG::2012-12-02 23:49:09,453::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-6078::DEBUG::2012-12-02 23:49:09,453::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-6078::DEBUG::2012-12-02 23:49:09,453::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-6078::DEBUG::2012-12-02 23:49:09,454::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-6078::DEBUG::2012-12-02 23:49:09,480::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-6078::DEBUG::2012-12-02 23:49:09,480::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-6078::DEBUG::2012-12-02 23:49:10,114::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-6078::DEBUG::2012-12-02 23:49:10,186::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-6078::DEBUG::2012-12-02 23:49:10,187::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,187::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,187::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,187::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,187::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,187::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,188::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-6078::DEBUG::2012-12-02 23:49:10,188::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,189::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-6078::DEBUG::2012-12-02 23:49:10,393::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-6078::WARNING::2012-12-02 23:49:10,394::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-6078::DEBUG::2012-12-02 23:49:10,395::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:10,431::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-6078::DEBUG::2012-12-02 23:49:10,431::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-6078::DEBUG::2012-12-02 23:49:10,440::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Regular', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=13af620df16809ea6c7bcc14ec8d9c9cb6d0a070'] >Thread-6078::DEBUG::2012-12-02 23:49:10,472::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-6078::DEBUG::2012-12-02 23:49:10,473::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS' >Thread-6078::DEBUG::2012-12-02 23:49:10,473::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS' >Thread-6078::INFO::2012-12-02 23:49:10,473::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 (id: 2) >Thread-6142::DEBUG::2012-12-02 23:49:11,905::task::568::TaskManager.Task::(_updateState) Task=`87f1cf6f-8afd-4aef-b40a-0de9856bd074`::moving from state init -> state preparing >Thread-6142::INFO::2012-12-02 23:49:11,905::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6142::INFO::2012-12-02 23:49:11,905::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00771617889404', 'lastCheck': 1354484946.0660319, 'code': 0, 'valid': True}} >Thread-6142::DEBUG::2012-12-02 23:49:11,906::task::1151::TaskManager.Task::(prepare) Task=`87f1cf6f-8afd-4aef-b40a-0de9856bd074`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00771617889404', 'lastCheck': 1354484946.0660319, 'code': 0, 'valid': True}} >Thread-6142::DEBUG::2012-12-02 23:49:11,906::task::568::TaskManager.Task::(_updateState) Task=`87f1cf6f-8afd-4aef-b40a-0de9856bd074`::moving from state preparing -> state finished >Thread-6142::DEBUG::2012-12-02 23:49:11,906::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6142::DEBUG::2012-12-02 23:49:11,906::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6142::DEBUG::2012-12-02 23:49:11,906::task::957::TaskManager.Task::(_decref) Task=`87f1cf6f-8afd-4aef-b40a-0de9856bd074`::ref 0 aborting False >Thread-6148::DEBUG::2012-12-02 23:49:22,876::task::568::TaskManager.Task::(_updateState) Task=`d43e35ff-eed9-4d8c-a3ec-96483cf2dd7a`::moving from state init -> state preparing >Thread-6148::INFO::2012-12-02 23:49:22,877::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6148::INFO::2012-12-02 23:49:22,877::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00988078117371', 'lastCheck': 1354484956.0892701, 'code': 0, 'valid': True}} >Thread-6148::DEBUG::2012-12-02 23:49:22,877::task::1151::TaskManager.Task::(prepare) Task=`d43e35ff-eed9-4d8c-a3ec-96483cf2dd7a`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00988078117371', 'lastCheck': 1354484956.0892701, 'code': 0, 'valid': True}} >Thread-6148::DEBUG::2012-12-02 23:49:22,877::task::568::TaskManager.Task::(_updateState) Task=`d43e35ff-eed9-4d8c-a3ec-96483cf2dd7a`::moving from state preparing -> state finished >Thread-6148::DEBUG::2012-12-02 23:49:22,877::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6148::DEBUG::2012-12-02 23:49:22,877::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6148::DEBUG::2012-12-02 23:49:22,877::task::957::TaskManager.Task::(_decref) Task=`d43e35ff-eed9-4d8c-a3ec-96483cf2dd7a`::ref 0 aborting False >Thread-6078::DEBUG::2012-12-02 23:49:31,474::safelease::178::SANLock::(acquireHostId) Host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully acquired (id: 2) >Thread-6078::INFO::2012-12-02 23:49:31,475::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 (id: 2) >Thread-6078::DEBUG::2012-12-02 23:49:31,830::safelease::235::SANLock::(acquire) Cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully acquired (id: 2) >Thread-6078::DEBUG::2012-12-02 23:49:31,835::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Regular', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=13af620df16809ea6c7bcc14ec8d9c9cb6d0a070'] >Thread-6078::DEBUG::2012-12-02 23:49:31,835::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-6078::DEBUG::2012-12-02 23:49:31,835::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-6078::DEBUG::2012-12-02 23:49:31,835::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Regular', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=b96bbe0b3e52d1781d857ab6865f799bcdf62d20'] >Thread-6078::DEBUG::2012-12-02 23:49:31,928::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-6078::DEBUG::2012-12-02 23:49:31,928::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-6078::DEBUG::2012-12-02 23:49:31,928::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-6078::DEBUG::2012-12-02 23:49:31,928::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=408b7356e66abe7a33bac0a20f4a181b0adfe9d3'] >Thread-6078::DEBUG::2012-12-02 23:49:31,971::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-6078::INFO::2012-12-02 23:49:31,971::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-6078::DEBUG::2012-12-02 23:49:31,971::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-6078::DEBUG::2012-12-02 23:49:31,972::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-6078::DEBUG::2012-12-02 23:49:31,972::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-6078::DEBUG::2012-12-02 23:49:31,972::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-6078::DEBUG::2012-12-02 23:49:31,972::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Regular', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be271c49029e16b3d6130b524ec37986e9416cbd'] >Thread-6078::DEBUG::2012-12-02 23:49:32,011::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-6078::INFO::2012-12-02 23:49:32,012::safelease::239::SANLock::(release) Releasing cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-6078::DEBUG::2012-12-02 23:49:32,027::safelease::250::SANLock::(release) Cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully released >Thread-6078::INFO::2012-12-02 23:49:32,028::safelease::183::SANLock::(releaseHostId) Releasing host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 (id: 2) >Thread-6154::DEBUG::2012-12-02 23:49:33,849::task::568::TaskManager.Task::(_updateState) Task=`10114f6c-55c0-4c2c-a6b6-a13b6d3b7166`::moving from state init -> state preparing >Thread-6154::INFO::2012-12-02 23:49:33,849::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6154::INFO::2012-12-02 23:49:33,849::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00751209259033', 'lastCheck': 1354484966.108573, 'code': 0, 'valid': True}} >Thread-6154::DEBUG::2012-12-02 23:49:33,849::task::1151::TaskManager.Task::(prepare) Task=`10114f6c-55c0-4c2c-a6b6-a13b6d3b7166`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00751209259033', 'lastCheck': 1354484966.108573, 'code': 0, 'valid': True}} >Thread-6154::DEBUG::2012-12-02 23:49:33,849::task::568::TaskManager.Task::(_updateState) Task=`10114f6c-55c0-4c2c-a6b6-a13b6d3b7166`::moving from state preparing -> state finished >Thread-6154::DEBUG::2012-12-02 23:49:33,849::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6154::DEBUG::2012-12-02 23:49:33,850::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6154::DEBUG::2012-12-02 23:49:33,850::task::957::TaskManager.Task::(_decref) Task=`10114f6c-55c0-4c2c-a6b6-a13b6d3b7166`::ref 0 aborting False >Thread-6078::DEBUG::2012-12-02 23:49:35,028::safelease::193::SANLock::(releaseHostId) Host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 released successfully (id: 2) >Thread-6078::DEBUG::2012-12-02 23:49:35,028::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-6078::DEBUG::2012-12-02 23:49:35,028::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-6078::DEBUG::2012-12-02 23:49:35,029::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:35,029::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6078::DEBUG::2012-12-02 23:49:35,233::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6078::WARNING::2012-12-02 23:49:35,235::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6078::DEBUG::2012-12-02 23:49:35,235::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6078::ERROR::2012-12-02 23:49:35,267::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6078::DEBUG::2012-12-02 23:49:35,267::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6078::DEBUG::2012-12-02 23:49:35,267::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6078::DEBUG::2012-12-02 23:49:35,473::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6078::WARNING::2012-12-02 23:49:35,474::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6078::DEBUG::2012-12-02 23:49:35,474::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6078::ERROR::2012-12-02 23:49:35,504::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6078::DEBUG::2012-12-02 23:49:35,504::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-6078::INFO::2012-12-02 23:49:35,504::logUtils::39::dispatcher::(wrapper) Run and protect: attachStorageDomain, Return response: None >Thread-6078::DEBUG::2012-12-02 23:49:35,504::task::1151::TaskManager.Task::(prepare) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::finished: None >Thread-6078::DEBUG::2012-12-02 23:49:35,505::task::568::TaskManager.Task::(_updateState) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::moving from state preparing -> state finished >Thread-6078::DEBUG::2012-12-02 23:49:35,505::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>, 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76': < ResourceRef 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76', isValid: 'True' obj: 'None'>} >Thread-6078::DEBUG::2012-12-02 23:49:35,505::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6078::DEBUG::2012-12-02 23:49:35,505::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6078::DEBUG::2012-12-02 23:49:35,505::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6078::DEBUG::2012-12-02 23:49:35,505::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6078::DEBUG::2012-12-02 23:49:35,506::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6078::DEBUG::2012-12-02 23:49:35,506::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' >Thread-6078::DEBUG::2012-12-02 23:49:35,506::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' (0 active users) >Thread-6078::DEBUG::2012-12-02 23:49:35,506::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' is free, finding out if anyone is waiting for it. >Thread-6078::DEBUG::2012-12-02 23:49:35,506::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76', Clearing records. >Thread-6078::DEBUG::2012-12-02 23:49:35,506::task::957::TaskManager.Task::(_decref) Task=`9136f6b1-daa6-4c72-a855-ff917c434cd8`::ref 0 aborting False >Thread-6280::DEBUG::2012-12-02 23:49:35,647::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6280::DEBUG::2012-12-02 23:49:35,647::task::568::TaskManager.Task::(_updateState) Task=`19fe361e-24f8-4d4b-8eee-52f67945c572`::moving from state init -> state preparing >Thread-6280::INFO::2012-12-02 23:49:35,647::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6280::INFO::2012-12-02 23:49:35,648::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6280::DEBUG::2012-12-02 23:49:35,648::task::1151::TaskManager.Task::(prepare) Task=`19fe361e-24f8-4d4b-8eee-52f67945c572`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6280::DEBUG::2012-12-02 23:49:35,648::task::568::TaskManager.Task::(_updateState) Task=`19fe361e-24f8-4d4b-8eee-52f67945c572`::moving from state preparing -> state finished >Thread-6280::DEBUG::2012-12-02 23:49:35,648::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6280::DEBUG::2012-12-02 23:49:35,648::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6280::DEBUG::2012-12-02 23:49:35,648::task::957::TaskManager.Task::(_decref) Task=`19fe361e-24f8-4d4b-8eee-52f67945c572`::ref 0 aborting False >Thread-6281::DEBUG::2012-12-02 23:49:35,801::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6281::DEBUG::2012-12-02 23:49:35,801::task::568::TaskManager.Task::(_updateState) Task=`efa92260-81b8-4fdf-899e-e7eb50806ab6`::moving from state init -> state preparing >Thread-6281::INFO::2012-12-02 23:49:35,802::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6281::DEBUG::2012-12-02 23:49:35,802::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`aaf1c140-034c-43e9-baba-a9f6083ce7d5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6281::DEBUG::2012-12-02 23:49:35,802::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6281::DEBUG::2012-12-02 23:49:35,802::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6281::DEBUG::2012-12-02 23:49:35,802::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`aaf1c140-034c-43e9-baba-a9f6083ce7d5`::Granted request >Thread-6281::DEBUG::2012-12-02 23:49:35,803::task::794::TaskManager.Task::(resourceAcquired) Task=`efa92260-81b8-4fdf-899e-e7eb50806ab6`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6281::DEBUG::2012-12-02 23:49:35,803::task::957::TaskManager.Task::(_decref) Task=`efa92260-81b8-4fdf-899e-e7eb50806ab6`::ref 1 aborting False >Thread-6281::DEBUG::2012-12-02 23:49:35,808::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6281::DEBUG::2012-12-02 23:49:35,808::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6281::DEBUG::2012-12-02 23:49:36,012::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6281::WARNING::2012-12-02 23:49:36,014::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6281::DEBUG::2012-12-02 23:49:36,014::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6281::WARNING::2012-12-02 23:49:36,045::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6281::DEBUG::2012-12-02 23:49:36,045::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6281::DEBUG::2012-12-02 23:49:36,046::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6281::DEBUG::2012-12-02 23:49:36,253::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6281::WARNING::2012-12-02 23:49:36,254::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6281::DEBUG::2012-12-02 23:49:36,255::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6281::WARNING::2012-12-02 23:49:36,283::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6281::INFO::2012-12-02 23:49:36,284::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6281::DEBUG::2012-12-02 23:49:36,286::task::1151::TaskManager.Task::(prepare) Task=`efa92260-81b8-4fdf-899e-e7eb50806ab6`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6281::DEBUG::2012-12-02 23:49:36,286::task::568::TaskManager.Task::(_updateState) Task=`efa92260-81b8-4fdf-899e-e7eb50806ab6`::moving from state preparing -> state finished >Thread-6281::DEBUG::2012-12-02 23:49:36,288::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6281::DEBUG::2012-12-02 23:49:36,290::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6281::DEBUG::2012-12-02 23:49:36,291::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6281::DEBUG::2012-12-02 23:49:36,291::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6281::DEBUG::2012-12-02 23:49:36,292::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6281::DEBUG::2012-12-02 23:49:36,292::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6281::DEBUG::2012-12-02 23:49:36,292::task::957::TaskManager.Task::(_decref) Task=`efa92260-81b8-4fdf-899e-e7eb50806ab6`::ref 0 aborting False >Thread-6410::DEBUG::2012-12-02 23:49:44,755::task::568::TaskManager.Task::(_updateState) Task=`01dc23c8-f586-4e63-ac18-996da579dd7b`::moving from state init -> state preparing >Thread-6410::INFO::2012-12-02 23:49:44,755::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6410::INFO::2012-12-02 23:49:44,755::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00811791419983', 'lastCheck': 1354484976.30353, 'code': 0, 'valid': True}} >Thread-6410::DEBUG::2012-12-02 23:49:44,755::task::1151::TaskManager.Task::(prepare) Task=`01dc23c8-f586-4e63-ac18-996da579dd7b`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00811791419983', 'lastCheck': 1354484976.30353, 'code': 0, 'valid': True}} >Thread-6410::DEBUG::2012-12-02 23:49:44,755::task::568::TaskManager.Task::(_updateState) Task=`01dc23c8-f586-4e63-ac18-996da579dd7b`::moving from state preparing -> state finished >Thread-6410::DEBUG::2012-12-02 23:49:44,755::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6410::DEBUG::2012-12-02 23:49:44,755::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6410::DEBUG::2012-12-02 23:49:44,756::task::957::TaskManager.Task::(_decref) Task=`01dc23c8-f586-4e63-ac18-996da579dd7b`::ref 0 aborting False >Thread-6412::DEBUG::2012-12-02 23:49:46,537::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6412::DEBUG::2012-12-02 23:49:46,537::task::568::TaskManager.Task::(_updateState) Task=`f5c6a2b0-a611-486a-9a0e-51d8d59e3de2`::moving from state init -> state preparing >Thread-6412::INFO::2012-12-02 23:49:46,537::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6412::INFO::2012-12-02 23:49:46,537::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6412::DEBUG::2012-12-02 23:49:46,538::task::1151::TaskManager.Task::(prepare) Task=`f5c6a2b0-a611-486a-9a0e-51d8d59e3de2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6412::DEBUG::2012-12-02 23:49:46,538::task::568::TaskManager.Task::(_updateState) Task=`f5c6a2b0-a611-486a-9a0e-51d8d59e3de2`::moving from state preparing -> state finished >Thread-6412::DEBUG::2012-12-02 23:49:46,538::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6412::DEBUG::2012-12-02 23:49:46,538::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6412::DEBUG::2012-12-02 23:49:46,538::task::957::TaskManager.Task::(_decref) Task=`f5c6a2b0-a611-486a-9a0e-51d8d59e3de2`::ref 0 aborting False >Thread-6413::DEBUG::2012-12-02 23:49:46,683::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6413::DEBUG::2012-12-02 23:49:46,683::task::568::TaskManager.Task::(_updateState) Task=`68e30224-584d-4943-990d-74d928eb5750`::moving from state init -> state preparing >Thread-6413::INFO::2012-12-02 23:49:46,683::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6413::DEBUG::2012-12-02 23:49:46,684::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`32e4e887-37a1-414b-b942-bdd7cc57c87e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6413::DEBUG::2012-12-02 23:49:46,684::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6413::DEBUG::2012-12-02 23:49:46,684::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6413::DEBUG::2012-12-02 23:49:46,684::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`32e4e887-37a1-414b-b942-bdd7cc57c87e`::Granted request >Thread-6413::DEBUG::2012-12-02 23:49:46,684::task::794::TaskManager.Task::(resourceAcquired) Task=`68e30224-584d-4943-990d-74d928eb5750`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6413::DEBUG::2012-12-02 23:49:46,685::task::957::TaskManager.Task::(_decref) Task=`68e30224-584d-4943-990d-74d928eb5750`::ref 1 aborting False >Thread-6413::DEBUG::2012-12-02 23:49:46,689::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6413::DEBUG::2012-12-02 23:49:46,689::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6413::DEBUG::2012-12-02 23:49:46,893::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6413::WARNING::2012-12-02 23:49:46,894::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6413::DEBUG::2012-12-02 23:49:46,894::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6413::WARNING::2012-12-02 23:49:46,923::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6413::DEBUG::2012-12-02 23:49:46,923::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6413::DEBUG::2012-12-02 23:49:46,924::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6413::DEBUG::2012-12-02 23:49:47,131::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6413::WARNING::2012-12-02 23:49:47,132::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6413::DEBUG::2012-12-02 23:49:47,132::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6413::WARNING::2012-12-02 23:49:47,161::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6413::INFO::2012-12-02 23:49:47,162::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6413::DEBUG::2012-12-02 23:49:47,162::task::1151::TaskManager.Task::(prepare) Task=`68e30224-584d-4943-990d-74d928eb5750`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6413::DEBUG::2012-12-02 23:49:47,162::task::568::TaskManager.Task::(_updateState) Task=`68e30224-584d-4943-990d-74d928eb5750`::moving from state preparing -> state finished >Thread-6413::DEBUG::2012-12-02 23:49:47,162::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6413::DEBUG::2012-12-02 23:49:47,162::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6413::DEBUG::2012-12-02 23:49:47,162::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6413::DEBUG::2012-12-02 23:49:47,163::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6413::DEBUG::2012-12-02 23:49:47,163::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6413::DEBUG::2012-12-02 23:49:47,163::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6413::DEBUG::2012-12-02 23:49:47,163::task::957::TaskManager.Task::(_decref) Task=`68e30224-584d-4943-990d-74d928eb5750`::ref 0 aborting False >Thread-6542::DEBUG::2012-12-02 23:49:55,688::task::568::TaskManager.Task::(_updateState) Task=`c08ee424-44a0-42fb-9f08-d3a9933a9def`::moving from state init -> state preparing >Thread-6542::INFO::2012-12-02 23:49:55,688::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6542::INFO::2012-12-02 23:49:55,689::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00995516777039', 'lastCheck': 1354484986.32516, 'code': 0, 'valid': True}} >Thread-6542::DEBUG::2012-12-02 23:49:55,689::task::1151::TaskManager.Task::(prepare) Task=`c08ee424-44a0-42fb-9f08-d3a9933a9def`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00995516777039', 'lastCheck': 1354484986.32516, 'code': 0, 'valid': True}} >Thread-6542::DEBUG::2012-12-02 23:49:55,689::task::568::TaskManager.Task::(_updateState) Task=`c08ee424-44a0-42fb-9f08-d3a9933a9def`::moving from state preparing -> state finished >Thread-6542::DEBUG::2012-12-02 23:49:55,689::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6542::DEBUG::2012-12-02 23:49:55,689::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6542::DEBUG::2012-12-02 23:49:55,689::task::957::TaskManager.Task::(_decref) Task=`c08ee424-44a0-42fb-9f08-d3a9933a9def`::ref 0 aborting False >Thread-6544::DEBUG::2012-12-02 23:49:57,402::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6544::DEBUG::2012-12-02 23:49:57,403::task::568::TaskManager.Task::(_updateState) Task=`7b45d3ea-a1e1-4c92-a599-2fcd67915a65`::moving from state init -> state preparing >Thread-6544::INFO::2012-12-02 23:49:57,403::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6544::INFO::2012-12-02 23:49:57,403::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6544::DEBUG::2012-12-02 23:49:57,403::task::1151::TaskManager.Task::(prepare) Task=`7b45d3ea-a1e1-4c92-a599-2fcd67915a65`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6544::DEBUG::2012-12-02 23:49:57,403::task::568::TaskManager.Task::(_updateState) Task=`7b45d3ea-a1e1-4c92-a599-2fcd67915a65`::moving from state preparing -> state finished >Thread-6544::DEBUG::2012-12-02 23:49:57,403::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6544::DEBUG::2012-12-02 23:49:57,403::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6544::DEBUG::2012-12-02 23:49:57,404::task::957::TaskManager.Task::(_decref) Task=`7b45d3ea-a1e1-4c92-a599-2fcd67915a65`::ref 0 aborting False >Thread-6545::DEBUG::2012-12-02 23:49:57,547::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6545::DEBUG::2012-12-02 23:49:57,547::task::568::TaskManager.Task::(_updateState) Task=`c0bb9c57-868e-48ea-be15-ee58f14d746d`::moving from state init -> state preparing >Thread-6545::INFO::2012-12-02 23:49:57,548::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6545::DEBUG::2012-12-02 23:49:57,548::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`103332d1-a66a-474e-9ebb-bd647c54a955`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6545::DEBUG::2012-12-02 23:49:57,548::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6545::DEBUG::2012-12-02 23:49:57,548::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6545::DEBUG::2012-12-02 23:49:57,548::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`103332d1-a66a-474e-9ebb-bd647c54a955`::Granted request >Thread-6545::DEBUG::2012-12-02 23:49:57,549::task::794::TaskManager.Task::(resourceAcquired) Task=`c0bb9c57-868e-48ea-be15-ee58f14d746d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6545::DEBUG::2012-12-02 23:49:57,549::task::957::TaskManager.Task::(_decref) Task=`c0bb9c57-868e-48ea-be15-ee58f14d746d`::ref 1 aborting False >Thread-6545::DEBUG::2012-12-02 23:49:57,555::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6545::DEBUG::2012-12-02 23:49:57,556::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6545::DEBUG::2012-12-02 23:49:57,759::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6545::WARNING::2012-12-02 23:49:57,761::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6545::DEBUG::2012-12-02 23:49:57,761::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6605::WARNING::2012-12-02 23:49:57,789::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6607::WARNING::2012-12-02 23:49:57,790::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6545::WARNING::2012-12-02 23:49:57,797::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6545::DEBUG::2012-12-02 23:49:57,797::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6545::DEBUG::2012-12-02 23:49:57,798::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6545::DEBUG::2012-12-02 23:49:58,002::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6545::WARNING::2012-12-02 23:49:58,004::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6545::DEBUG::2012-12-02 23:49:58,004::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6545::WARNING::2012-12-02 23:49:58,035::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6545::INFO::2012-12-02 23:49:58,035::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6545::DEBUG::2012-12-02 23:49:58,036::task::1151::TaskManager.Task::(prepare) Task=`c0bb9c57-868e-48ea-be15-ee58f14d746d`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6545::DEBUG::2012-12-02 23:49:58,036::task::568::TaskManager.Task::(_updateState) Task=`c0bb9c57-868e-48ea-be15-ee58f14d746d`::moving from state preparing -> state finished >Thread-6545::DEBUG::2012-12-02 23:49:58,036::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6545::DEBUG::2012-12-02 23:49:58,036::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6545::DEBUG::2012-12-02 23:49:58,036::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6545::DEBUG::2012-12-02 23:49:58,037::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6545::DEBUG::2012-12-02 23:49:58,037::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6545::DEBUG::2012-12-02 23:49:58,037::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6545::DEBUG::2012-12-02 23:49:58,037::task::957::TaskManager.Task::(_decref) Task=`c0bb9c57-868e-48ea-be15-ee58f14d746d`::ref 0 aborting False >Thread-6674::DEBUG::2012-12-02 23:50:06,625::task::568::TaskManager.Task::(_updateState) Task=`7b37a07c-daf0-472e-9d17-cf2d9539983f`::moving from state init -> state preparing >Thread-6674::INFO::2012-12-02 23:50:06,628::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6674::INFO::2012-12-02 23:50:06,628::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00927400588989', 'lastCheck': 1354485006.366792, 'code': 0, 'valid': True}} >Thread-6674::DEBUG::2012-12-02 23:50:06,628::task::1151::TaskManager.Task::(prepare) Task=`7b37a07c-daf0-472e-9d17-cf2d9539983f`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00927400588989', 'lastCheck': 1354485006.366792, 'code': 0, 'valid': True}} >Thread-6674::DEBUG::2012-12-02 23:50:06,628::task::568::TaskManager.Task::(_updateState) Task=`7b37a07c-daf0-472e-9d17-cf2d9539983f`::moving from state preparing -> state finished >Thread-6674::DEBUG::2012-12-02 23:50:06,628::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6674::DEBUG::2012-12-02 23:50:06,628::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6674::DEBUG::2012-12-02 23:50:06,628::task::957::TaskManager.Task::(_decref) Task=`7b37a07c-daf0-472e-9d17-cf2d9539983f`::ref 0 aborting False >Thread-6676::DEBUG::2012-12-02 23:50:08,282::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6676::DEBUG::2012-12-02 23:50:08,282::task::568::TaskManager.Task::(_updateState) Task=`88d1996c-237a-49fa-a516-5d64d2c3e3b4`::moving from state init -> state preparing >Thread-6676::INFO::2012-12-02 23:50:08,283::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6676::INFO::2012-12-02 23:50:08,283::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6676::DEBUG::2012-12-02 23:50:08,283::task::1151::TaskManager.Task::(prepare) Task=`88d1996c-237a-49fa-a516-5d64d2c3e3b4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6676::DEBUG::2012-12-02 23:50:08,283::task::568::TaskManager.Task::(_updateState) Task=`88d1996c-237a-49fa-a516-5d64d2c3e3b4`::moving from state preparing -> state finished >Thread-6676::DEBUG::2012-12-02 23:50:08,283::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6676::DEBUG::2012-12-02 23:50:08,283::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6676::DEBUG::2012-12-02 23:50:08,284::task::957::TaskManager.Task::(_decref) Task=`88d1996c-237a-49fa-a516-5d64d2c3e3b4`::ref 0 aborting False >Thread-6677::DEBUG::2012-12-02 23:50:08,440::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6677::DEBUG::2012-12-02 23:50:08,441::task::568::TaskManager.Task::(_updateState) Task=`98290ac4-4bf6-4b3c-b9ef-702d343c735e`::moving from state init -> state preparing >Thread-6677::INFO::2012-12-02 23:50:08,441::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6677::DEBUG::2012-12-02 23:50:08,441::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b41a99ef-d98c-4ce6-b493-8d5c75107bea`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6677::DEBUG::2012-12-02 23:50:08,441::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6677::DEBUG::2012-12-02 23:50:08,441::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6677::DEBUG::2012-12-02 23:50:08,442::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b41a99ef-d98c-4ce6-b493-8d5c75107bea`::Granted request >Thread-6677::DEBUG::2012-12-02 23:50:08,442::task::794::TaskManager.Task::(resourceAcquired) Task=`98290ac4-4bf6-4b3c-b9ef-702d343c735e`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6677::DEBUG::2012-12-02 23:50:08,442::task::957::TaskManager.Task::(_decref) Task=`98290ac4-4bf6-4b3c-b9ef-702d343c735e`::ref 1 aborting False >Thread-6677::DEBUG::2012-12-02 23:50:08,448::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6677::DEBUG::2012-12-02 23:50:08,449::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6677::DEBUG::2012-12-02 23:50:08,658::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6677::WARNING::2012-12-02 23:50:08,660::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6677::DEBUG::2012-12-02 23:50:08,660::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6677::WARNING::2012-12-02 23:50:08,691::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6677::DEBUG::2012-12-02 23:50:08,691::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6677::DEBUG::2012-12-02 23:50:08,691::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6677::DEBUG::2012-12-02 23:50:08,895::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6677::WARNING::2012-12-02 23:50:08,897::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6677::DEBUG::2012-12-02 23:50:08,897::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6677::WARNING::2012-12-02 23:50:08,927::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6677::INFO::2012-12-02 23:50:08,928::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6677::DEBUG::2012-12-02 23:50:08,928::task::1151::TaskManager.Task::(prepare) Task=`98290ac4-4bf6-4b3c-b9ef-702d343c735e`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6677::DEBUG::2012-12-02 23:50:08,928::task::568::TaskManager.Task::(_updateState) Task=`98290ac4-4bf6-4b3c-b9ef-702d343c735e`::moving from state preparing -> state finished >Thread-6677::DEBUG::2012-12-02 23:50:08,928::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6677::DEBUG::2012-12-02 23:50:08,929::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6677::DEBUG::2012-12-02 23:50:08,929::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6677::DEBUG::2012-12-02 23:50:08,929::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6677::DEBUG::2012-12-02 23:50:08,929::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6677::DEBUG::2012-12-02 23:50:08,929::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6677::DEBUG::2012-12-02 23:50:08,929::task::957::TaskManager.Task::(_decref) Task=`98290ac4-4bf6-4b3c-b9ef-702d343c735e`::ref 0 aborting False >Thread-6806::DEBUG::2012-12-02 23:50:17,728::task::568::TaskManager.Task::(_updateState) Task=`6ff95ee4-b1e4-4e1e-9d84-afb98b72a615`::moving from state init -> state preparing >Thread-6806::INFO::2012-12-02 23:50:17,729::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6806::INFO::2012-12-02 23:50:17,729::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00823187828064', 'lastCheck': 1354485016.388376, 'code': 0, 'valid': True}} >Thread-6806::DEBUG::2012-12-02 23:50:17,729::task::1151::TaskManager.Task::(prepare) Task=`6ff95ee4-b1e4-4e1e-9d84-afb98b72a615`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00823187828064', 'lastCheck': 1354485016.388376, 'code': 0, 'valid': True}} >Thread-6806::DEBUG::2012-12-02 23:50:17,729::task::568::TaskManager.Task::(_updateState) Task=`6ff95ee4-b1e4-4e1e-9d84-afb98b72a615`::moving from state preparing -> state finished >Thread-6806::DEBUG::2012-12-02 23:50:17,729::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6806::DEBUG::2012-12-02 23:50:17,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6806::DEBUG::2012-12-02 23:50:17,730::task::957::TaskManager.Task::(_decref) Task=`6ff95ee4-b1e4-4e1e-9d84-afb98b72a615`::ref 0 aborting False >Thread-6808::DEBUG::2012-12-02 23:50:19,219::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6808::DEBUG::2012-12-02 23:50:19,219::task::568::TaskManager.Task::(_updateState) Task=`ad07bd21-170b-4f25-a0c7-d695d7776371`::moving from state init -> state preparing >Thread-6808::INFO::2012-12-02 23:50:19,219::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6808::INFO::2012-12-02 23:50:19,220::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6808::DEBUG::2012-12-02 23:50:19,220::task::1151::TaskManager.Task::(prepare) Task=`ad07bd21-170b-4f25-a0c7-d695d7776371`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6808::DEBUG::2012-12-02 23:50:19,220::task::568::TaskManager.Task::(_updateState) Task=`ad07bd21-170b-4f25-a0c7-d695d7776371`::moving from state preparing -> state finished >Thread-6808::DEBUG::2012-12-02 23:50:19,220::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6808::DEBUG::2012-12-02 23:50:19,220::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6808::DEBUG::2012-12-02 23:50:19,220::task::957::TaskManager.Task::(_decref) Task=`ad07bd21-170b-4f25-a0c7-d695d7776371`::ref 0 aborting False >Thread-6809::DEBUG::2012-12-02 23:50:19,453::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6809::DEBUG::2012-12-02 23:50:19,453::task::568::TaskManager.Task::(_updateState) Task=`0b5102cb-11a2-47e6-9559-bb6204c94170`::moving from state init -> state preparing >Thread-6809::INFO::2012-12-02 23:50:19,453::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6809::DEBUG::2012-12-02 23:50:19,454::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`163f66f2-79db-451d-ad04-930b243c5847`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6809::DEBUG::2012-12-02 23:50:19,454::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6809::DEBUG::2012-12-02 23:50:19,454::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6809::DEBUG::2012-12-02 23:50:19,454::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`163f66f2-79db-451d-ad04-930b243c5847`::Granted request >Thread-6809::DEBUG::2012-12-02 23:50:19,454::task::794::TaskManager.Task::(resourceAcquired) Task=`0b5102cb-11a2-47e6-9559-bb6204c94170`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6809::DEBUG::2012-12-02 23:50:19,454::task::957::TaskManager.Task::(_decref) Task=`0b5102cb-11a2-47e6-9559-bb6204c94170`::ref 1 aborting False >Thread-6809::DEBUG::2012-12-02 23:50:19,459::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6809::DEBUG::2012-12-02 23:50:19,459::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6809::DEBUG::2012-12-02 23:50:19,663::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6809::WARNING::2012-12-02 23:50:19,664::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6809::DEBUG::2012-12-02 23:50:19,664::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6809::WARNING::2012-12-02 23:50:19,695::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6809::DEBUG::2012-12-02 23:50:19,695::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6809::DEBUG::2012-12-02 23:50:19,696::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6809::DEBUG::2012-12-02 23:50:19,898::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6809::WARNING::2012-12-02 23:50:19,899::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6809::DEBUG::2012-12-02 23:50:19,900::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6809::WARNING::2012-12-02 23:50:19,929::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6809::INFO::2012-12-02 23:50:19,929::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6809::DEBUG::2012-12-02 23:50:19,929::task::1151::TaskManager.Task::(prepare) Task=`0b5102cb-11a2-47e6-9559-bb6204c94170`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6809::DEBUG::2012-12-02 23:50:19,929::task::568::TaskManager.Task::(_updateState) Task=`0b5102cb-11a2-47e6-9559-bb6204c94170`::moving from state preparing -> state finished >Thread-6809::DEBUG::2012-12-02 23:50:19,930::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6809::DEBUG::2012-12-02 23:50:19,930::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6809::DEBUG::2012-12-02 23:50:19,930::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6809::DEBUG::2012-12-02 23:50:19,930::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6809::DEBUG::2012-12-02 23:50:19,930::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6809::DEBUG::2012-12-02 23:50:19,930::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6809::DEBUG::2012-12-02 23:50:19,930::task::957::TaskManager.Task::(_decref) Task=`0b5102cb-11a2-47e6-9559-bb6204c94170`::ref 0 aborting False >Thread-6938::DEBUG::2012-12-02 23:50:29,833::task::568::TaskManager.Task::(_updateState) Task=`a030edae-8476-4477-b899-743c1ed5fcbe`::moving from state init -> state preparing >Thread-6938::INFO::2012-12-02 23:50:29,833::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-6938::INFO::2012-12-02 23:50:29,833::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00839996337891', 'lastCheck': 1354485026.4078751, 'code': 0, 'valid': True}} >Thread-6938::DEBUG::2012-12-02 23:50:29,833::task::1151::TaskManager.Task::(prepare) Task=`a030edae-8476-4477-b899-743c1ed5fcbe`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00839996337891', 'lastCheck': 1354485026.4078751, 'code': 0, 'valid': True}} >Thread-6938::DEBUG::2012-12-02 23:50:29,833::task::568::TaskManager.Task::(_updateState) Task=`a030edae-8476-4477-b899-743c1ed5fcbe`::moving from state preparing -> state finished >Thread-6938::DEBUG::2012-12-02 23:50:29,834::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6938::DEBUG::2012-12-02 23:50:29,834::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6938::DEBUG::2012-12-02 23:50:29,834::task::957::TaskManager.Task::(_decref) Task=`a030edae-8476-4477-b899-743c1ed5fcbe`::ref 0 aborting False >Thread-6940::DEBUG::2012-12-02 23:50:30,180::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6940::DEBUG::2012-12-02 23:50:30,180::task::568::TaskManager.Task::(_updateState) Task=`c35bda82-9d56-453f-863e-87a1667b0ef0`::moving from state init -> state preparing >Thread-6940::INFO::2012-12-02 23:50:30,180::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6940::INFO::2012-12-02 23:50:30,181::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6940::DEBUG::2012-12-02 23:50:30,181::task::1151::TaskManager.Task::(prepare) Task=`c35bda82-9d56-453f-863e-87a1667b0ef0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-6940::DEBUG::2012-12-02 23:50:30,181::task::568::TaskManager.Task::(_updateState) Task=`c35bda82-9d56-453f-863e-87a1667b0ef0`::moving from state preparing -> state finished >Thread-6940::DEBUG::2012-12-02 23:50:30,181::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-6940::DEBUG::2012-12-02 23:50:30,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6940::DEBUG::2012-12-02 23:50:30,181::task::957::TaskManager.Task::(_decref) Task=`c35bda82-9d56-453f-863e-87a1667b0ef0`::ref 0 aborting False >Thread-6941::DEBUG::2012-12-02 23:50:30,319::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-6941::DEBUG::2012-12-02 23:50:30,319::task::568::TaskManager.Task::(_updateState) Task=`e51b6d83-7667-4ba7-bb4b-182df1e74aff`::moving from state init -> state preparing >Thread-6941::INFO::2012-12-02 23:50:30,319::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-6941::DEBUG::2012-12-02 23:50:30,320::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a3febbbb-4ca6-4503-b556-488b670caa21`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-6941::DEBUG::2012-12-02 23:50:30,320::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-6941::DEBUG::2012-12-02 23:50:30,320::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-6941::DEBUG::2012-12-02 23:50:30,320::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a3febbbb-4ca6-4503-b556-488b670caa21`::Granted request >Thread-6941::DEBUG::2012-12-02 23:50:30,320::task::794::TaskManager.Task::(resourceAcquired) Task=`e51b6d83-7667-4ba7-bb4b-182df1e74aff`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-6941::DEBUG::2012-12-02 23:50:30,321::task::957::TaskManager.Task::(_decref) Task=`e51b6d83-7667-4ba7-bb4b-182df1e74aff`::ref 1 aborting False >Thread-6941::DEBUG::2012-12-02 23:50:30,326::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6941::DEBUG::2012-12-02 23:50:30,327::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-6941::DEBUG::2012-12-02 23:50:30,532::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-6941::WARNING::2012-12-02 23:50:30,533::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-6941::DEBUG::2012-12-02 23:50:30,533::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6941::WARNING::2012-12-02 23:50:30,564::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-6941::DEBUG::2012-12-02 23:50:30,565::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-6941::DEBUG::2012-12-02 23:50:30,565::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-6941::DEBUG::2012-12-02 23:50:30,768::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-6941::WARNING::2012-12-02 23:50:30,769::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-6941::DEBUG::2012-12-02 23:50:30,770::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6941::WARNING::2012-12-02 23:50:30,798::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-6941::INFO::2012-12-02 23:50:30,798::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6941::DEBUG::2012-12-02 23:50:30,798::task::1151::TaskManager.Task::(prepare) Task=`e51b6d83-7667-4ba7-bb4b-182df1e74aff`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Attached', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Attached'}}} >Thread-6941::DEBUG::2012-12-02 23:50:30,799::task::568::TaskManager.Task::(_updateState) Task=`e51b6d83-7667-4ba7-bb4b-182df1e74aff`::moving from state preparing -> state finished >Thread-6941::DEBUG::2012-12-02 23:50:30,799::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-6941::DEBUG::2012-12-02 23:50:30,799::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-6941::DEBUG::2012-12-02 23:50:30,799::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-6941::DEBUG::2012-12-02 23:50:30,799::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-6941::DEBUG::2012-12-02 23:50:30,799::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-6941::DEBUG::2012-12-02 23:50:30,800::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-6941::DEBUG::2012-12-02 23:50:30,800::task::957::TaskManager.Task::(_decref) Task=`e51b6d83-7667-4ba7-bb4b-182df1e74aff`::ref 0 aborting False >Thread-7069::DEBUG::2012-12-02 23:50:37,553::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7069::DEBUG::2012-12-02 23:50:37,553::task::568::TaskManager.Task::(_updateState) Task=`182e77f8-4d06-41a3-8256-0a8b05ab6678`::moving from state init -> state preparing >Thread-7069::INFO::2012-12-02 23:50:37,553::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-7069::INFO::2012-12-02 23:50:37,554::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-7069::DEBUG::2012-12-02 23:50:37,554::task::1151::TaskManager.Task::(prepare) Task=`182e77f8-4d06-41a3-8256-0a8b05ab6678`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-7069::DEBUG::2012-12-02 23:50:37,554::task::568::TaskManager.Task::(_updateState) Task=`182e77f8-4d06-41a3-8256-0a8b05ab6678`::moving from state preparing -> state finished >Thread-7069::DEBUG::2012-12-02 23:50:37,554::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7069::DEBUG::2012-12-02 23:50:37,554::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7069::DEBUG::2012-12-02 23:50:37,554::task::957::TaskManager.Task::(_decref) Task=`182e77f8-4d06-41a3-8256-0a8b05ab6678`::ref 0 aborting False >Thread-7070::DEBUG::2012-12-02 23:50:37,717::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7070::DEBUG::2012-12-02 23:50:37,718::task::568::TaskManager.Task::(_updateState) Task=`cc80b92c-b4eb-47ea-b6cd-c162d38d4f6f`::moving from state init -> state preparing >Thread-7070::INFO::2012-12-02 23:50:37,718::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-7070::DEBUG::2012-12-02 23:50:37,724::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7070::DEBUG::2012-12-02 23:50:37,724::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7070::DEBUG::2012-12-02 23:50:37,724::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7070::DEBUG::2012-12-02 23:50:37,725::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7070::DEBUG::2012-12-02 23:50:37,725::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7070::DEBUG::2012-12-02 23:50:37,725::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7070::INFO::2012-12-02 23:50:37,725::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-7070::DEBUG::2012-12-02 23:50:37,725::task::1151::TaskManager.Task::(prepare) Task=`cc80b92c-b4eb-47ea-b6cd-c162d38d4f6f`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-7070::DEBUG::2012-12-02 23:50:37,725::task::568::TaskManager.Task::(_updateState) Task=`cc80b92c-b4eb-47ea-b6cd-c162d38d4f6f`::moving from state preparing -> state finished >Thread-7070::DEBUG::2012-12-02 23:50:37,725::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7070::DEBUG::2012-12-02 23:50:37,725::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7070::DEBUG::2012-12-02 23:50:37,726::task::957::TaskManager.Task::(_decref) Task=`cc80b92c-b4eb-47ea-b6cd-c162d38d4f6f`::ref 0 aborting False >Thread-7071::DEBUG::2012-12-02 23:50:37,866::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7071::DEBUG::2012-12-02 23:50:37,867::task::568::TaskManager.Task::(_updateState) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::moving from state init -> state preparing >Thread-7071::INFO::2012-12-02 23:50:37,867::logUtils::37::dispatcher::(wrapper) Run and protect: detachStorageDomain(sdUUID='a00efcc1-5685-4dbe-ae70-440e9f403b76', spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='00000000-0000-0000-0000-000000000000', masterVersion=3, options=None) >Thread-7071::DEBUG::2012-12-02 23:50:37,867::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8cb9862c-8e17-439e-98f5-51bd4ea46d6f`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7071::DEBUG::2012-12-02 23:50:37,867::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-7071::DEBUG::2012-12-02 23:50:37,868::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-7071::DEBUG::2012-12-02 23:50:37,868::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8cb9862c-8e17-439e-98f5-51bd4ea46d6f`::Granted request >Thread-7071::DEBUG::2012-12-02 23:50:37,868::task::794::TaskManager.Task::(resourceAcquired) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-7071::DEBUG::2012-12-02 23:50:37,868::task::957::TaskManager.Task::(_decref) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::ref 1 aborting False >Thread-7071::DEBUG::2012-12-02 23:50:37,868::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76`ReqID=`c8f22abd-7c84-4a7c-a58d-11fe1d52e92e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7071::DEBUG::2012-12-02 23:50:37,868::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' for lock type 'exclusive' >Thread-7071::DEBUG::2012-12-02 23:50:37,869::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' is free. Now locking as 'exclusive' (1 active user) >Thread-7071::DEBUG::2012-12-02 23:50:37,869::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76`ReqID=`c8f22abd-7c84-4a7c-a58d-11fe1d52e92e`::Granted request >Thread-7071::DEBUG::2012-12-02 23:50:37,869::task::794::TaskManager.Task::(resourceAcquired) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::_resourcesAcquired: Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76 (exclusive) >Thread-7071::DEBUG::2012-12-02 23:50:37,869::task::957::TaskManager.Task::(_decref) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::ref 1 aborting False >Thread-7071::INFO::2012-12-02 23:50:37,869::sp::978::Storage.StoragePool::(detachSD) sdUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76 spUUID=479cf195-bb99-45c5-b400-433d810eaa48 >Thread-7071::DEBUG::2012-12-02 23:50:37,869::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-7071::DEBUG::2012-12-02 23:50:37,870::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7071::DEBUG::2012-12-02 23:50:37,870::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-7071::DEBUG::2012-12-02 23:50:37,870::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7071::DEBUG::2012-12-02 23:50:37,870::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-7071::DEBUG::2012-12-02 23:50:37,895::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-7071::DEBUG::2012-12-02 23:50:37,895::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7071::DEBUG::2012-12-02 23:50:38,529::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-7071::DEBUG::2012-12-02 23:50:38,611::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-7071::DEBUG::2012-12-02 23:50:38,612::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,612::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,612::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,612::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,613::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,613::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,613::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7071::DEBUG::2012-12-02 23:50:38,629::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Regular', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=be271c49029e16b3d6130b524ec37986e9416cbd'] >Thread-7071::DEBUG::2012-12-02 23:50:38,630::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-7071::DEBUG::2012-12-02 23:50:38,630::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-7071::DEBUG::2012-12-02 23:50:38,630::persistentDict::160::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-7071::DEBUG::2012-12-02 23:50:38,630::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-7071::DEBUG::2012-12-02 23:50:38,630::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Regular', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=5c00791b0d1a76768c10e5dfa4771f21625ea254'] >Thread-7071::DEBUG::2012-12-02 23:50:38,673::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-7071::WARNING::2012-12-02 23:50:38,673::sp::951::Storage.StoragePool::(forcedDetachSD) Force detaching domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` >Thread-7071::DEBUG::2012-12-02 23:50:38,674::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-7071::DEBUG::2012-12-02 23:50:38,674::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-7071::DEBUG::2012-12-02 23:50:38,674::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >Thread-7071::DEBUG::2012-12-02 23:50:38,720::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-7071::DEBUG::2012-12-02 23:50:38,720::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-7071::DEBUG::2012-12-02 23:50:38,720::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7071::DEBUG::2012-12-02 23:50:38,720::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:38,722::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7071::DEBUG::2012-12-02 23:50:38,933::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7071::WARNING::2012-12-02 23:50:38,934::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7071::DEBUG::2012-12-02 23:50:38,934::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7071::ERROR::2012-12-02 23:50:39,010::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7071::DEBUG::2012-12-02 23:50:39,011::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7071::DEBUG::2012-12-02 23:50:39,011::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7071::DEBUG::2012-12-02 23:50:39,215::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7071::WARNING::2012-12-02 23:50:39,216::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7071::DEBUG::2012-12-02 23:50:39,217::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7071::ERROR::2012-12-02 23:50:39,247::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7071::DEBUG::2012-12-02 23:50:39,247::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7071::DEBUG::2012-12-02 23:50:39,248::sp::968::Storage.StoragePool::(forcedDetachSD) Force detach for domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` is done >Thread-7071::INFO::2012-12-02 23:50:39,248::logUtils::39::dispatcher::(wrapper) Run and protect: detachStorageDomain, Return response: None >Thread-7071::DEBUG::2012-12-02 23:50:39,248::task::1151::TaskManager.Task::(prepare) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::finished: None >Thread-7071::DEBUG::2012-12-02 23:50:39,248::task::568::TaskManager.Task::(_updateState) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::moving from state preparing -> state finished >Thread-7071::DEBUG::2012-12-02 23:50:39,248::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>, 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76': < ResourceRef 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76', isValid: 'True' obj: 'None'>} >Thread-7071::DEBUG::2012-12-02 23:50:39,248::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' (0 active users) >Thread-7071::DEBUG::2012-12-02 23:50:39,249::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' is free, finding out if anyone is waiting for it. >Thread-7071::DEBUG::2012-12-02 23:50:39,250::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76', Clearing records. >Thread-7071::DEBUG::2012-12-02 23:50:39,250::task::957::TaskManager.Task::(_decref) Task=`fab89f96-4bfe-46ac-bb5f-8258a66fc54d`::ref 0 aborting False >Thread-7197::DEBUG::2012-12-02 23:50:39,481::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7197::DEBUG::2012-12-02 23:50:39,482::task::568::TaskManager.Task::(_updateState) Task=`e7928fa9-d972-4aa8-909d-14f907b2e2ae`::moving from state init -> state preparing >Thread-7197::INFO::2012-12-02 23:50:39,482::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='5971512c-ee51-475e-b956-3872c38ce193', masterVersion=3, options=None) >Thread-7197::DEBUG::2012-12-02 23:50:39,482::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b89b87d1-3d46-4485-bfac-63def0fe3a6e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7197::DEBUG::2012-12-02 23:50:39,482::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7197::DEBUG::2012-12-02 23:50:39,483::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7197::DEBUG::2012-12-02 23:50:39,483::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b89b87d1-3d46-4485-bfac-63def0fe3a6e`::Granted request >Thread-7197::DEBUG::2012-12-02 23:50:39,483::task::794::TaskManager.Task::(resourceAcquired) Task=`e7928fa9-d972-4aa8-909d-14f907b2e2ae`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7197::DEBUG::2012-12-02 23:50:39,483::task::957::TaskManager.Task::(_decref) Task=`e7928fa9-d972-4aa8-909d-14f907b2e2ae`::ref 1 aborting False >Thread-7197::INFO::2012-12-02 23:50:39,483::fileSD::260::Storage.StorageDomain::(validate) sdUUID=5971512c-ee51-475e-b956-3872c38ce193 >Thread-7197::DEBUG::2012-12-02 23:50:39,495::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >Thread-7197::DEBUG::2012-12-02 23:50:39,495::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:39,495::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:39,496::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:39,496::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:39,496::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:39,496::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:39,496::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-7197::DEBUG::2012-12-02 23:50:39,496::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7197::DEBUG::2012-12-02 23:50:39,497::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-7197::DEBUG::2012-12-02 23:50:39,497::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7197::DEBUG::2012-12-02 23:50:39,497::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-7197::DEBUG::2012-12-02 23:50:39,522::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-7197::DEBUG::2012-12-02 23:50:39,522::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7197::DEBUG::2012-12-02 23:50:40,156::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-7197::DEBUG::2012-12-02 23:50:40,258::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-7197::DEBUG::2012-12-02 23:50:40,259::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,259::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,259::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,259::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,259::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,259::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,260::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7197::DEBUG::2012-12-02 23:50:40,260::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,262::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-7197::DEBUG::2012-12-02 23:50:40,466::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-7197::WARNING::2012-12-02 23:50:40,467::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-7197::DEBUG::2012-12-02 23:50:40,468::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,503::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-7197::DEBUG::2012-12-02 23:50:40,504::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-7197::DEBUG::2012-12-02 23:50:40,512::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >Thread-7197::DEBUG::2012-12-02 23:50:40,515::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-7197::WARNING::2012-12-02 23:50:40,515::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >Thread-7197::WARNING::2012-12-02 23:50:40,515::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >Thread-7197::DEBUG::2012-12-02 23:50:40,515::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain 5971512c-ee51-475e-b956-3872c38ce193 verified, version 3 >Thread-7197::DEBUG::2012-12-02 23:50:40,515::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-7197::DEBUG::2012-12-02 23:50:40,515::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7197::DEBUG::2012-12-02 23:50:40,516::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,516::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7197::DEBUG::2012-12-02 23:50:40,723::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7197::WARNING::2012-12-02 23:50:40,724::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7197::DEBUG::2012-12-02 23:50:40,725::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7197::ERROR::2012-12-02 23:50:40,754::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7197::DEBUG::2012-12-02 23:50:40,755::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,755::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7322::DEBUG::2012-12-02 23:50:40,778::task::568::TaskManager.Task::(_updateState) Task=`6c02cbb1-ef86-4dbe-9e1d-53952e7d5a25`::moving from state init -> state preparing >Thread-7322::INFO::2012-12-02 23:50:40,779::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7322::INFO::2012-12-02 23:50:40,779::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0084240436554', 'lastCheck': 1354485036.4266081, 'code': 0, 'valid': True}} >Thread-7322::DEBUG::2012-12-02 23:50:40,779::task::1151::TaskManager.Task::(prepare) Task=`6c02cbb1-ef86-4dbe-9e1d-53952e7d5a25`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0084240436554', 'lastCheck': 1354485036.4266081, 'code': 0, 'valid': True}} >Thread-7322::DEBUG::2012-12-02 23:50:40,779::task::568::TaskManager.Task::(_updateState) Task=`6c02cbb1-ef86-4dbe-9e1d-53952e7d5a25`::moving from state preparing -> state finished >Thread-7322::DEBUG::2012-12-02 23:50:40,779::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7322::DEBUG::2012-12-02 23:50:40,780::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7322::DEBUG::2012-12-02 23:50:40,780::task::957::TaskManager.Task::(_decref) Task=`6c02cbb1-ef86-4dbe-9e1d-53952e7d5a25`::ref 0 aborting False >Thread-7197::DEBUG::2012-12-02 23:50:40,963::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7197::WARNING::2012-12-02 23:50:40,965::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7197::DEBUG::2012-12-02 23:50:40,965::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7197::ERROR::2012-12-02 23:50:40,995::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7197::DEBUG::2012-12-02 23:50:40,995::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7197::DEBUG::2012-12-02 23:50:40,995::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7197::DEBUG::2012-12-02 23:50:40,996::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7386::DEBUG::2012-12-02 23:50:41,083::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7386::DEBUG::2012-12-02 23:50:41,084::task::568::TaskManager.Task::(_updateState) Task=`2154e108-c979-4b03-9128-173c2e224610`::moving from state init -> state preparing >Thread-7386::INFO::2012-12-02 23:50:41,084::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7386::INFO::2012-12-02 23:50:41,084::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7386::DEBUG::2012-12-02 23:50:41,085::task::1151::TaskManager.Task::(prepare) Task=`2154e108-c979-4b03-9128-173c2e224610`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7386::DEBUG::2012-12-02 23:50:41,085::task::568::TaskManager.Task::(_updateState) Task=`2154e108-c979-4b03-9128-173c2e224610`::moving from state preparing -> state finished >Thread-7386::DEBUG::2012-12-02 23:50:41,085::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7386::DEBUG::2012-12-02 23:50:41,085::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7386::DEBUG::2012-12-02 23:50:41,085::task::957::TaskManager.Task::(_decref) Task=`2154e108-c979-4b03-9128-173c2e224610`::ref 0 aborting False >Thread-7387::DEBUG::2012-12-02 23:50:41,225::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7387::DEBUG::2012-12-02 23:50:41,225::task::568::TaskManager.Task::(_updateState) Task=`6bae0e0c-1097-457d-aaa2-49c93f1bf8ba`::moving from state init -> state preparing >Thread-7387::INFO::2012-12-02 23:50:41,225::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7387::DEBUG::2012-12-02 23:50:41,226::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`352566c9-6a92-41f0-85b4-4fd11485c8aa`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7387::DEBUG::2012-12-02 23:50:41,226::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7387::DEBUG::2012-12-02 23:50:41,226::resourceManager::504::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-7387::DEBUG::2012-12-02 23:50:41,226::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`352566c9-6a92-41f0-85b4-4fd11485c8aa`::Granted request >Thread-7387::DEBUG::2012-12-02 23:50:41,226::task::794::TaskManager.Task::(resourceAcquired) Task=`6bae0e0c-1097-457d-aaa2-49c93f1bf8ba`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7387::DEBUG::2012-12-02 23:50:41,227::task::957::TaskManager.Task::(_decref) Task=`6bae0e0c-1097-457d-aaa2-49c93f1bf8ba`::ref 1 aborting False >Thread-7387::DEBUG::2012-12-02 23:50:41,234::lvm::368::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-7387::DEBUG::2012-12-02 23:50:41,234::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7197::DEBUG::2012-12-02 23:50:41,249::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7197::WARNING::2012-12-02 23:50:41,251::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found', ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7387::DEBUG::2012-12-02 23:50:41,444::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7387::WARNING::2012-12-02 23:50:41,446::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7387::DEBUG::2012-12-02 23:50:41,446::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7197::INFO::2012-12-02 23:50:41,474::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/5971512c-ee51-475e-b956-3872c38ce193 >Thread-7387::WARNING::2012-12-02 23:50:41,474::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7387::DEBUG::2012-12-02 23:50:41,474::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7387::DEBUG::2012-12-02 23:50:41,475::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7387::DEBUG::2012-12-02 23:50:41,682::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7387::WARNING::2012-12-02 23:50:41,683::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7387::DEBUG::2012-12-02 23:50:41,683::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7197::INFO::2012-12-02 23:50:41,713::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-7387::WARNING::2012-12-02 23:50:41,712::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7387::INFO::2012-12-02 23:50:41,713::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7197::INFO::2012-12-02 23:50:41,713::logUtils::39::dispatcher::(wrapper) Run and protect: refreshStoragePool, Return response: None >Thread-7387::DEBUG::2012-12-02 23:50:41,714::task::1151::TaskManager.Task::(prepare) Task=`6bae0e0c-1097-457d-aaa2-49c93f1bf8ba`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7197::DEBUG::2012-12-02 23:50:41,714::task::1151::TaskManager.Task::(prepare) Task=`e7928fa9-d972-4aa8-909d-14f907b2e2ae`::finished: None >Thread-7387::DEBUG::2012-12-02 23:50:41,714::task::568::TaskManager.Task::(_updateState) Task=`6bae0e0c-1097-457d-aaa2-49c93f1bf8ba`::moving from state preparing -> state finished >Thread-7197::DEBUG::2012-12-02 23:50:41,714::task::568::TaskManager.Task::(_updateState) Task=`e7928fa9-d972-4aa8-909d-14f907b2e2ae`::moving from state preparing -> state finished >Thread-7387::DEBUG::2012-12-02 23:50:41,714::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7197::DEBUG::2012-12-02 23:50:41,715::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7387::DEBUG::2012-12-02 23:50:41,715::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7197::DEBUG::2012-12-02 23:50:41,715::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7387::DEBUG::2012-12-02 23:50:41,715::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7197::DEBUG::2012-12-02 23:50:41,715::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7387::DEBUG::2012-12-02 23:50:41,716::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (1 active users) >Thread-7197::DEBUG::2012-12-02 23:50:41,716::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7387::DEBUG::2012-12-02 23:50:41,716::task::957::TaskManager.Task::(_decref) Task=`6bae0e0c-1097-457d-aaa2-49c93f1bf8ba`::ref 0 aborting False >Thread-7197::DEBUG::2012-12-02 23:50:41,716::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7197::DEBUG::2012-12-02 23:50:41,717::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7197::DEBUG::2012-12-02 23:50:41,717::task::957::TaskManager.Task::(_decref) Task=`e7928fa9-d972-4aa8-909d-14f907b2e2ae`::ref 0 aborting False >Thread-7573::DEBUG::2012-12-02 23:50:41,890::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7573::DEBUG::2012-12-02 23:50:41,890::task::568::TaskManager.Task::(_updateState) Task=`e5044e76-dcef-40f2-b15e-b42e21c0164f`::moving from state init -> state preparing >Thread-7573::INFO::2012-12-02 23:50:41,890::logUtils::37::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-7573::DEBUG::2012-12-02 23:50:41,890::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount -f -l /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348' (cwd None) >Thread-7573::DEBUG::2012-12-02 23:50:41,912::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-7573::DEBUG::2012-12-02 23:50:41,912::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7573::DEBUG::2012-12-02 23:50:41,912::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-7573::DEBUG::2012-12-02 23:50:41,912::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-7573::DEBUG::2012-12-02 23:50:41,912::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-7573::DEBUG::2012-12-02 23:50:41,936::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-7573::DEBUG::2012-12-02 23:50:41,936::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7573::DEBUG::2012-12-02 23:50:42,581::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-7573::DEBUG::2012-12-02 23:50:42,654::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-7573::DEBUG::2012-12-02 23:50:42,654::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7573::DEBUG::2012-12-02 23:50:42,654::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7573::DEBUG::2012-12-02 23:50:42,655::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7573::DEBUG::2012-12-02 23:50:42,655::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7573::DEBUG::2012-12-02 23:50:42,655::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-7573::DEBUG::2012-12-02 23:50:42,655::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-7573::DEBUG::2012-12-02 23:50:42,655::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-7573::INFO::2012-12-02 23:50:42,656::logUtils::39::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-7573::DEBUG::2012-12-02 23:50:42,656::task::1151::TaskManager.Task::(prepare) Task=`e5044e76-dcef-40f2-b15e-b42e21c0164f`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-7573::DEBUG::2012-12-02 23:50:42,656::task::568::TaskManager.Task::(_updateState) Task=`e5044e76-dcef-40f2-b15e-b42e21c0164f`::moving from state preparing -> state finished >Thread-7573::DEBUG::2012-12-02 23:50:42,656::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7573::DEBUG::2012-12-02 23:50:42,656::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7573::DEBUG::2012-12-02 23:50:42,656::task::957::TaskManager.Task::(_decref) Task=`e5044e76-dcef-40f2-b15e-b42e21c0164f`::ref 0 aborting False >Thread-7578::DEBUG::2012-12-02 23:50:51,719::task::568::TaskManager.Task::(_updateState) Task=`1fa7c4c0-4d3d-46b5-9ec7-c6931a59b2c0`::moving from state init -> state preparing >Thread-7578::INFO::2012-12-02 23:50:51,719::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7578::INFO::2012-12-02 23:50:51,719::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00908708572388', 'lastCheck': 1354485046.448374, 'code': 0, 'valid': True}} >Thread-7578::DEBUG::2012-12-02 23:50:51,720::task::1151::TaskManager.Task::(prepare) Task=`1fa7c4c0-4d3d-46b5-9ec7-c6931a59b2c0`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00908708572388', 'lastCheck': 1354485046.448374, 'code': 0, 'valid': True}} >Thread-7578::DEBUG::2012-12-02 23:50:51,720::task::568::TaskManager.Task::(_updateState) Task=`1fa7c4c0-4d3d-46b5-9ec7-c6931a59b2c0`::moving from state preparing -> state finished >Thread-7578::DEBUG::2012-12-02 23:50:51,720::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7578::DEBUG::2012-12-02 23:50:51,720::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7578::DEBUG::2012-12-02 23:50:51,720::task::957::TaskManager.Task::(_decref) Task=`1fa7c4c0-4d3d-46b5-9ec7-c6931a59b2c0`::ref 0 aborting False >Thread-7580::DEBUG::2012-12-02 23:50:51,939::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7580::DEBUG::2012-12-02 23:50:51,939::task::568::TaskManager.Task::(_updateState) Task=`13c2a27a-86b1-4995-a0e4-1f869dbd1291`::moving from state init -> state preparing >Thread-7580::INFO::2012-12-02 23:50:51,939::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7580::INFO::2012-12-02 23:50:51,940::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7580::DEBUG::2012-12-02 23:50:51,940::task::1151::TaskManager.Task::(prepare) Task=`13c2a27a-86b1-4995-a0e4-1f869dbd1291`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7580::DEBUG::2012-12-02 23:50:51,940::task::568::TaskManager.Task::(_updateState) Task=`13c2a27a-86b1-4995-a0e4-1f869dbd1291`::moving from state preparing -> state finished >Thread-7580::DEBUG::2012-12-02 23:50:51,940::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7580::DEBUG::2012-12-02 23:50:51,940::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7580::DEBUG::2012-12-02 23:50:51,940::task::957::TaskManager.Task::(_decref) Task=`13c2a27a-86b1-4995-a0e4-1f869dbd1291`::ref 0 aborting False >Thread-7581::DEBUG::2012-12-02 23:50:52,082::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7581::DEBUG::2012-12-02 23:50:52,083::task::568::TaskManager.Task::(_updateState) Task=`79abe014-3e1e-458a-ad8a-7068f14fe7d0`::moving from state init -> state preparing >Thread-7581::INFO::2012-12-02 23:50:52,083::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7581::DEBUG::2012-12-02 23:50:52,083::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`fc17012f-ebe8-4f6e-8052-25bad20398d5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7581::DEBUG::2012-12-02 23:50:52,083::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7581::DEBUG::2012-12-02 23:50:52,084::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7581::DEBUG::2012-12-02 23:50:52,084::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`fc17012f-ebe8-4f6e-8052-25bad20398d5`::Granted request >Thread-7581::DEBUG::2012-12-02 23:50:52,084::task::794::TaskManager.Task::(resourceAcquired) Task=`79abe014-3e1e-458a-ad8a-7068f14fe7d0`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7581::DEBUG::2012-12-02 23:50:52,084::task::957::TaskManager.Task::(_decref) Task=`79abe014-3e1e-458a-ad8a-7068f14fe7d0`::ref 1 aborting False >Thread-7581::DEBUG::2012-12-02 23:50:52,089::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7581::DEBUG::2012-12-02 23:50:52,090::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7581::DEBUG::2012-12-02 23:50:52,296::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7581::WARNING::2012-12-02 23:50:52,297::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7581::DEBUG::2012-12-02 23:50:52,298::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7581::WARNING::2012-12-02 23:50:52,327::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7581::DEBUG::2012-12-02 23:50:52,327::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7581::DEBUG::2012-12-02 23:50:52,328::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7581::DEBUG::2012-12-02 23:50:52,532::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7581::WARNING::2012-12-02 23:50:52,533::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7581::DEBUG::2012-12-02 23:50:52,533::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7581::WARNING::2012-12-02 23:50:52,562::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7581::INFO::2012-12-02 23:50:52,562::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7581::DEBUG::2012-12-02 23:50:52,563::task::1151::TaskManager.Task::(prepare) Task=`79abe014-3e1e-458a-ad8a-7068f14fe7d0`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7581::DEBUG::2012-12-02 23:50:52,563::task::568::TaskManager.Task::(_updateState) Task=`79abe014-3e1e-458a-ad8a-7068f14fe7d0`::moving from state preparing -> state finished >Thread-7581::DEBUG::2012-12-02 23:50:52,563::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7581::DEBUG::2012-12-02 23:50:52,563::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7581::DEBUG::2012-12-02 23:50:52,563::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7581::DEBUG::2012-12-02 23:50:52,563::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7581::DEBUG::2012-12-02 23:50:52,564::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7581::DEBUG::2012-12-02 23:50:52,564::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7581::DEBUG::2012-12-02 23:50:52,564::task::957::TaskManager.Task::(_decref) Task=`79abe014-3e1e-458a-ad8a-7068f14fe7d0`::ref 0 aborting False >Thread-7708::DEBUG::2012-12-02 23:51:02,670::task::568::TaskManager.Task::(_updateState) Task=`b87b3bbb-308e-4cca-a04c-0e066f0f45c1`::moving from state init -> state preparing >Thread-7708::INFO::2012-12-02 23:51:02,671::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7708::INFO::2012-12-02 23:51:02,671::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00834798812866', 'lastCheck': 1354485056.467536, 'code': 0, 'valid': True}} >Thread-7708::DEBUG::2012-12-02 23:51:02,671::task::1151::TaskManager.Task::(prepare) Task=`b87b3bbb-308e-4cca-a04c-0e066f0f45c1`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00834798812866', 'lastCheck': 1354485056.467536, 'code': 0, 'valid': True}} >Thread-7708::DEBUG::2012-12-02 23:51:02,671::task::568::TaskManager.Task::(_updateState) Task=`b87b3bbb-308e-4cca-a04c-0e066f0f45c1`::moving from state preparing -> state finished >Thread-7708::DEBUG::2012-12-02 23:51:02,671::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7708::DEBUG::2012-12-02 23:51:02,671::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7708::DEBUG::2012-12-02 23:51:02,671::task::957::TaskManager.Task::(_decref) Task=`b87b3bbb-308e-4cca-a04c-0e066f0f45c1`::ref 0 aborting False >Thread-7709::DEBUG::2012-12-02 23:51:02,773::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7709::DEBUG::2012-12-02 23:51:02,773::task::568::TaskManager.Task::(_updateState) Task=`25f59583-56f3-4647-bc1b-897f958615ab`::moving from state init -> state preparing >Thread-7709::INFO::2012-12-02 23:51:02,773::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7709::INFO::2012-12-02 23:51:02,773::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7709::DEBUG::2012-12-02 23:51:02,774::task::1151::TaskManager.Task::(prepare) Task=`25f59583-56f3-4647-bc1b-897f958615ab`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7709::DEBUG::2012-12-02 23:51:02,774::task::568::TaskManager.Task::(_updateState) Task=`25f59583-56f3-4647-bc1b-897f958615ab`::moving from state preparing -> state finished >Thread-7709::DEBUG::2012-12-02 23:51:02,774::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7709::DEBUG::2012-12-02 23:51:02,774::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7709::DEBUG::2012-12-02 23:51:02,774::task::957::TaskManager.Task::(_decref) Task=`25f59583-56f3-4647-bc1b-897f958615ab`::ref 0 aborting False >Thread-7711::DEBUG::2012-12-02 23:51:02,919::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7711::DEBUG::2012-12-02 23:51:02,920::task::568::TaskManager.Task::(_updateState) Task=`a20eff26-fdd1-451e-bb0e-aed2144d98be`::moving from state init -> state preparing >Thread-7711::INFO::2012-12-02 23:51:02,920::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7711::DEBUG::2012-12-02 23:51:02,920::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6b35329c-fbc1-45e1-a176-05452c0b1b42`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7711::DEBUG::2012-12-02 23:51:02,920::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7711::DEBUG::2012-12-02 23:51:02,921::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7711::DEBUG::2012-12-02 23:51:02,921::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`6b35329c-fbc1-45e1-a176-05452c0b1b42`::Granted request >Thread-7711::DEBUG::2012-12-02 23:51:02,921::task::794::TaskManager.Task::(resourceAcquired) Task=`a20eff26-fdd1-451e-bb0e-aed2144d98be`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7711::DEBUG::2012-12-02 23:51:02,921::task::957::TaskManager.Task::(_decref) Task=`a20eff26-fdd1-451e-bb0e-aed2144d98be`::ref 1 aborting False >Thread-7711::DEBUG::2012-12-02 23:51:02,927::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7711::DEBUG::2012-12-02 23:51:02,927::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7711::DEBUG::2012-12-02 23:51:03,134::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7711::WARNING::2012-12-02 23:51:03,135::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7711::DEBUG::2012-12-02 23:51:03,136::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7711::WARNING::2012-12-02 23:51:03,169::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7711::DEBUG::2012-12-02 23:51:03,170::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7711::DEBUG::2012-12-02 23:51:03,170::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7711::DEBUG::2012-12-02 23:51:03,377::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7711::WARNING::2012-12-02 23:51:03,379::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7711::DEBUG::2012-12-02 23:51:03,379::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7711::WARNING::2012-12-02 23:51:03,408::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7711::INFO::2012-12-02 23:51:03,409::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7711::DEBUG::2012-12-02 23:51:03,409::task::1151::TaskManager.Task::(prepare) Task=`a20eff26-fdd1-451e-bb0e-aed2144d98be`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7711::DEBUG::2012-12-02 23:51:03,409::task::568::TaskManager.Task::(_updateState) Task=`a20eff26-fdd1-451e-bb0e-aed2144d98be`::moving from state preparing -> state finished >Thread-7711::DEBUG::2012-12-02 23:51:03,409::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7711::DEBUG::2012-12-02 23:51:03,409::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7711::DEBUG::2012-12-02 23:51:03,410::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7711::DEBUG::2012-12-02 23:51:03,410::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7711::DEBUG::2012-12-02 23:51:03,410::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7711::DEBUG::2012-12-02 23:51:03,410::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7711::DEBUG::2012-12-02 23:51:03,410::task::957::TaskManager.Task::(_decref) Task=`a20eff26-fdd1-451e-bb0e-aed2144d98be`::ref 0 aborting False >Thread-7838::DEBUG::2012-12-02 23:51:13,718::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7838::DEBUG::2012-12-02 23:51:13,718::task::568::TaskManager.Task::(_updateState) Task=`05b71b0f-578d-4c9a-a6f5-297b977792fb`::moving from state init -> state preparing >Thread-7838::INFO::2012-12-02 23:51:13,718::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7838::INFO::2012-12-02 23:51:13,719::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7838::DEBUG::2012-12-02 23:51:13,719::task::1151::TaskManager.Task::(prepare) Task=`05b71b0f-578d-4c9a-a6f5-297b977792fb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7838::DEBUG::2012-12-02 23:51:13,719::task::568::TaskManager.Task::(_updateState) Task=`05b71b0f-578d-4c9a-a6f5-297b977792fb`::moving from state preparing -> state finished >Thread-7838::DEBUG::2012-12-02 23:51:13,719::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7838::DEBUG::2012-12-02 23:51:13,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7838::DEBUG::2012-12-02 23:51:13,719::task::957::TaskManager.Task::(_decref) Task=`05b71b0f-578d-4c9a-a6f5-297b977792fb`::ref 0 aborting False >Thread-7839::DEBUG::2012-12-02 23:51:13,770::task::568::TaskManager.Task::(_updateState) Task=`27fabb77-5a44-498d-a508-72809ac92446`::moving from state init -> state preparing >Thread-7839::INFO::2012-12-02 23:51:13,770::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7839::INFO::2012-12-02 23:51:13,770::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00842094421387', 'lastCheck': 1354485066.4866049, 'code': 0, 'valid': True}} >Thread-7839::DEBUG::2012-12-02 23:51:13,771::task::1151::TaskManager.Task::(prepare) Task=`27fabb77-5a44-498d-a508-72809ac92446`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00842094421387', 'lastCheck': 1354485066.4866049, 'code': 0, 'valid': True}} >Thread-7839::DEBUG::2012-12-02 23:51:13,771::task::568::TaskManager.Task::(_updateState) Task=`27fabb77-5a44-498d-a508-72809ac92446`::moving from state preparing -> state finished >Thread-7839::DEBUG::2012-12-02 23:51:13,771::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7839::DEBUG::2012-12-02 23:51:13,771::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7839::DEBUG::2012-12-02 23:51:13,771::task::957::TaskManager.Task::(_decref) Task=`27fabb77-5a44-498d-a508-72809ac92446`::ref 0 aborting False >Thread-7840::DEBUG::2012-12-02 23:51:13,914::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7840::DEBUG::2012-12-02 23:51:13,915::task::568::TaskManager.Task::(_updateState) Task=`0d5b28a8-9d24-48ad-85f9-ce6fc3a8e9e8`::moving from state init -> state preparing >Thread-7840::INFO::2012-12-02 23:51:13,915::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7840::DEBUG::2012-12-02 23:51:13,915::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`5c0f7ff1-cb82-4af9-822b-6f92749addbf`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7840::DEBUG::2012-12-02 23:51:13,915::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7840::DEBUG::2012-12-02 23:51:13,916::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7840::DEBUG::2012-12-02 23:51:13,916::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`5c0f7ff1-cb82-4af9-822b-6f92749addbf`::Granted request >Thread-7840::DEBUG::2012-12-02 23:51:13,916::task::794::TaskManager.Task::(resourceAcquired) Task=`0d5b28a8-9d24-48ad-85f9-ce6fc3a8e9e8`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7840::DEBUG::2012-12-02 23:51:13,916::task::957::TaskManager.Task::(_decref) Task=`0d5b28a8-9d24-48ad-85f9-ce6fc3a8e9e8`::ref 1 aborting False >Thread-7840::DEBUG::2012-12-02 23:51:13,923::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7840::DEBUG::2012-12-02 23:51:13,923::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7840::DEBUG::2012-12-02 23:51:14,130::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7840::WARNING::2012-12-02 23:51:14,131::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7840::DEBUG::2012-12-02 23:51:14,131::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7840::WARNING::2012-12-02 23:51:14,160::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7840::DEBUG::2012-12-02 23:51:14,161::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7840::DEBUG::2012-12-02 23:51:14,161::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7840::DEBUG::2012-12-02 23:51:14,366::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7840::WARNING::2012-12-02 23:51:14,367::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7840::DEBUG::2012-12-02 23:51:14,367::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7840::WARNING::2012-12-02 23:51:14,397::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7840::INFO::2012-12-02 23:51:14,398::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7840::DEBUG::2012-12-02 23:51:14,398::task::1151::TaskManager.Task::(prepare) Task=`0d5b28a8-9d24-48ad-85f9-ce6fc3a8e9e8`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7840::DEBUG::2012-12-02 23:51:14,398::task::568::TaskManager.Task::(_updateState) Task=`0d5b28a8-9d24-48ad-85f9-ce6fc3a8e9e8`::moving from state preparing -> state finished >Thread-7840::DEBUG::2012-12-02 23:51:14,398::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7840::DEBUG::2012-12-02 23:51:14,399::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7840::DEBUG::2012-12-02 23:51:14,399::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7840::DEBUG::2012-12-02 23:51:14,399::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7840::DEBUG::2012-12-02 23:51:14,399::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7840::DEBUG::2012-12-02 23:51:14,399::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7840::DEBUG::2012-12-02 23:51:14,399::task::957::TaskManager.Task::(_decref) Task=`0d5b28a8-9d24-48ad-85f9-ce6fc3a8e9e8`::ref 0 aborting False >Thread-7968::DEBUG::2012-12-02 23:51:24,617::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7968::DEBUG::2012-12-02 23:51:24,618::task::568::TaskManager.Task::(_updateState) Task=`b985cf8d-f336-4ceb-aa5e-2a2600465f13`::moving from state init -> state preparing >Thread-7968::INFO::2012-12-02 23:51:24,618::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7968::INFO::2012-12-02 23:51:24,618::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7968::DEBUG::2012-12-02 23:51:24,618::task::1151::TaskManager.Task::(prepare) Task=`b985cf8d-f336-4ceb-aa5e-2a2600465f13`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-7968::DEBUG::2012-12-02 23:51:24,618::task::568::TaskManager.Task::(_updateState) Task=`b985cf8d-f336-4ceb-aa5e-2a2600465f13`::moving from state preparing -> state finished >Thread-7968::DEBUG::2012-12-02 23:51:24,618::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7968::DEBUG::2012-12-02 23:51:24,619::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7968::DEBUG::2012-12-02 23:51:24,619::task::957::TaskManager.Task::(_decref) Task=`b985cf8d-f336-4ceb-aa5e-2a2600465f13`::ref 0 aborting False >Thread-7969::DEBUG::2012-12-02 23:51:24,756::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-7969::DEBUG::2012-12-02 23:51:24,756::task::568::TaskManager.Task::(_updateState) Task=`9a806b80-9fb6-4baf-a04e-6b0523e520c3`::moving from state init -> state preparing >Thread-7969::INFO::2012-12-02 23:51:24,756::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-7969::DEBUG::2012-12-02 23:51:24,757::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f154fb4f-57d9-4b1d-b83f-939f075e6df8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-7969::DEBUG::2012-12-02 23:51:24,757::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-7969::DEBUG::2012-12-02 23:51:24,757::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-7969::DEBUG::2012-12-02 23:51:24,757::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f154fb4f-57d9-4b1d-b83f-939f075e6df8`::Granted request >Thread-7969::DEBUG::2012-12-02 23:51:24,757::task::794::TaskManager.Task::(resourceAcquired) Task=`9a806b80-9fb6-4baf-a04e-6b0523e520c3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-7969::DEBUG::2012-12-02 23:51:24,758::task::957::TaskManager.Task::(_decref) Task=`9a806b80-9fb6-4baf-a04e-6b0523e520c3`::ref 1 aborting False >Thread-7969::DEBUG::2012-12-02 23:51:24,764::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7969::DEBUG::2012-12-02 23:51:24,765::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-7970::DEBUG::2012-12-02 23:51:24,769::task::568::TaskManager.Task::(_updateState) Task=`0070dbc6-1a55-4fc5-a055-197aa465927b`::moving from state init -> state preparing >Thread-7970::INFO::2012-12-02 23:51:24,769::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-7970::INFO::2012-12-02 23:51:24,769::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00901794433594', 'lastCheck': 1354485076.5081389, 'code': 0, 'valid': True}} >Thread-7970::DEBUG::2012-12-02 23:51:24,770::task::1151::TaskManager.Task::(prepare) Task=`0070dbc6-1a55-4fc5-a055-197aa465927b`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00901794433594', 'lastCheck': 1354485076.5081389, 'code': 0, 'valid': True}} >Thread-7970::DEBUG::2012-12-02 23:51:24,770::task::568::TaskManager.Task::(_updateState) Task=`0070dbc6-1a55-4fc5-a055-197aa465927b`::moving from state preparing -> state finished >Thread-7970::DEBUG::2012-12-02 23:51:24,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-7970::DEBUG::2012-12-02 23:51:24,770::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7970::DEBUG::2012-12-02 23:51:24,770::task::957::TaskManager.Task::(_decref) Task=`0070dbc6-1a55-4fc5-a055-197aa465927b`::ref 0 aborting False >Thread-7969::DEBUG::2012-12-02 23:51:24,969::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-7969::WARNING::2012-12-02 23:51:24,970::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-7969::DEBUG::2012-12-02 23:51:24,971::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7969::WARNING::2012-12-02 23:51:25,000::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-7969::DEBUG::2012-12-02 23:51:25,000::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-7969::DEBUG::2012-12-02 23:51:25,001::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-7969::DEBUG::2012-12-02 23:51:25,206::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-7969::WARNING::2012-12-02 23:51:25,207::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-7969::DEBUG::2012-12-02 23:51:25,208::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7969::WARNING::2012-12-02 23:51:25,237::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-7969::INFO::2012-12-02 23:51:25,238::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7969::DEBUG::2012-12-02 23:51:25,238::task::1151::TaskManager.Task::(prepare) Task=`9a806b80-9fb6-4baf-a04e-6b0523e520c3`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-7969::DEBUG::2012-12-02 23:51:25,238::task::568::TaskManager.Task::(_updateState) Task=`9a806b80-9fb6-4baf-a04e-6b0523e520c3`::moving from state preparing -> state finished >Thread-7969::DEBUG::2012-12-02 23:51:25,238::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-7969::DEBUG::2012-12-02 23:51:25,238::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-7969::DEBUG::2012-12-02 23:51:25,239::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-7969::DEBUG::2012-12-02 23:51:25,239::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-7969::DEBUG::2012-12-02 23:51:25,239::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-7969::DEBUG::2012-12-02 23:51:25,239::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-7969::DEBUG::2012-12-02 23:51:25,239::task::957::TaskManager.Task::(_decref) Task=`9a806b80-9fb6-4baf-a04e-6b0523e520c3`::ref 0 aborting False >Thread-8098::DEBUG::2012-12-02 23:51:35,509::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8098::DEBUG::2012-12-02 23:51:35,509::task::568::TaskManager.Task::(_updateState) Task=`dc8caa9b-9fb6-4ba8-b044-e33376389ee7`::moving from state init -> state preparing >Thread-8098::INFO::2012-12-02 23:51:35,509::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8098::INFO::2012-12-02 23:51:35,510::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8098::DEBUG::2012-12-02 23:51:35,510::task::1151::TaskManager.Task::(prepare) Task=`dc8caa9b-9fb6-4ba8-b044-e33376389ee7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8098::DEBUG::2012-12-02 23:51:35,510::task::568::TaskManager.Task::(_updateState) Task=`dc8caa9b-9fb6-4ba8-b044-e33376389ee7`::moving from state preparing -> state finished >Thread-8098::DEBUG::2012-12-02 23:51:35,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8098::DEBUG::2012-12-02 23:51:35,510::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8098::DEBUG::2012-12-02 23:51:35,510::task::957::TaskManager.Task::(_decref) Task=`dc8caa9b-9fb6-4ba8-b044-e33376389ee7`::ref 0 aborting False >Thread-8099::DEBUG::2012-12-02 23:51:35,652::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8099::DEBUG::2012-12-02 23:51:35,652::task::568::TaskManager.Task::(_updateState) Task=`4b0a5052-7b2b-4e40-8f7d-9072494c067a`::moving from state init -> state preparing >Thread-8099::INFO::2012-12-02 23:51:35,652::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8099::DEBUG::2012-12-02 23:51:35,653::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`57f9559b-f647-4e49-ae04-2e6e2db562fa`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8099::DEBUG::2012-12-02 23:51:35,653::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8099::DEBUG::2012-12-02 23:51:35,653::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8099::DEBUG::2012-12-02 23:51:35,653::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`57f9559b-f647-4e49-ae04-2e6e2db562fa`::Granted request >Thread-8099::DEBUG::2012-12-02 23:51:35,653::task::794::TaskManager.Task::(resourceAcquired) Task=`4b0a5052-7b2b-4e40-8f7d-9072494c067a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8099::DEBUG::2012-12-02 23:51:35,654::task::957::TaskManager.Task::(_decref) Task=`4b0a5052-7b2b-4e40-8f7d-9072494c067a`::ref 1 aborting False >Thread-8099::DEBUG::2012-12-02 23:51:35,658::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8099::DEBUG::2012-12-02 23:51:35,659::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8100::DEBUG::2012-12-02 23:51:35,692::task::568::TaskManager.Task::(_updateState) Task=`1d4348b2-1be4-43f8-a87e-e42203548828`::moving from state init -> state preparing >Thread-8100::INFO::2012-12-02 23:51:35,692::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8100::INFO::2012-12-02 23:51:35,693::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00746583938599', 'lastCheck': 1354485086.525743, 'code': 0, 'valid': True}} >Thread-8100::DEBUG::2012-12-02 23:51:35,693::task::1151::TaskManager.Task::(prepare) Task=`1d4348b2-1be4-43f8-a87e-e42203548828`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00746583938599', 'lastCheck': 1354485086.525743, 'code': 0, 'valid': True}} >Thread-8100::DEBUG::2012-12-02 23:51:35,693::task::568::TaskManager.Task::(_updateState) Task=`1d4348b2-1be4-43f8-a87e-e42203548828`::moving from state preparing -> state finished >Thread-8100::DEBUG::2012-12-02 23:51:35,693::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8100::DEBUG::2012-12-02 23:51:35,693::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8100::DEBUG::2012-12-02 23:51:35,693::task::957::TaskManager.Task::(_decref) Task=`1d4348b2-1be4-43f8-a87e-e42203548828`::ref 0 aborting False >Thread-8099::DEBUG::2012-12-02 23:51:35,863::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8099::WARNING::2012-12-02 23:51:35,865::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8099::DEBUG::2012-12-02 23:51:35,865::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8099::WARNING::2012-12-02 23:51:35,895::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8099::DEBUG::2012-12-02 23:51:35,895::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8099::DEBUG::2012-12-02 23:51:35,896::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8099::DEBUG::2012-12-02 23:51:36,103::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8099::WARNING::2012-12-02 23:51:36,104::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8099::DEBUG::2012-12-02 23:51:36,104::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8099::WARNING::2012-12-02 23:51:36,132::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8099::INFO::2012-12-02 23:51:36,133::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8099::DEBUG::2012-12-02 23:51:36,133::task::1151::TaskManager.Task::(prepare) Task=`4b0a5052-7b2b-4e40-8f7d-9072494c067a`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8099::DEBUG::2012-12-02 23:51:36,133::task::568::TaskManager.Task::(_updateState) Task=`4b0a5052-7b2b-4e40-8f7d-9072494c067a`::moving from state preparing -> state finished >Thread-8099::DEBUG::2012-12-02 23:51:36,133::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8099::DEBUG::2012-12-02 23:51:36,134::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8099::DEBUG::2012-12-02 23:51:36,134::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8099::DEBUG::2012-12-02 23:51:36,134::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8099::DEBUG::2012-12-02 23:51:36,134::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8099::DEBUG::2012-12-02 23:51:36,134::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8099::DEBUG::2012-12-02 23:51:36,134::task::957::TaskManager.Task::(_decref) Task=`4b0a5052-7b2b-4e40-8f7d-9072494c067a`::ref 0 aborting False >Thread-8228::DEBUG::2012-12-02 23:51:46,352::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8228::DEBUG::2012-12-02 23:51:46,352::task::568::TaskManager.Task::(_updateState) Task=`aac0f127-d66e-423a-b699-43958f4f2009`::moving from state init -> state preparing >Thread-8228::INFO::2012-12-02 23:51:46,352::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8228::INFO::2012-12-02 23:51:46,352::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8228::DEBUG::2012-12-02 23:51:46,353::task::1151::TaskManager.Task::(prepare) Task=`aac0f127-d66e-423a-b699-43958f4f2009`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8228::DEBUG::2012-12-02 23:51:46,353::task::568::TaskManager.Task::(_updateState) Task=`aac0f127-d66e-423a-b699-43958f4f2009`::moving from state preparing -> state finished >Thread-8228::DEBUG::2012-12-02 23:51:46,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8228::DEBUG::2012-12-02 23:51:46,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8228::DEBUG::2012-12-02 23:51:46,353::task::957::TaskManager.Task::(_decref) Task=`aac0f127-d66e-423a-b699-43958f4f2009`::ref 0 aborting False >Thread-8229::DEBUG::2012-12-02 23:51:46,493::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8229::DEBUG::2012-12-02 23:51:46,493::task::568::TaskManager.Task::(_updateState) Task=`43b99927-6820-49ee-be48-b21243d2486e`::moving from state init -> state preparing >Thread-8229::INFO::2012-12-02 23:51:46,493::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8229::DEBUG::2012-12-02 23:51:46,493::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ce299da4-d723-4243-b223-9eb0e1595848`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8229::DEBUG::2012-12-02 23:51:46,494::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8229::DEBUG::2012-12-02 23:51:46,494::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8229::DEBUG::2012-12-02 23:51:46,494::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ce299da4-d723-4243-b223-9eb0e1595848`::Granted request >Thread-8229::DEBUG::2012-12-02 23:51:46,494::task::794::TaskManager.Task::(resourceAcquired) Task=`43b99927-6820-49ee-be48-b21243d2486e`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8229::DEBUG::2012-12-02 23:51:46,494::task::957::TaskManager.Task::(_decref) Task=`43b99927-6820-49ee-be48-b21243d2486e`::ref 1 aborting False >Thread-8229::DEBUG::2012-12-02 23:51:46,499::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8229::DEBUG::2012-12-02 23:51:46,499::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8230::DEBUG::2012-12-02 23:51:46,701::task::568::TaskManager.Task::(_updateState) Task=`73e8f20b-19c3-4907-bf3c-a86a0bbd9834`::moving from state init -> state preparing >Thread-8230::INFO::2012-12-02 23:51:46,702::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8230::INFO::2012-12-02 23:51:46,702::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00689911842346', 'lastCheck': 1354485096.542563, 'code': 0, 'valid': True}} >Thread-8230::DEBUG::2012-12-02 23:51:46,702::task::1151::TaskManager.Task::(prepare) Task=`73e8f20b-19c3-4907-bf3c-a86a0bbd9834`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00689911842346', 'lastCheck': 1354485096.542563, 'code': 0, 'valid': True}} >Thread-8230::DEBUG::2012-12-02 23:51:46,702::task::568::TaskManager.Task::(_updateState) Task=`73e8f20b-19c3-4907-bf3c-a86a0bbd9834`::moving from state preparing -> state finished >Thread-8230::DEBUG::2012-12-02 23:51:46,702::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8230::DEBUG::2012-12-02 23:51:46,702::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8230::DEBUG::2012-12-02 23:51:46,702::task::957::TaskManager.Task::(_decref) Task=`73e8f20b-19c3-4907-bf3c-a86a0bbd9834`::ref 0 aborting False >Thread-8229::DEBUG::2012-12-02 23:51:46,708::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8229::WARNING::2012-12-02 23:51:46,709::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8229::DEBUG::2012-12-02 23:51:46,710::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8229::WARNING::2012-12-02 23:51:46,740::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8229::DEBUG::2012-12-02 23:51:46,740::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8229::DEBUG::2012-12-02 23:51:46,742::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8229::DEBUG::2012-12-02 23:51:46,947::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8229::WARNING::2012-12-02 23:51:46,949::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8229::DEBUG::2012-12-02 23:51:46,949::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8229::WARNING::2012-12-02 23:51:46,978::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8229::INFO::2012-12-02 23:51:46,979::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8229::DEBUG::2012-12-02 23:51:46,979::task::1151::TaskManager.Task::(prepare) Task=`43b99927-6820-49ee-be48-b21243d2486e`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8229::DEBUG::2012-12-02 23:51:46,979::task::568::TaskManager.Task::(_updateState) Task=`43b99927-6820-49ee-be48-b21243d2486e`::moving from state preparing -> state finished >Thread-8229::DEBUG::2012-12-02 23:51:46,980::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8229::DEBUG::2012-12-02 23:51:46,980::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8229::DEBUG::2012-12-02 23:51:46,980::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8229::DEBUG::2012-12-02 23:51:46,980::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8229::DEBUG::2012-12-02 23:51:46,980::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8229::DEBUG::2012-12-02 23:51:46,980::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8229::DEBUG::2012-12-02 23:51:46,981::task::957::TaskManager.Task::(_decref) Task=`43b99927-6820-49ee-be48-b21243d2486e`::ref 0 aborting False >Thread-8358::DEBUG::2012-12-02 23:51:57,185::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8358::DEBUG::2012-12-02 23:51:57,185::task::568::TaskManager.Task::(_updateState) Task=`2171901c-7f53-4baf-ac85-6839fa986568`::moving from state init -> state preparing >Thread-8358::INFO::2012-12-02 23:51:57,185::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8358::INFO::2012-12-02 23:51:57,186::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8358::DEBUG::2012-12-02 23:51:57,186::task::1151::TaskManager.Task::(prepare) Task=`2171901c-7f53-4baf-ac85-6839fa986568`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8358::DEBUG::2012-12-02 23:51:57,186::task::568::TaskManager.Task::(_updateState) Task=`2171901c-7f53-4baf-ac85-6839fa986568`::moving from state preparing -> state finished >Thread-8358::DEBUG::2012-12-02 23:51:57,186::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8358::DEBUG::2012-12-02 23:51:57,186::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8358::DEBUG::2012-12-02 23:51:57,186::task::957::TaskManager.Task::(_decref) Task=`2171901c-7f53-4baf-ac85-6839fa986568`::ref 0 aborting False >Thread-8359::DEBUG::2012-12-02 23:51:57,333::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8359::DEBUG::2012-12-02 23:51:57,334::task::568::TaskManager.Task::(_updateState) Task=`ae52dbee-e721-4870-82e1-09f01dda0147`::moving from state init -> state preparing >Thread-8359::INFO::2012-12-02 23:51:57,334::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8359::DEBUG::2012-12-02 23:51:57,334::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7d60419f-2e1f-40c9-9a45-8cc40e28c3aa`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8359::DEBUG::2012-12-02 23:51:57,334::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8359::DEBUG::2012-12-02 23:51:57,335::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8359::DEBUG::2012-12-02 23:51:57,335::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7d60419f-2e1f-40c9-9a45-8cc40e28c3aa`::Granted request >Thread-8359::DEBUG::2012-12-02 23:51:57,335::task::794::TaskManager.Task::(resourceAcquired) Task=`ae52dbee-e721-4870-82e1-09f01dda0147`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8359::DEBUG::2012-12-02 23:51:57,335::task::957::TaskManager.Task::(_decref) Task=`ae52dbee-e721-4870-82e1-09f01dda0147`::ref 1 aborting False >Thread-8359::DEBUG::2012-12-02 23:51:57,339::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8359::DEBUG::2012-12-02 23:51:57,340::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8359::DEBUG::2012-12-02 23:51:57,544::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8359::WARNING::2012-12-02 23:51:57,546::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8359::DEBUG::2012-12-02 23:51:57,546::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8359::WARNING::2012-12-02 23:51:57,575::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8359::DEBUG::2012-12-02 23:51:57,576::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8359::DEBUG::2012-12-02 23:51:57,576::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8421::DEBUG::2012-12-02 23:51:57,648::task::568::TaskManager.Task::(_updateState) Task=`4d7ee098-f446-4cc3-83b0-0c26398717f3`::moving from state init -> state preparing >Thread-8421::INFO::2012-12-02 23:51:57,648::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8421::INFO::2012-12-02 23:51:57,648::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00362801551819', 'lastCheck': 1354485116.997824, 'code': 0, 'valid': True}} >Thread-8421::DEBUG::2012-12-02 23:51:57,648::task::1151::TaskManager.Task::(prepare) Task=`4d7ee098-f446-4cc3-83b0-0c26398717f3`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00362801551819', 'lastCheck': 1354485116.997824, 'code': 0, 'valid': True}} >Thread-8421::DEBUG::2012-12-02 23:51:57,649::task::568::TaskManager.Task::(_updateState) Task=`4d7ee098-f446-4cc3-83b0-0c26398717f3`::moving from state preparing -> state finished >Thread-8421::DEBUG::2012-12-02 23:51:57,649::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8421::DEBUG::2012-12-02 23:51:57,649::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8421::DEBUG::2012-12-02 23:51:57,649::task::957::TaskManager.Task::(_decref) Task=`4d7ee098-f446-4cc3-83b0-0c26398717f3`::ref 0 aborting False >Thread-8359::DEBUG::2012-12-02 23:51:57,782::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8359::WARNING::2012-12-02 23:51:57,784::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8359::DEBUG::2012-12-02 23:51:57,784::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8359::WARNING::2012-12-02 23:51:57,814::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8359::INFO::2012-12-02 23:51:57,814::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8359::DEBUG::2012-12-02 23:51:57,815::task::1151::TaskManager.Task::(prepare) Task=`ae52dbee-e721-4870-82e1-09f01dda0147`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8359::DEBUG::2012-12-02 23:51:57,815::task::568::TaskManager.Task::(_updateState) Task=`ae52dbee-e721-4870-82e1-09f01dda0147`::moving from state preparing -> state finished >Thread-8359::DEBUG::2012-12-02 23:51:57,815::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8359::DEBUG::2012-12-02 23:51:57,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8359::DEBUG::2012-12-02 23:51:57,815::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8359::DEBUG::2012-12-02 23:51:57,815::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8359::DEBUG::2012-12-02 23:51:57,816::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8359::DEBUG::2012-12-02 23:51:57,816::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8359::DEBUG::2012-12-02 23:51:57,816::task::957::TaskManager.Task::(_decref) Task=`ae52dbee-e721-4870-82e1-09f01dda0147`::ref 0 aborting False >Thread-8488::DEBUG::2012-12-02 23:52:08,038::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8488::DEBUG::2012-12-02 23:52:08,039::task::568::TaskManager.Task::(_updateState) Task=`b4e5ffcd-07b8-4747-b245-cf679fe635c1`::moving from state init -> state preparing >Thread-8488::INFO::2012-12-02 23:52:08,039::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8488::INFO::2012-12-02 23:52:08,039::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8488::DEBUG::2012-12-02 23:52:08,039::task::1151::TaskManager.Task::(prepare) Task=`b4e5ffcd-07b8-4747-b245-cf679fe635c1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8488::DEBUG::2012-12-02 23:52:08,040::task::568::TaskManager.Task::(_updateState) Task=`b4e5ffcd-07b8-4747-b245-cf679fe635c1`::moving from state preparing -> state finished >Thread-8488::DEBUG::2012-12-02 23:52:08,040::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8488::DEBUG::2012-12-02 23:52:08,040::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8488::DEBUG::2012-12-02 23:52:08,040::task::957::TaskManager.Task::(_decref) Task=`b4e5ffcd-07b8-4747-b245-cf679fe635c1`::ref 0 aborting False >Thread-8489::DEBUG::2012-12-02 23:52:08,191::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8489::DEBUG::2012-12-02 23:52:08,192::task::568::TaskManager.Task::(_updateState) Task=`7f1b5fdd-55d6-4ba9-8727-aff9617a8f27`::moving from state init -> state preparing >Thread-8489::INFO::2012-12-02 23:52:08,192::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8489::DEBUG::2012-12-02 23:52:08,192::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`28378378-3ae3-4bbc-8d8d-9cb843e861bb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8489::DEBUG::2012-12-02 23:52:08,192::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8489::DEBUG::2012-12-02 23:52:08,193::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8489::DEBUG::2012-12-02 23:52:08,193::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`28378378-3ae3-4bbc-8d8d-9cb843e861bb`::Granted request >Thread-8489::DEBUG::2012-12-02 23:52:08,193::task::794::TaskManager.Task::(resourceAcquired) Task=`7f1b5fdd-55d6-4ba9-8727-aff9617a8f27`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8489::DEBUG::2012-12-02 23:52:08,193::task::957::TaskManager.Task::(_decref) Task=`7f1b5fdd-55d6-4ba9-8727-aff9617a8f27`::ref 1 aborting False >Thread-8489::DEBUG::2012-12-02 23:52:08,198::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8489::DEBUG::2012-12-02 23:52:08,198::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8489::DEBUG::2012-12-02 23:52:08,403::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8489::WARNING::2012-12-02 23:52:08,405::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8489::DEBUG::2012-12-02 23:52:08,405::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8489::WARNING::2012-12-02 23:52:08,440::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8489::DEBUG::2012-12-02 23:52:08,440::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8489::DEBUG::2012-12-02 23:52:08,441::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8551::DEBUG::2012-12-02 23:52:08,599::task::568::TaskManager.Task::(_updateState) Task=`5feb778d-77bc-44b0-ab43-02b365e00e1b`::moving from state init -> state preparing >Thread-8551::INFO::2012-12-02 23:52:08,599::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8551::INFO::2012-12-02 23:52:08,599::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00697112083435', 'lastCheck': 1354485127.015882, 'code': 0, 'valid': True}} >Thread-8551::DEBUG::2012-12-02 23:52:08,599::task::1151::TaskManager.Task::(prepare) Task=`5feb778d-77bc-44b0-ab43-02b365e00e1b`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00697112083435', 'lastCheck': 1354485127.015882, 'code': 0, 'valid': True}} >Thread-8551::DEBUG::2012-12-02 23:52:08,599::task::568::TaskManager.Task::(_updateState) Task=`5feb778d-77bc-44b0-ab43-02b365e00e1b`::moving from state preparing -> state finished >Thread-8551::DEBUG::2012-12-02 23:52:08,599::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8551::DEBUG::2012-12-02 23:52:08,599::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8551::DEBUG::2012-12-02 23:52:08,599::task::957::TaskManager.Task::(_decref) Task=`5feb778d-77bc-44b0-ab43-02b365e00e1b`::ref 0 aborting False >Thread-8489::DEBUG::2012-12-02 23:52:08,647::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8489::WARNING::2012-12-02 23:52:08,649::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8489::DEBUG::2012-12-02 23:52:08,649::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8489::WARNING::2012-12-02 23:52:08,678::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8489::INFO::2012-12-02 23:52:08,678::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8489::DEBUG::2012-12-02 23:52:08,679::task::1151::TaskManager.Task::(prepare) Task=`7f1b5fdd-55d6-4ba9-8727-aff9617a8f27`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8489::DEBUG::2012-12-02 23:52:08,679::task::568::TaskManager.Task::(_updateState) Task=`7f1b5fdd-55d6-4ba9-8727-aff9617a8f27`::moving from state preparing -> state finished >Thread-8489::DEBUG::2012-12-02 23:52:08,679::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8489::DEBUG::2012-12-02 23:52:08,679::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8489::DEBUG::2012-12-02 23:52:08,679::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8489::DEBUG::2012-12-02 23:52:08,679::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8489::DEBUG::2012-12-02 23:52:08,680::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8489::DEBUG::2012-12-02 23:52:08,680::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8489::DEBUG::2012-12-02 23:52:08,680::task::957::TaskManager.Task::(_decref) Task=`7f1b5fdd-55d6-4ba9-8727-aff9617a8f27`::ref 0 aborting False >Thread-2443::DEBUG::2012-12-02 23:52:17,016::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain 5971512c-ee51-475e-b956-3872c38ce193 >Thread-2443::DEBUG::2012-12-02 23:52:17,016::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2443::DEBUG::2012-12-02 23:52:17,016::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-2443::DEBUG::2012-12-02 23:52:17,219::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-2443::WARNING::2012-12-02 23:52:17,220::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-2443::DEBUG::2012-12-02 23:52:17,221::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2443::DEBUG::2012-12-02 23:52:17,255::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193 >Thread-2443::DEBUG::2012-12-02 23:52:17,255::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-2443::DEBUG::2012-12-02 23:52:17,264::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=aaa', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=3', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'ROLE=Master', 'SDUUID=5971512c-ee51-475e-b956-3872c38ce193', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=3f2e0d7f41d7f402d78a822aa720dc6b9e57d85b'] >Thread-2443::DEBUG::2012-12-02 23:52:17,267::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-2443::WARNING::2012-12-02 23:52:17,267::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_imageNS already registered >Thread-2443::WARNING::2012-12-02 23:52:17,267::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 5971512c-ee51-475e-b956-3872c38ce193_volumeNS already registered >Thread-8679::DEBUG::2012-12-02 23:52:18,907::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8679::DEBUG::2012-12-02 23:52:18,907::task::568::TaskManager.Task::(_updateState) Task=`42bc1ed5-654c-4f09-8379-45fa52bf61b9`::moving from state init -> state preparing >Thread-8679::INFO::2012-12-02 23:52:18,908::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8679::INFO::2012-12-02 23:52:18,908::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8679::DEBUG::2012-12-02 23:52:18,908::task::1151::TaskManager.Task::(prepare) Task=`42bc1ed5-654c-4f09-8379-45fa52bf61b9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8679::DEBUG::2012-12-02 23:52:18,908::task::568::TaskManager.Task::(_updateState) Task=`42bc1ed5-654c-4f09-8379-45fa52bf61b9`::moving from state preparing -> state finished >Thread-8679::DEBUG::2012-12-02 23:52:18,908::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8679::DEBUG::2012-12-02 23:52:18,908::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8679::DEBUG::2012-12-02 23:52:18,909::task::957::TaskManager.Task::(_decref) Task=`42bc1ed5-654c-4f09-8379-45fa52bf61b9`::ref 0 aborting False >Thread-8680::DEBUG::2012-12-02 23:52:19,052::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8680::DEBUG::2012-12-02 23:52:19,052::task::568::TaskManager.Task::(_updateState) Task=`62d07444-2643-4431-a543-46790a5ecf4c`::moving from state init -> state preparing >Thread-8680::INFO::2012-12-02 23:52:19,052::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8680::DEBUG::2012-12-02 23:52:19,053::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1b8daf3c-688c-491b-9c77-af4e2c6abcd8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8680::DEBUG::2012-12-02 23:52:19,053::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8680::DEBUG::2012-12-02 23:52:19,053::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8680::DEBUG::2012-12-02 23:52:19,053::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1b8daf3c-688c-491b-9c77-af4e2c6abcd8`::Granted request >Thread-8680::DEBUG::2012-12-02 23:52:19,053::task::794::TaskManager.Task::(resourceAcquired) Task=`62d07444-2643-4431-a543-46790a5ecf4c`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8680::DEBUG::2012-12-02 23:52:19,053::task::957::TaskManager.Task::(_decref) Task=`62d07444-2643-4431-a543-46790a5ecf4c`::ref 1 aborting False >Thread-8680::DEBUG::2012-12-02 23:52:19,058::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8680::DEBUG::2012-12-02 23:52:19,058::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8680::DEBUG::2012-12-02 23:52:19,264::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8680::WARNING::2012-12-02 23:52:19,265::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8680::DEBUG::2012-12-02 23:52:19,265::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8680::WARNING::2012-12-02 23:52:19,297::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8680::DEBUG::2012-12-02 23:52:19,297::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8680::DEBUG::2012-12-02 23:52:19,298::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8680::DEBUG::2012-12-02 23:52:19,504::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8680::WARNING::2012-12-02 23:52:19,505::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8680::DEBUG::2012-12-02 23:52:19,505::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8788::DEBUG::2012-12-02 23:52:19,529::task::568::TaskManager.Task::(_updateState) Task=`f21c3001-1a1d-4d00-86e4-f3df3a03b4f5`::moving from state init -> state preparing >Thread-8788::INFO::2012-12-02 23:52:19,529::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8788::INFO::2012-12-02 23:52:19,530::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00609993934631', 'lastCheck': 1354485137.28635, 'code': 0, 'valid': True}} >Thread-8788::DEBUG::2012-12-02 23:52:19,531::task::1151::TaskManager.Task::(prepare) Task=`f21c3001-1a1d-4d00-86e4-f3df3a03b4f5`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00609993934631', 'lastCheck': 1354485137.28635, 'code': 0, 'valid': True}} >Thread-8788::DEBUG::2012-12-02 23:52:19,532::task::568::TaskManager.Task::(_updateState) Task=`f21c3001-1a1d-4d00-86e4-f3df3a03b4f5`::moving from state preparing -> state finished >Thread-8788::DEBUG::2012-12-02 23:52:19,532::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8788::DEBUG::2012-12-02 23:52:19,533::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8788::DEBUG::2012-12-02 23:52:19,533::task::957::TaskManager.Task::(_decref) Task=`f21c3001-1a1d-4d00-86e4-f3df3a03b4f5`::ref 0 aborting False >Thread-8680::WARNING::2012-12-02 23:52:19,541::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8680::INFO::2012-12-02 23:52:19,541::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8680::DEBUG::2012-12-02 23:52:19,541::task::1151::TaskManager.Task::(prepare) Task=`62d07444-2643-4431-a543-46790a5ecf4c`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8680::DEBUG::2012-12-02 23:52:19,542::task::568::TaskManager.Task::(_updateState) Task=`62d07444-2643-4431-a543-46790a5ecf4c`::moving from state preparing -> state finished >Thread-8680::DEBUG::2012-12-02 23:52:19,542::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8680::DEBUG::2012-12-02 23:52:19,542::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8680::DEBUG::2012-12-02 23:52:19,542::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8680::DEBUG::2012-12-02 23:52:19,542::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8680::DEBUG::2012-12-02 23:52:19,542::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8680::DEBUG::2012-12-02 23:52:19,543::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8680::DEBUG::2012-12-02 23:52:19,543::task::957::TaskManager.Task::(_decref) Task=`62d07444-2643-4431-a543-46790a5ecf4c`::ref 0 aborting False >Thread-8809::DEBUG::2012-12-02 23:52:29,769::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8809::DEBUG::2012-12-02 23:52:29,769::task::568::TaskManager.Task::(_updateState) Task=`1b429c33-0d72-4394-8ddb-e98d257f2a32`::moving from state init -> state preparing >Thread-8809::INFO::2012-12-02 23:52:29,769::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8809::INFO::2012-12-02 23:52:29,769::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8809::DEBUG::2012-12-02 23:52:29,769::task::1151::TaskManager.Task::(prepare) Task=`1b429c33-0d72-4394-8ddb-e98d257f2a32`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8809::DEBUG::2012-12-02 23:52:29,770::task::568::TaskManager.Task::(_updateState) Task=`1b429c33-0d72-4394-8ddb-e98d257f2a32`::moving from state preparing -> state finished >Thread-8809::DEBUG::2012-12-02 23:52:29,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8809::DEBUG::2012-12-02 23:52:29,770::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8809::DEBUG::2012-12-02 23:52:29,770::task::957::TaskManager.Task::(_decref) Task=`1b429c33-0d72-4394-8ddb-e98d257f2a32`::ref 0 aborting False >Thread-8810::DEBUG::2012-12-02 23:52:29,917::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8810::DEBUG::2012-12-02 23:52:29,917::task::568::TaskManager.Task::(_updateState) Task=`0025e5a3-1d21-426f-80fc-a32aed47264b`::moving from state init -> state preparing >Thread-8810::INFO::2012-12-02 23:52:29,917::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8810::DEBUG::2012-12-02 23:52:29,917::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`762f3e61-e434-49b2-8f5d-76049b742c0e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8810::DEBUG::2012-12-02 23:52:29,918::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8810::DEBUG::2012-12-02 23:52:29,918::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8810::DEBUG::2012-12-02 23:52:29,918::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`762f3e61-e434-49b2-8f5d-76049b742c0e`::Granted request >Thread-8810::DEBUG::2012-12-02 23:52:29,918::task::794::TaskManager.Task::(resourceAcquired) Task=`0025e5a3-1d21-426f-80fc-a32aed47264b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8810::DEBUG::2012-12-02 23:52:29,918::task::957::TaskManager.Task::(_decref) Task=`0025e5a3-1d21-426f-80fc-a32aed47264b`::ref 1 aborting False >Thread-8810::DEBUG::2012-12-02 23:52:29,925::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8810::DEBUG::2012-12-02 23:52:29,925::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8810::DEBUG::2012-12-02 23:52:30,133::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8810::WARNING::2012-12-02 23:52:30,134::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8810::DEBUG::2012-12-02 23:52:30,134::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8810::WARNING::2012-12-02 23:52:30,164::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8810::DEBUG::2012-12-02 23:52:30,164::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8810::DEBUG::2012-12-02 23:52:30,164::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8810::DEBUG::2012-12-02 23:52:30,372::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8810::WARNING::2012-12-02 23:52:30,374::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8810::DEBUG::2012-12-02 23:52:30,374::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8810::WARNING::2012-12-02 23:52:30,404::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8810::INFO::2012-12-02 23:52:30,404::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8810::DEBUG::2012-12-02 23:52:30,404::task::1151::TaskManager.Task::(prepare) Task=`0025e5a3-1d21-426f-80fc-a32aed47264b`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8810::DEBUG::2012-12-02 23:52:30,405::task::568::TaskManager.Task::(_updateState) Task=`0025e5a3-1d21-426f-80fc-a32aed47264b`::moving from state preparing -> state finished >Thread-8810::DEBUG::2012-12-02 23:52:30,405::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8810::DEBUG::2012-12-02 23:52:30,405::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8810::DEBUG::2012-12-02 23:52:30,405::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8810::DEBUG::2012-12-02 23:52:30,405::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8810::DEBUG::2012-12-02 23:52:30,405::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8810::DEBUG::2012-12-02 23:52:30,406::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8810::DEBUG::2012-12-02 23:52:30,406::task::957::TaskManager.Task::(_decref) Task=`0025e5a3-1d21-426f-80fc-a32aed47264b`::ref 0 aborting False >Thread-8933::DEBUG::2012-12-02 23:52:30,493::task::568::TaskManager.Task::(_updateState) Task=`ad29a1be-74df-4c4f-b4be-b90515808144`::moving from state init -> state preparing >Thread-8933::INFO::2012-12-02 23:52:30,493::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-8933::INFO::2012-12-02 23:52:30,494::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00802612304688', 'lastCheck': 1354485147.3064229, 'code': 0, 'valid': True}} >Thread-8933::DEBUG::2012-12-02 23:52:30,494::task::1151::TaskManager.Task::(prepare) Task=`ad29a1be-74df-4c4f-b4be-b90515808144`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00802612304688', 'lastCheck': 1354485147.3064229, 'code': 0, 'valid': True}} >Thread-8933::DEBUG::2012-12-02 23:52:30,494::task::568::TaskManager.Task::(_updateState) Task=`ad29a1be-74df-4c4f-b4be-b90515808144`::moving from state preparing -> state finished >Thread-8933::DEBUG::2012-12-02 23:52:30,494::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8933::DEBUG::2012-12-02 23:52:30,494::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8933::DEBUG::2012-12-02 23:52:30,494::task::957::TaskManager.Task::(_decref) Task=`ad29a1be-74df-4c4f-b4be-b90515808144`::ref 0 aborting False >Thread-8939::DEBUG::2012-12-02 23:52:40,613::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8939::DEBUG::2012-12-02 23:52:40,614::task::568::TaskManager.Task::(_updateState) Task=`586ac039-bd2e-4143-9b29-f9ebea1b06c5`::moving from state init -> state preparing >Thread-8939::INFO::2012-12-02 23:52:40,614::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8939::INFO::2012-12-02 23:52:40,614::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8939::DEBUG::2012-12-02 23:52:40,615::task::1151::TaskManager.Task::(prepare) Task=`586ac039-bd2e-4143-9b29-f9ebea1b06c5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-8939::DEBUG::2012-12-02 23:52:40,615::task::568::TaskManager.Task::(_updateState) Task=`586ac039-bd2e-4143-9b29-f9ebea1b06c5`::moving from state preparing -> state finished >Thread-8939::DEBUG::2012-12-02 23:52:40,615::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-8939::DEBUG::2012-12-02 23:52:40,615::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8939::DEBUG::2012-12-02 23:52:40,615::task::957::TaskManager.Task::(_decref) Task=`586ac039-bd2e-4143-9b29-f9ebea1b06c5`::ref 0 aborting False >Thread-8940::DEBUG::2012-12-02 23:52:40,756::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-8940::DEBUG::2012-12-02 23:52:40,756::task::568::TaskManager.Task::(_updateState) Task=`824a5b7d-2fbd-4a99-a9d7-2a09744c993d`::moving from state init -> state preparing >Thread-8940::INFO::2012-12-02 23:52:40,756::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-8940::DEBUG::2012-12-02 23:52:40,756::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1dd7df9d-df2f-4358-837b-953dccc1ce14`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-8940::DEBUG::2012-12-02 23:52:40,757::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-8940::DEBUG::2012-12-02 23:52:40,757::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-8940::DEBUG::2012-12-02 23:52:40,757::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`1dd7df9d-df2f-4358-837b-953dccc1ce14`::Granted request >Thread-8940::DEBUG::2012-12-02 23:52:40,757::task::794::TaskManager.Task::(resourceAcquired) Task=`824a5b7d-2fbd-4a99-a9d7-2a09744c993d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-8940::DEBUG::2012-12-02 23:52:40,757::task::957::TaskManager.Task::(_decref) Task=`824a5b7d-2fbd-4a99-a9d7-2a09744c993d`::ref 1 aborting False >Thread-8940::DEBUG::2012-12-02 23:52:40,764::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8940::DEBUG::2012-12-02 23:52:40,765::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-8940::DEBUG::2012-12-02 23:52:40,981::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-8940::WARNING::2012-12-02 23:52:40,983::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-8940::DEBUG::2012-12-02 23:52:40,983::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8940::WARNING::2012-12-02 23:52:41,014::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-8940::DEBUG::2012-12-02 23:52:41,014::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-8940::DEBUG::2012-12-02 23:52:41,015::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-8940::DEBUG::2012-12-02 23:52:41,221::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-8940::WARNING::2012-12-02 23:52:41,223::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-8940::DEBUG::2012-12-02 23:52:41,223::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8940::WARNING::2012-12-02 23:52:41,254::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-8940::INFO::2012-12-02 23:52:41,254::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8940::DEBUG::2012-12-02 23:52:41,255::task::1151::TaskManager.Task::(prepare) Task=`824a5b7d-2fbd-4a99-a9d7-2a09744c993d`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-8940::DEBUG::2012-12-02 23:52:41,255::task::568::TaskManager.Task::(_updateState) Task=`824a5b7d-2fbd-4a99-a9d7-2a09744c993d`::moving from state preparing -> state finished >Thread-8940::DEBUG::2012-12-02 23:52:41,255::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-8940::DEBUG::2012-12-02 23:52:41,255::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-8940::DEBUG::2012-12-02 23:52:41,255::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-8940::DEBUG::2012-12-02 23:52:41,255::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-8940::DEBUG::2012-12-02 23:52:41,256::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-8940::DEBUG::2012-12-02 23:52:41,256::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-8940::DEBUG::2012-12-02 23:52:41,256::task::957::TaskManager.Task::(_decref) Task=`824a5b7d-2fbd-4a99-a9d7-2a09744c993d`::ref 0 aborting False >Thread-9063::DEBUG::2012-12-02 23:52:41,401::task::568::TaskManager.Task::(_updateState) Task=`29bd63aa-b93c-4bdc-9b25-cb3c780c0dff`::moving from state init -> state preparing >Thread-9063::INFO::2012-12-02 23:52:41,401::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9063::INFO::2012-12-02 23:52:41,402::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00722408294678', 'lastCheck': 1354485157.324682, 'code': 0, 'valid': True}} >Thread-9063::DEBUG::2012-12-02 23:52:41,402::task::1151::TaskManager.Task::(prepare) Task=`29bd63aa-b93c-4bdc-9b25-cb3c780c0dff`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00722408294678', 'lastCheck': 1354485157.324682, 'code': 0, 'valid': True}} >Thread-9063::DEBUG::2012-12-02 23:52:41,402::task::568::TaskManager.Task::(_updateState) Task=`29bd63aa-b93c-4bdc-9b25-cb3c780c0dff`::moving from state preparing -> state finished >Thread-9063::DEBUG::2012-12-02 23:52:41,402::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9063::DEBUG::2012-12-02 23:52:41,402::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9063::DEBUG::2012-12-02 23:52:41,402::task::957::TaskManager.Task::(_decref) Task=`29bd63aa-b93c-4bdc-9b25-cb3c780c0dff`::ref 0 aborting False >Thread-9069::DEBUG::2012-12-02 23:52:51,472::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9069::DEBUG::2012-12-02 23:52:51,473::task::568::TaskManager.Task::(_updateState) Task=`c6c7fa64-00f3-4062-9013-56d0f8ba553d`::moving from state init -> state preparing >Thread-9069::INFO::2012-12-02 23:52:51,473::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9069::INFO::2012-12-02 23:52:51,473::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9069::DEBUG::2012-12-02 23:52:51,473::task::1151::TaskManager.Task::(prepare) Task=`c6c7fa64-00f3-4062-9013-56d0f8ba553d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9069::DEBUG::2012-12-02 23:52:51,473::task::568::TaskManager.Task::(_updateState) Task=`c6c7fa64-00f3-4062-9013-56d0f8ba553d`::moving from state preparing -> state finished >Thread-9069::DEBUG::2012-12-02 23:52:51,474::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9069::DEBUG::2012-12-02 23:52:51,474::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9069::DEBUG::2012-12-02 23:52:51,474::task::957::TaskManager.Task::(_decref) Task=`c6c7fa64-00f3-4062-9013-56d0f8ba553d`::ref 0 aborting False >Thread-9070::DEBUG::2012-12-02 23:52:51,613::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9070::DEBUG::2012-12-02 23:52:51,614::task::568::TaskManager.Task::(_updateState) Task=`603abeba-d171-49b8-8762-928a485bccbd`::moving from state init -> state preparing >Thread-9070::INFO::2012-12-02 23:52:51,614::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9070::DEBUG::2012-12-02 23:52:51,614::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4b727464-3781-4f44-b074-c36f4d05ce79`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9070::DEBUG::2012-12-02 23:52:51,614::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9070::DEBUG::2012-12-02 23:52:51,614::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9070::DEBUG::2012-12-02 23:52:51,615::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`4b727464-3781-4f44-b074-c36f4d05ce79`::Granted request >Thread-9070::DEBUG::2012-12-02 23:52:51,615::task::794::TaskManager.Task::(resourceAcquired) Task=`603abeba-d171-49b8-8762-928a485bccbd`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9070::DEBUG::2012-12-02 23:52:51,615::task::957::TaskManager.Task::(_decref) Task=`603abeba-d171-49b8-8762-928a485bccbd`::ref 1 aborting False >Thread-9070::DEBUG::2012-12-02 23:52:51,619::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9070::DEBUG::2012-12-02 23:52:51,620::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9070::DEBUG::2012-12-02 23:52:51,827::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9070::WARNING::2012-12-02 23:52:51,828::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9070::DEBUG::2012-12-02 23:52:51,829::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9070::WARNING::2012-12-02 23:52:51,857::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9070::DEBUG::2012-12-02 23:52:51,857::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9070::DEBUG::2012-12-02 23:52:51,858::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9070::DEBUG::2012-12-02 23:52:52,065::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9070::WARNING::2012-12-02 23:52:52,066::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9070::DEBUG::2012-12-02 23:52:52,066::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9070::WARNING::2012-12-02 23:52:52,094::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9070::INFO::2012-12-02 23:52:52,095::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9070::DEBUG::2012-12-02 23:52:52,095::task::1151::TaskManager.Task::(prepare) Task=`603abeba-d171-49b8-8762-928a485bccbd`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9070::DEBUG::2012-12-02 23:52:52,095::task::568::TaskManager.Task::(_updateState) Task=`603abeba-d171-49b8-8762-928a485bccbd`::moving from state preparing -> state finished >Thread-9070::DEBUG::2012-12-02 23:52:52,095::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9070::DEBUG::2012-12-02 23:52:52,095::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9070::DEBUG::2012-12-02 23:52:52,096::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9070::DEBUG::2012-12-02 23:52:52,096::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9070::DEBUG::2012-12-02 23:52:52,096::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9070::DEBUG::2012-12-02 23:52:52,096::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9070::DEBUG::2012-12-02 23:52:52,096::task::957::TaskManager.Task::(_decref) Task=`603abeba-d171-49b8-8762-928a485bccbd`::ref 0 aborting False >Thread-9193::DEBUG::2012-12-02 23:52:52,338::task::568::TaskManager.Task::(_updateState) Task=`89f7e551-d79e-4150-a857-20538cfbd8e8`::moving from state init -> state preparing >Thread-9193::INFO::2012-12-02 23:52:52,339::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9193::INFO::2012-12-02 23:52:52,339::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00832104682922', 'lastCheck': 1354485167.34496, 'code': 0, 'valid': True}} >Thread-9193::DEBUG::2012-12-02 23:52:52,339::task::1151::TaskManager.Task::(prepare) Task=`89f7e551-d79e-4150-a857-20538cfbd8e8`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00832104682922', 'lastCheck': 1354485167.34496, 'code': 0, 'valid': True}} >Thread-9193::DEBUG::2012-12-02 23:52:52,339::task::568::TaskManager.Task::(_updateState) Task=`89f7e551-d79e-4150-a857-20538cfbd8e8`::moving from state preparing -> state finished >Thread-9193::DEBUG::2012-12-02 23:52:52,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9193::DEBUG::2012-12-02 23:52:52,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9193::DEBUG::2012-12-02 23:52:52,339::task::957::TaskManager.Task::(_decref) Task=`89f7e551-d79e-4150-a857-20538cfbd8e8`::ref 0 aborting False >Thread-9199::DEBUG::2012-12-02 23:53:02,346::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9199::DEBUG::2012-12-02 23:53:02,347::task::568::TaskManager.Task::(_updateState) Task=`96a5c30b-ba1c-4591-a22c-f259531ceb0c`::moving from state init -> state preparing >Thread-9199::INFO::2012-12-02 23:53:02,347::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9199::INFO::2012-12-02 23:53:02,347::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9199::DEBUG::2012-12-02 23:53:02,347::task::1151::TaskManager.Task::(prepare) Task=`96a5c30b-ba1c-4591-a22c-f259531ceb0c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9199::DEBUG::2012-12-02 23:53:02,347::task::568::TaskManager.Task::(_updateState) Task=`96a5c30b-ba1c-4591-a22c-f259531ceb0c`::moving from state preparing -> state finished >Thread-9199::DEBUG::2012-12-02 23:53:02,348::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9199::DEBUG::2012-12-02 23:53:02,348::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9199::DEBUG::2012-12-02 23:53:02,348::task::957::TaskManager.Task::(_decref) Task=`96a5c30b-ba1c-4591-a22c-f259531ceb0c`::ref 0 aborting False >Thread-9200::DEBUG::2012-12-02 23:53:02,484::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9200::DEBUG::2012-12-02 23:53:02,485::task::568::TaskManager.Task::(_updateState) Task=`48500557-c193-40e8-af9d-fe64f6106130`::moving from state init -> state preparing >Thread-9200::INFO::2012-12-02 23:53:02,485::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9200::DEBUG::2012-12-02 23:53:02,485::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f799b19b-dd51-4b5f-b03f-420c53079ad1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9200::DEBUG::2012-12-02 23:53:02,485::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9200::DEBUG::2012-12-02 23:53:02,485::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9200::DEBUG::2012-12-02 23:53:02,485::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f799b19b-dd51-4b5f-b03f-420c53079ad1`::Granted request >Thread-9200::DEBUG::2012-12-02 23:53:02,486::task::794::TaskManager.Task::(resourceAcquired) Task=`48500557-c193-40e8-af9d-fe64f6106130`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9200::DEBUG::2012-12-02 23:53:02,486::task::957::TaskManager.Task::(_decref) Task=`48500557-c193-40e8-af9d-fe64f6106130`::ref 1 aborting False >Thread-9200::DEBUG::2012-12-02 23:53:02,490::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9200::DEBUG::2012-12-02 23:53:02,490::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9200::DEBUG::2012-12-02 23:53:02,705::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9200::WARNING::2012-12-02 23:53:02,706::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9200::DEBUG::2012-12-02 23:53:02,706::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9200::WARNING::2012-12-02 23:53:02,738::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9200::DEBUG::2012-12-02 23:53:02,738::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9200::DEBUG::2012-12-02 23:53:02,738::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9200::DEBUG::2012-12-02 23:53:02,943::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9200::WARNING::2012-12-02 23:53:02,944::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9200::DEBUG::2012-12-02 23:53:02,945::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9200::WARNING::2012-12-02 23:53:02,973::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9200::INFO::2012-12-02 23:53:02,974::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9200::DEBUG::2012-12-02 23:53:02,974::task::1151::TaskManager.Task::(prepare) Task=`48500557-c193-40e8-af9d-fe64f6106130`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9200::DEBUG::2012-12-02 23:53:02,974::task::568::TaskManager.Task::(_updateState) Task=`48500557-c193-40e8-af9d-fe64f6106130`::moving from state preparing -> state finished >Thread-9200::DEBUG::2012-12-02 23:53:02,974::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9200::DEBUG::2012-12-02 23:53:02,974::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9200::DEBUG::2012-12-02 23:53:02,975::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9200::DEBUG::2012-12-02 23:53:02,975::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9200::DEBUG::2012-12-02 23:53:02,975::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9200::DEBUG::2012-12-02 23:53:02,975::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9200::DEBUG::2012-12-02 23:53:02,975::task::957::TaskManager.Task::(_decref) Task=`48500557-c193-40e8-af9d-fe64f6106130`::ref 0 aborting False >Thread-9323::DEBUG::2012-12-02 23:53:03,256::task::568::TaskManager.Task::(_updateState) Task=`6ecffc14-b672-413d-82b2-8278118272e9`::moving from state init -> state preparing >Thread-9323::INFO::2012-12-02 23:53:03,256::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9323::INFO::2012-12-02 23:53:03,257::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00931715965271', 'lastCheck': 1354485177.3652201, 'code': 0, 'valid': True}} >Thread-9323::DEBUG::2012-12-02 23:53:03,257::task::1151::TaskManager.Task::(prepare) Task=`6ecffc14-b672-413d-82b2-8278118272e9`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00931715965271', 'lastCheck': 1354485177.3652201, 'code': 0, 'valid': True}} >Thread-9323::DEBUG::2012-12-02 23:53:03,257::task::568::TaskManager.Task::(_updateState) Task=`6ecffc14-b672-413d-82b2-8278118272e9`::moving from state preparing -> state finished >Thread-9323::DEBUG::2012-12-02 23:53:03,257::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9323::DEBUG::2012-12-02 23:53:03,257::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9323::DEBUG::2012-12-02 23:53:03,257::task::957::TaskManager.Task::(_decref) Task=`6ecffc14-b672-413d-82b2-8278118272e9`::ref 0 aborting False >Thread-9329::DEBUG::2012-12-02 23:53:13,219::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9329::DEBUG::2012-12-02 23:53:13,219::task::568::TaskManager.Task::(_updateState) Task=`9b1dbb09-199e-4bc0-a2e1-ed124a2c5245`::moving from state init -> state preparing >Thread-9329::INFO::2012-12-02 23:53:13,219::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9329::INFO::2012-12-02 23:53:13,220::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9329::DEBUG::2012-12-02 23:53:13,220::task::1151::TaskManager.Task::(prepare) Task=`9b1dbb09-199e-4bc0-a2e1-ed124a2c5245`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9329::DEBUG::2012-12-02 23:53:13,220::task::568::TaskManager.Task::(_updateState) Task=`9b1dbb09-199e-4bc0-a2e1-ed124a2c5245`::moving from state preparing -> state finished >Thread-9329::DEBUG::2012-12-02 23:53:13,220::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9329::DEBUG::2012-12-02 23:53:13,220::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9329::DEBUG::2012-12-02 23:53:13,220::task::957::TaskManager.Task::(_decref) Task=`9b1dbb09-199e-4bc0-a2e1-ed124a2c5245`::ref 0 aborting False >Thread-9330::DEBUG::2012-12-02 23:53:13,357::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9330::DEBUG::2012-12-02 23:53:13,357::task::568::TaskManager.Task::(_updateState) Task=`c4a56860-5d95-48f1-a120-78f6e339d81d`::moving from state init -> state preparing >Thread-9330::INFO::2012-12-02 23:53:13,357::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9330::DEBUG::2012-12-02 23:53:13,358::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f873d854-62eb-4b1b-827d-a823f2f61ef6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9330::DEBUG::2012-12-02 23:53:13,358::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9330::DEBUG::2012-12-02 23:53:13,358::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9330::DEBUG::2012-12-02 23:53:13,358::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f873d854-62eb-4b1b-827d-a823f2f61ef6`::Granted request >Thread-9330::DEBUG::2012-12-02 23:53:13,358::task::794::TaskManager.Task::(resourceAcquired) Task=`c4a56860-5d95-48f1-a120-78f6e339d81d`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9330::DEBUG::2012-12-02 23:53:13,359::task::957::TaskManager.Task::(_decref) Task=`c4a56860-5d95-48f1-a120-78f6e339d81d`::ref 1 aborting False >Thread-9330::DEBUG::2012-12-02 23:53:13,365::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9330::DEBUG::2012-12-02 23:53:13,365::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9330::DEBUG::2012-12-02 23:53:13,571::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9330::WARNING::2012-12-02 23:53:13,572::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9330::DEBUG::2012-12-02 23:53:13,573::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9330::WARNING::2012-12-02 23:53:13,604::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9330::DEBUG::2012-12-02 23:53:13,605::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9330::DEBUG::2012-12-02 23:53:13,605::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9330::DEBUG::2012-12-02 23:53:13,813::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9330::WARNING::2012-12-02 23:53:13,814::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9330::DEBUG::2012-12-02 23:53:13,815::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9330::WARNING::2012-12-02 23:53:13,844::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9330::INFO::2012-12-02 23:53:13,844::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9330::DEBUG::2012-12-02 23:53:13,844::task::1151::TaskManager.Task::(prepare) Task=`c4a56860-5d95-48f1-a120-78f6e339d81d`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9330::DEBUG::2012-12-02 23:53:13,844::task::568::TaskManager.Task::(_updateState) Task=`c4a56860-5d95-48f1-a120-78f6e339d81d`::moving from state preparing -> state finished >Thread-9330::DEBUG::2012-12-02 23:53:13,845::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9330::DEBUG::2012-12-02 23:53:13,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9330::DEBUG::2012-12-02 23:53:13,845::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9330::DEBUG::2012-12-02 23:53:13,845::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9330::DEBUG::2012-12-02 23:53:13,845::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9330::DEBUG::2012-12-02 23:53:13,845::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9330::DEBUG::2012-12-02 23:53:13,846::task::957::TaskManager.Task::(_decref) Task=`c4a56860-5d95-48f1-a120-78f6e339d81d`::ref 0 aborting False >Thread-9453::DEBUG::2012-12-02 23:53:14,177::task::568::TaskManager.Task::(_updateState) Task=`af4f95e1-355a-416e-a075-0e0fbd8b8ad5`::moving from state init -> state preparing >Thread-9453::INFO::2012-12-02 23:53:14,177::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9453::INFO::2012-12-02 23:53:14,177::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0068678855896', 'lastCheck': 1354485187.382668, 'code': 0, 'valid': True}} >Thread-9453::DEBUG::2012-12-02 23:53:14,177::task::1151::TaskManager.Task::(prepare) Task=`af4f95e1-355a-416e-a075-0e0fbd8b8ad5`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.0068678855896', 'lastCheck': 1354485187.382668, 'code': 0, 'valid': True}} >Thread-9453::DEBUG::2012-12-02 23:53:14,177::task::568::TaskManager.Task::(_updateState) Task=`af4f95e1-355a-416e-a075-0e0fbd8b8ad5`::moving from state preparing -> state finished >Thread-9453::DEBUG::2012-12-02 23:53:14,177::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9453::DEBUG::2012-12-02 23:53:14,178::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9453::DEBUG::2012-12-02 23:53:14,178::task::957::TaskManager.Task::(_decref) Task=`af4f95e1-355a-416e-a075-0e0fbd8b8ad5`::ref 0 aborting False >Thread-9459::DEBUG::2012-12-02 23:53:24,049::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9459::DEBUG::2012-12-02 23:53:24,049::task::568::TaskManager.Task::(_updateState) Task=`7f47158a-e7a1-4628-8c14-5ee0dbb547bd`::moving from state init -> state preparing >Thread-9459::INFO::2012-12-02 23:53:24,049::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9459::INFO::2012-12-02 23:53:24,050::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9459::DEBUG::2012-12-02 23:53:24,050::task::1151::TaskManager.Task::(prepare) Task=`7f47158a-e7a1-4628-8c14-5ee0dbb547bd`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9459::DEBUG::2012-12-02 23:53:24,050::task::568::TaskManager.Task::(_updateState) Task=`7f47158a-e7a1-4628-8c14-5ee0dbb547bd`::moving from state preparing -> state finished >Thread-9459::DEBUG::2012-12-02 23:53:24,050::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9459::DEBUG::2012-12-02 23:53:24,050::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9459::DEBUG::2012-12-02 23:53:24,050::task::957::TaskManager.Task::(_decref) Task=`7f47158a-e7a1-4628-8c14-5ee0dbb547bd`::ref 0 aborting False >Thread-9460::DEBUG::2012-12-02 23:53:24,190::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9460::DEBUG::2012-12-02 23:53:24,190::task::568::TaskManager.Task::(_updateState) Task=`89c4fa47-9b88-4fc8-bce4-48af95a09bb7`::moving from state init -> state preparing >Thread-9460::INFO::2012-12-02 23:53:24,191::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9460::DEBUG::2012-12-02 23:53:24,191::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3301cd75-e925-4ab8-9ba5-2f33dc56d8eb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9460::DEBUG::2012-12-02 23:53:24,191::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9460::DEBUG::2012-12-02 23:53:24,191::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9460::DEBUG::2012-12-02 23:53:24,191::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`3301cd75-e925-4ab8-9ba5-2f33dc56d8eb`::Granted request >Thread-9460::DEBUG::2012-12-02 23:53:24,192::task::794::TaskManager.Task::(resourceAcquired) Task=`89c4fa47-9b88-4fc8-bce4-48af95a09bb7`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9460::DEBUG::2012-12-02 23:53:24,192::task::957::TaskManager.Task::(_decref) Task=`89c4fa47-9b88-4fc8-bce4-48af95a09bb7`::ref 1 aborting False >Thread-9460::DEBUG::2012-12-02 23:53:24,197::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9460::DEBUG::2012-12-02 23:53:24,197::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9460::DEBUG::2012-12-02 23:53:24,402::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9460::WARNING::2012-12-02 23:53:24,403::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9460::DEBUG::2012-12-02 23:53:24,404::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9460::WARNING::2012-12-02 23:53:24,434::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9460::DEBUG::2012-12-02 23:53:24,434::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9460::DEBUG::2012-12-02 23:53:24,435::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9460::DEBUG::2012-12-02 23:53:24,641::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9460::WARNING::2012-12-02 23:53:24,642::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9460::DEBUG::2012-12-02 23:53:24,643::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9460::WARNING::2012-12-02 23:53:24,672::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9460::INFO::2012-12-02 23:53:24,673::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9460::DEBUG::2012-12-02 23:53:24,673::task::1151::TaskManager.Task::(prepare) Task=`89c4fa47-9b88-4fc8-bce4-48af95a09bb7`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9460::DEBUG::2012-12-02 23:53:24,673::task::568::TaskManager.Task::(_updateState) Task=`89c4fa47-9b88-4fc8-bce4-48af95a09bb7`::moving from state preparing -> state finished >Thread-9460::DEBUG::2012-12-02 23:53:24,673::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9460::DEBUG::2012-12-02 23:53:24,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9460::DEBUG::2012-12-02 23:53:24,674::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9460::DEBUG::2012-12-02 23:53:24,674::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9460::DEBUG::2012-12-02 23:53:24,674::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9460::DEBUG::2012-12-02 23:53:24,674::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9460::DEBUG::2012-12-02 23:53:24,674::task::957::TaskManager.Task::(_decref) Task=`89c4fa47-9b88-4fc8-bce4-48af95a09bb7`::ref 0 aborting False >Thread-9583::DEBUG::2012-12-02 23:53:25,171::task::568::TaskManager.Task::(_updateState) Task=`c35d4856-232c-4bdb-a831-b24f9ea1c00d`::moving from state init -> state preparing >Thread-9583::INFO::2012-12-02 23:53:25,171::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9583::INFO::2012-12-02 23:53:25,171::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00739502906799', 'lastCheck': 1354485197.4016311, 'code': 0, 'valid': True}} >Thread-9583::DEBUG::2012-12-02 23:53:25,172::task::1151::TaskManager.Task::(prepare) Task=`c35d4856-232c-4bdb-a831-b24f9ea1c00d`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00739502906799', 'lastCheck': 1354485197.4016311, 'code': 0, 'valid': True}} >Thread-9583::DEBUG::2012-12-02 23:53:25,172::task::568::TaskManager.Task::(_updateState) Task=`c35d4856-232c-4bdb-a831-b24f9ea1c00d`::moving from state preparing -> state finished >Thread-9583::DEBUG::2012-12-02 23:53:25,172::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9583::DEBUG::2012-12-02 23:53:25,172::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9583::DEBUG::2012-12-02 23:53:25,172::task::957::TaskManager.Task::(_decref) Task=`c35d4856-232c-4bdb-a831-b24f9ea1c00d`::ref 0 aborting False >Thread-9589::DEBUG::2012-12-02 23:53:34,921::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9589::DEBUG::2012-12-02 23:53:34,921::task::568::TaskManager.Task::(_updateState) Task=`4ec598eb-736d-4888-952d-2d571da50d55`::moving from state init -> state preparing >Thread-9589::INFO::2012-12-02 23:53:34,922::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9589::INFO::2012-12-02 23:53:34,922::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9589::DEBUG::2012-12-02 23:53:34,922::task::1151::TaskManager.Task::(prepare) Task=`4ec598eb-736d-4888-952d-2d571da50d55`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9589::DEBUG::2012-12-02 23:53:34,922::task::568::TaskManager.Task::(_updateState) Task=`4ec598eb-736d-4888-952d-2d571da50d55`::moving from state preparing -> state finished >Thread-9589::DEBUG::2012-12-02 23:53:34,922::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9589::DEBUG::2012-12-02 23:53:34,922::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9589::DEBUG::2012-12-02 23:53:34,923::task::957::TaskManager.Task::(_decref) Task=`4ec598eb-736d-4888-952d-2d571da50d55`::ref 0 aborting False >Thread-9590::DEBUG::2012-12-02 23:53:35,066::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9590::DEBUG::2012-12-02 23:53:35,066::task::568::TaskManager.Task::(_updateState) Task=`3af16dd0-f193-4547-b972-a626d19b090e`::moving from state init -> state preparing >Thread-9590::INFO::2012-12-02 23:53:35,067::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9590::DEBUG::2012-12-02 23:53:35,067::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f887d510-c59b-4ec0-8058-fe1193769a6b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9590::DEBUG::2012-12-02 23:53:35,067::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9590::DEBUG::2012-12-02 23:53:35,067::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9590::DEBUG::2012-12-02 23:53:35,067::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f887d510-c59b-4ec0-8058-fe1193769a6b`::Granted request >Thread-9590::DEBUG::2012-12-02 23:53:35,068::task::794::TaskManager.Task::(resourceAcquired) Task=`3af16dd0-f193-4547-b972-a626d19b090e`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9590::DEBUG::2012-12-02 23:53:35,068::task::957::TaskManager.Task::(_decref) Task=`3af16dd0-f193-4547-b972-a626d19b090e`::ref 1 aborting False >Thread-9590::DEBUG::2012-12-02 23:53:35,074::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9590::DEBUG::2012-12-02 23:53:35,075::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9590::DEBUG::2012-12-02 23:53:35,279::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9590::WARNING::2012-12-02 23:53:35,280::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9590::DEBUG::2012-12-02 23:53:35,281::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9590::WARNING::2012-12-02 23:53:35,310::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9590::DEBUG::2012-12-02 23:53:35,310::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9590::DEBUG::2012-12-02 23:53:35,311::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9590::DEBUG::2012-12-02 23:53:35,516::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9590::WARNING::2012-12-02 23:53:35,518::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9590::DEBUG::2012-12-02 23:53:35,518::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9590::WARNING::2012-12-02 23:53:35,546::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9590::INFO::2012-12-02 23:53:35,547::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9590::DEBUG::2012-12-02 23:53:35,547::task::1151::TaskManager.Task::(prepare) Task=`3af16dd0-f193-4547-b972-a626d19b090e`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9590::DEBUG::2012-12-02 23:53:35,547::task::568::TaskManager.Task::(_updateState) Task=`3af16dd0-f193-4547-b972-a626d19b090e`::moving from state preparing -> state finished >Thread-9590::DEBUG::2012-12-02 23:53:35,547::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9590::DEBUG::2012-12-02 23:53:35,548::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9590::DEBUG::2012-12-02 23:53:35,548::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9590::DEBUG::2012-12-02 23:53:35,548::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9590::DEBUG::2012-12-02 23:53:35,548::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9590::DEBUG::2012-12-02 23:53:35,548::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9590::DEBUG::2012-12-02 23:53:35,548::task::957::TaskManager.Task::(_decref) Task=`3af16dd0-f193-4547-b972-a626d19b090e`::ref 0 aborting False >Thread-9713::DEBUG::2012-12-02 23:53:36,126::task::568::TaskManager.Task::(_updateState) Task=`d0e9fe89-a9cd-4915-adbd-14015821e87e`::moving from state init -> state preparing >Thread-9713::INFO::2012-12-02 23:53:36,127::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9713::INFO::2012-12-02 23:53:36,127::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00713801383972', 'lastCheck': 1354485207.420007, 'code': 0, 'valid': True}} >Thread-9713::DEBUG::2012-12-02 23:53:36,127::task::1151::TaskManager.Task::(prepare) Task=`d0e9fe89-a9cd-4915-adbd-14015821e87e`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00713801383972', 'lastCheck': 1354485207.420007, 'code': 0, 'valid': True}} >Thread-9713::DEBUG::2012-12-02 23:53:36,127::task::568::TaskManager.Task::(_updateState) Task=`d0e9fe89-a9cd-4915-adbd-14015821e87e`::moving from state preparing -> state finished >Thread-9713::DEBUG::2012-12-02 23:53:36,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9713::DEBUG::2012-12-02 23:53:36,127::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9713::DEBUG::2012-12-02 23:53:36,128::task::957::TaskManager.Task::(_decref) Task=`d0e9fe89-a9cd-4915-adbd-14015821e87e`::ref 0 aborting False >Thread-9719::DEBUG::2012-12-02 23:53:45,784::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9719::DEBUG::2012-12-02 23:53:45,784::task::568::TaskManager.Task::(_updateState) Task=`3895edb0-454a-4d5e-b336-154565008390`::moving from state init -> state preparing >Thread-9719::INFO::2012-12-02 23:53:45,785::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9719::INFO::2012-12-02 23:53:45,785::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9719::DEBUG::2012-12-02 23:53:45,785::task::1151::TaskManager.Task::(prepare) Task=`3895edb0-454a-4d5e-b336-154565008390`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9719::DEBUG::2012-12-02 23:53:45,785::task::568::TaskManager.Task::(_updateState) Task=`3895edb0-454a-4d5e-b336-154565008390`::moving from state preparing -> state finished >Thread-9719::DEBUG::2012-12-02 23:53:45,785::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9719::DEBUG::2012-12-02 23:53:45,785::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9719::DEBUG::2012-12-02 23:53:45,785::task::957::TaskManager.Task::(_decref) Task=`3895edb0-454a-4d5e-b336-154565008390`::ref 0 aborting False >Thread-9720::DEBUG::2012-12-02 23:53:45,957::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9720::DEBUG::2012-12-02 23:53:45,957::task::568::TaskManager.Task::(_updateState) Task=`d9f95bae-06f5-4e96-8549-3ad7764d77c2`::moving from state init -> state preparing >Thread-9720::INFO::2012-12-02 23:53:45,958::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9720::DEBUG::2012-12-02 23:53:45,958::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`be6d7976-0e8e-461e-b26b-64309fcae7d7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9720::DEBUG::2012-12-02 23:53:45,958::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9720::DEBUG::2012-12-02 23:53:45,958::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9720::DEBUG::2012-12-02 23:53:45,958::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`be6d7976-0e8e-461e-b26b-64309fcae7d7`::Granted request >Thread-9720::DEBUG::2012-12-02 23:53:45,958::task::794::TaskManager.Task::(resourceAcquired) Task=`d9f95bae-06f5-4e96-8549-3ad7764d77c2`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9720::DEBUG::2012-12-02 23:53:45,959::task::957::TaskManager.Task::(_decref) Task=`d9f95bae-06f5-4e96-8549-3ad7764d77c2`::ref 1 aborting False >Thread-9720::DEBUG::2012-12-02 23:53:45,965::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9720::DEBUG::2012-12-02 23:53:45,965::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9720::DEBUG::2012-12-02 23:53:46,172::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9720::WARNING::2012-12-02 23:53:46,173::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9720::DEBUG::2012-12-02 23:53:46,173::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9720::WARNING::2012-12-02 23:53:46,203::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9720::DEBUG::2012-12-02 23:53:46,204::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9720::DEBUG::2012-12-02 23:53:46,204::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9720::DEBUG::2012-12-02 23:53:46,411::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9720::WARNING::2012-12-02 23:53:46,413::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9720::DEBUG::2012-12-02 23:53:46,413::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9720::WARNING::2012-12-02 23:53:46,443::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9720::INFO::2012-12-02 23:53:46,443::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9720::DEBUG::2012-12-02 23:53:46,443::task::1151::TaskManager.Task::(prepare) Task=`d9f95bae-06f5-4e96-8549-3ad7764d77c2`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9720::DEBUG::2012-12-02 23:53:46,443::task::568::TaskManager.Task::(_updateState) Task=`d9f95bae-06f5-4e96-8549-3ad7764d77c2`::moving from state preparing -> state finished >Thread-9720::DEBUG::2012-12-02 23:53:46,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9720::DEBUG::2012-12-02 23:53:46,444::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9720::DEBUG::2012-12-02 23:53:46,444::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9720::DEBUG::2012-12-02 23:53:46,444::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9720::DEBUG::2012-12-02 23:53:46,444::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9720::DEBUG::2012-12-02 23:53:46,444::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9720::DEBUG::2012-12-02 23:53:46,445::task::957::TaskManager.Task::(_decref) Task=`d9f95bae-06f5-4e96-8549-3ad7764d77c2`::ref 0 aborting False >Thread-9843::DEBUG::2012-12-02 23:53:47,132::task::568::TaskManager.Task::(_updateState) Task=`09a05e2d-202f-477e-8e85-ea54669afff0`::moving from state init -> state preparing >Thread-9843::INFO::2012-12-02 23:53:47,133::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9843::INFO::2012-12-02 23:53:47,133::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00745391845703', 'lastCheck': 1354485217.438031, 'code': 0, 'valid': True}} >Thread-9843::DEBUG::2012-12-02 23:53:47,133::task::1151::TaskManager.Task::(prepare) Task=`09a05e2d-202f-477e-8e85-ea54669afff0`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00745391845703', 'lastCheck': 1354485217.438031, 'code': 0, 'valid': True}} >Thread-9843::DEBUG::2012-12-02 23:53:47,133::task::568::TaskManager.Task::(_updateState) Task=`09a05e2d-202f-477e-8e85-ea54669afff0`::moving from state preparing -> state finished >Thread-9843::DEBUG::2012-12-02 23:53:47,133::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9843::DEBUG::2012-12-02 23:53:47,133::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9843::DEBUG::2012-12-02 23:53:47,133::task::957::TaskManager.Task::(_decref) Task=`09a05e2d-202f-477e-8e85-ea54669afff0`::ref 0 aborting False >Thread-9849::DEBUG::2012-12-02 23:53:56,727::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9849::DEBUG::2012-12-02 23:53:56,727::task::568::TaskManager.Task::(_updateState) Task=`ce469dc1-fb87-47ed-8426-acc1ea52da43`::moving from state init -> state preparing >Thread-9849::INFO::2012-12-02 23:53:56,727::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9849::INFO::2012-12-02 23:53:56,727::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9849::DEBUG::2012-12-02 23:53:56,727::task::1151::TaskManager.Task::(prepare) Task=`ce469dc1-fb87-47ed-8426-acc1ea52da43`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9849::DEBUG::2012-12-02 23:53:56,728::task::568::TaskManager.Task::(_updateState) Task=`ce469dc1-fb87-47ed-8426-acc1ea52da43`::moving from state preparing -> state finished >Thread-9849::DEBUG::2012-12-02 23:53:56,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9849::DEBUG::2012-12-02 23:53:56,728::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9849::DEBUG::2012-12-02 23:53:56,728::task::957::TaskManager.Task::(_decref) Task=`ce469dc1-fb87-47ed-8426-acc1ea52da43`::ref 0 aborting False >Thread-9850::DEBUG::2012-12-02 23:53:56,867::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9850::DEBUG::2012-12-02 23:53:56,868::task::568::TaskManager.Task::(_updateState) Task=`274a645b-2ed6-4136-aa07-4fd6ddf169c3`::moving from state init -> state preparing >Thread-9850::INFO::2012-12-02 23:53:56,868::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9850::DEBUG::2012-12-02 23:53:56,868::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`055280ce-e90d-435c-827d-4f6910ea857a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9850::DEBUG::2012-12-02 23:53:56,868::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9850::DEBUG::2012-12-02 23:53:56,869::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9850::DEBUG::2012-12-02 23:53:56,869::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`055280ce-e90d-435c-827d-4f6910ea857a`::Granted request >Thread-9850::DEBUG::2012-12-02 23:53:56,869::task::794::TaskManager.Task::(resourceAcquired) Task=`274a645b-2ed6-4136-aa07-4fd6ddf169c3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9850::DEBUG::2012-12-02 23:53:56,869::task::957::TaskManager.Task::(_decref) Task=`274a645b-2ed6-4136-aa07-4fd6ddf169c3`::ref 1 aborting False >Thread-9850::DEBUG::2012-12-02 23:53:56,875::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9850::DEBUG::2012-12-02 23:53:56,876::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9850::DEBUG::2012-12-02 23:53:57,082::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9850::WARNING::2012-12-02 23:53:57,084::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9850::DEBUG::2012-12-02 23:53:57,084::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9850::WARNING::2012-12-02 23:53:57,114::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9850::DEBUG::2012-12-02 23:53:57,114::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9850::DEBUG::2012-12-02 23:53:57,115::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9850::DEBUG::2012-12-02 23:53:57,319::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9850::WARNING::2012-12-02 23:53:57,320::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9850::DEBUG::2012-12-02 23:53:57,321::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9850::WARNING::2012-12-02 23:53:57,353::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9850::INFO::2012-12-02 23:53:57,354::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9850::DEBUG::2012-12-02 23:53:57,354::task::1151::TaskManager.Task::(prepare) Task=`274a645b-2ed6-4136-aa07-4fd6ddf169c3`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9850::DEBUG::2012-12-02 23:53:57,354::task::568::TaskManager.Task::(_updateState) Task=`274a645b-2ed6-4136-aa07-4fd6ddf169c3`::moving from state preparing -> state finished >Thread-9850::DEBUG::2012-12-02 23:53:57,354::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9850::DEBUG::2012-12-02 23:53:57,354::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9850::DEBUG::2012-12-02 23:53:57,354::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9850::DEBUG::2012-12-02 23:53:57,355::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9850::DEBUG::2012-12-02 23:53:57,355::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9850::DEBUG::2012-12-02 23:53:57,355::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9850::DEBUG::2012-12-02 23:53:57,355::task::957::TaskManager.Task::(_decref) Task=`274a645b-2ed6-4136-aa07-4fd6ddf169c3`::ref 0 aborting False >Thread-9973::DEBUG::2012-12-02 23:53:58,064::task::568::TaskManager.Task::(_updateState) Task=`7ada522c-8ed7-4d3a-9b5d-37860966b025`::moving from state init -> state preparing >Thread-9973::INFO::2012-12-02 23:53:58,064::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9973::INFO::2012-12-02 23:53:58,064::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.010017156601', 'lastCheck': 1354485237.477916, 'code': 0, 'valid': True}} >Thread-9973::DEBUG::2012-12-02 23:53:58,065::task::1151::TaskManager.Task::(prepare) Task=`7ada522c-8ed7-4d3a-9b5d-37860966b025`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.010017156601', 'lastCheck': 1354485237.477916, 'code': 0, 'valid': True}} >Thread-9973::DEBUG::2012-12-02 23:53:58,065::task::568::TaskManager.Task::(_updateState) Task=`7ada522c-8ed7-4d3a-9b5d-37860966b025`::moving from state preparing -> state finished >Thread-9973::DEBUG::2012-12-02 23:53:58,065::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9973::DEBUG::2012-12-02 23:53:58,065::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9973::DEBUG::2012-12-02 23:53:58,065::task::957::TaskManager.Task::(_decref) Task=`7ada522c-8ed7-4d3a-9b5d-37860966b025`::ref 0 aborting False >Thread-9979::DEBUG::2012-12-02 23:54:07,583::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9979::DEBUG::2012-12-02 23:54:07,587::task::568::TaskManager.Task::(_updateState) Task=`44af2f0a-8564-4f05-b1fa-4d364fed19ac`::moving from state init -> state preparing >Thread-9979::INFO::2012-12-02 23:54:07,587::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9979::INFO::2012-12-02 23:54:07,587::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9979::DEBUG::2012-12-02 23:54:07,587::task::1151::TaskManager.Task::(prepare) Task=`44af2f0a-8564-4f05-b1fa-4d364fed19ac`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-9979::DEBUG::2012-12-02 23:54:07,587::task::568::TaskManager.Task::(_updateState) Task=`44af2f0a-8564-4f05-b1fa-4d364fed19ac`::moving from state preparing -> state finished >Thread-9979::DEBUG::2012-12-02 23:54:07,587::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9979::DEBUG::2012-12-02 23:54:07,587::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9979::DEBUG::2012-12-02 23:54:07,587::task::957::TaskManager.Task::(_decref) Task=`44af2f0a-8564-4f05-b1fa-4d364fed19ac`::ref 0 aborting False >Thread-9980::DEBUG::2012-12-02 23:54:07,730::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-9980::DEBUG::2012-12-02 23:54:07,731::task::568::TaskManager.Task::(_updateState) Task=`d48212e9-1b58-40bd-9b79-6d4372bb2379`::moving from state init -> state preparing >Thread-9980::INFO::2012-12-02 23:54:07,731::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9980::DEBUG::2012-12-02 23:54:07,731::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`245125e4-f09f-4c83-823f-bd4b75d28ccb`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9980::DEBUG::2012-12-02 23:54:07,732::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9980::DEBUG::2012-12-02 23:54:07,732::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9980::DEBUG::2012-12-02 23:54:07,732::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`245125e4-f09f-4c83-823f-bd4b75d28ccb`::Granted request >Thread-9980::DEBUG::2012-12-02 23:54:07,732::task::794::TaskManager.Task::(resourceAcquired) Task=`d48212e9-1b58-40bd-9b79-6d4372bb2379`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-9980::DEBUG::2012-12-02 23:54:07,732::task::957::TaskManager.Task::(_decref) Task=`d48212e9-1b58-40bd-9b79-6d4372bb2379`::ref 1 aborting False >Thread-9980::DEBUG::2012-12-02 23:54:07,738::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9980::DEBUG::2012-12-02 23:54:07,738::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-9980::DEBUG::2012-12-02 23:54:07,948::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-9980::WARNING::2012-12-02 23:54:07,950::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-9980::DEBUG::2012-12-02 23:54:07,950::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9980::WARNING::2012-12-02 23:54:08,019::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-9980::DEBUG::2012-12-02 23:54:08,019::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-9980::DEBUG::2012-12-02 23:54:08,020::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-9980::DEBUG::2012-12-02 23:54:08,225::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-9980::WARNING::2012-12-02 23:54:08,226::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-9980::DEBUG::2012-12-02 23:54:08,226::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9980::WARNING::2012-12-02 23:54:08,255::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-9980::INFO::2012-12-02 23:54:08,255::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9980::DEBUG::2012-12-02 23:54:08,255::task::1151::TaskManager.Task::(prepare) Task=`d48212e9-1b58-40bd-9b79-6d4372bb2379`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-9980::DEBUG::2012-12-02 23:54:08,256::task::568::TaskManager.Task::(_updateState) Task=`d48212e9-1b58-40bd-9b79-6d4372bb2379`::moving from state preparing -> state finished >Thread-9980::DEBUG::2012-12-02 23:54:08,256::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9980::DEBUG::2012-12-02 23:54:08,256::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9980::DEBUG::2012-12-02 23:54:08,256::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9980::DEBUG::2012-12-02 23:54:08,256::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9980::DEBUG::2012-12-02 23:54:08,256::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9980::DEBUG::2012-12-02 23:54:08,257::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9980::DEBUG::2012-12-02 23:54:08,257::task::957::TaskManager.Task::(_decref) Task=`d48212e9-1b58-40bd-9b79-6d4372bb2379`::ref 0 aborting False >Thread-10103::DEBUG::2012-12-02 23:54:09,034::task::568::TaskManager.Task::(_updateState) Task=`c0682e3c-04ce-4830-8d9a-e9f6b7e8c0dc`::moving from state init -> state preparing >Thread-10103::INFO::2012-12-02 23:54:09,034::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10103::INFO::2012-12-02 23:54:09,034::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00718593597412', 'lastCheck': 1354485247.4950931, 'code': 0, 'valid': True}} >Thread-10103::DEBUG::2012-12-02 23:54:09,034::task::1151::TaskManager.Task::(prepare) Task=`c0682e3c-04ce-4830-8d9a-e9f6b7e8c0dc`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00718593597412', 'lastCheck': 1354485247.4950931, 'code': 0, 'valid': True}} >Thread-10103::DEBUG::2012-12-02 23:54:09,034::task::568::TaskManager.Task::(_updateState) Task=`c0682e3c-04ce-4830-8d9a-e9f6b7e8c0dc`::moving from state preparing -> state finished >Thread-10103::DEBUG::2012-12-02 23:54:09,034::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10103::DEBUG::2012-12-02 23:54:09,035::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10103::DEBUG::2012-12-02 23:54:09,035::task::957::TaskManager.Task::(_decref) Task=`c0682e3c-04ce-4830-8d9a-e9f6b7e8c0dc`::ref 0 aborting False >Thread-10109::DEBUG::2012-12-02 23:54:18,514::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10109::DEBUG::2012-12-02 23:54:18,515::task::568::TaskManager.Task::(_updateState) Task=`4f55da87-3915-48ad-8558-28fe2947502e`::moving from state init -> state preparing >Thread-10109::INFO::2012-12-02 23:54:18,515::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10109::INFO::2012-12-02 23:54:18,515::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10109::DEBUG::2012-12-02 23:54:18,515::task::1151::TaskManager.Task::(prepare) Task=`4f55da87-3915-48ad-8558-28fe2947502e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10109::DEBUG::2012-12-02 23:54:18,515::task::568::TaskManager.Task::(_updateState) Task=`4f55da87-3915-48ad-8558-28fe2947502e`::moving from state preparing -> state finished >Thread-10109::DEBUG::2012-12-02 23:54:18,515::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10109::DEBUG::2012-12-02 23:54:18,516::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10109::DEBUG::2012-12-02 23:54:18,516::task::957::TaskManager.Task::(_decref) Task=`4f55da87-3915-48ad-8558-28fe2947502e`::ref 0 aborting False >Thread-10110::DEBUG::2012-12-02 23:54:18,653::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10110::DEBUG::2012-12-02 23:54:18,653::task::568::TaskManager.Task::(_updateState) Task=`389a4f83-e7d0-4840-bb5c-d705d65ed02a`::moving from state init -> state preparing >Thread-10110::INFO::2012-12-02 23:54:18,653::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10110::DEBUG::2012-12-02 23:54:18,654::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9a3094cc-568e-4223-9e09-c675cc67de3d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10110::DEBUG::2012-12-02 23:54:18,654::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10110::DEBUG::2012-12-02 23:54:18,654::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10110::DEBUG::2012-12-02 23:54:18,654::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9a3094cc-568e-4223-9e09-c675cc67de3d`::Granted request >Thread-10110::DEBUG::2012-12-02 23:54:18,654::task::794::TaskManager.Task::(resourceAcquired) Task=`389a4f83-e7d0-4840-bb5c-d705d65ed02a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10110::DEBUG::2012-12-02 23:54:18,654::task::957::TaskManager.Task::(_decref) Task=`389a4f83-e7d0-4840-bb5c-d705d65ed02a`::ref 1 aborting False >Thread-10110::DEBUG::2012-12-02 23:54:18,659::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10110::DEBUG::2012-12-02 23:54:18,659::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10110::DEBUG::2012-12-02 23:54:18,863::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10110::WARNING::2012-12-02 23:54:18,865::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10110::DEBUG::2012-12-02 23:54:18,865::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10110::WARNING::2012-12-02 23:54:18,899::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10110::DEBUG::2012-12-02 23:54:18,899::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10110::DEBUG::2012-12-02 23:54:18,900::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10110::DEBUG::2012-12-02 23:54:19,107::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10110::WARNING::2012-12-02 23:54:19,108::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10110::DEBUG::2012-12-02 23:54:19,108::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10110::WARNING::2012-12-02 23:54:19,137::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10110::INFO::2012-12-02 23:54:19,137::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10110::DEBUG::2012-12-02 23:54:19,137::task::1151::TaskManager.Task::(prepare) Task=`389a4f83-e7d0-4840-bb5c-d705d65ed02a`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10110::DEBUG::2012-12-02 23:54:19,138::task::568::TaskManager.Task::(_updateState) Task=`389a4f83-e7d0-4840-bb5c-d705d65ed02a`::moving from state preparing -> state finished >Thread-10110::DEBUG::2012-12-02 23:54:19,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10110::DEBUG::2012-12-02 23:54:19,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10110::DEBUG::2012-12-02 23:54:19,138::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10110::DEBUG::2012-12-02 23:54:19,138::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10110::DEBUG::2012-12-02 23:54:19,138::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10110::DEBUG::2012-12-02 23:54:19,139::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10110::DEBUG::2012-12-02 23:54:19,139::task::957::TaskManager.Task::(_decref) Task=`389a4f83-e7d0-4840-bb5c-d705d65ed02a`::ref 0 aborting False >Thread-10233::DEBUG::2012-12-02 23:54:19,965::task::568::TaskManager.Task::(_updateState) Task=`2021522e-d4ba-4912-a56e-e91fd1a61f51`::moving from state init -> state preparing >Thread-10233::INFO::2012-12-02 23:54:19,965::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10233::INFO::2012-12-02 23:54:19,966::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00837779045105', 'lastCheck': 1354485257.516422, 'code': 0, 'valid': True}} >Thread-10233::DEBUG::2012-12-02 23:54:19,966::task::1151::TaskManager.Task::(prepare) Task=`2021522e-d4ba-4912-a56e-e91fd1a61f51`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00837779045105', 'lastCheck': 1354485257.516422, 'code': 0, 'valid': True}} >Thread-10233::DEBUG::2012-12-02 23:54:19,966::task::568::TaskManager.Task::(_updateState) Task=`2021522e-d4ba-4912-a56e-e91fd1a61f51`::moving from state preparing -> state finished >Thread-10233::DEBUG::2012-12-02 23:54:19,966::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10233::DEBUG::2012-12-02 23:54:19,966::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10233::DEBUG::2012-12-02 23:54:19,966::task::957::TaskManager.Task::(_decref) Task=`2021522e-d4ba-4912-a56e-e91fd1a61f51`::ref 0 aborting False >Thread-10239::DEBUG::2012-12-02 23:54:29,394::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10239::DEBUG::2012-12-02 23:54:29,394::task::568::TaskManager.Task::(_updateState) Task=`88c689f4-83b5-4516-a6a1-ca5f81c10300`::moving from state init -> state preparing >Thread-10239::INFO::2012-12-02 23:54:29,395::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10239::INFO::2012-12-02 23:54:29,395::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10239::DEBUG::2012-12-02 23:54:29,395::task::1151::TaskManager.Task::(prepare) Task=`88c689f4-83b5-4516-a6a1-ca5f81c10300`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10239::DEBUG::2012-12-02 23:54:29,395::task::568::TaskManager.Task::(_updateState) Task=`88c689f4-83b5-4516-a6a1-ca5f81c10300`::moving from state preparing -> state finished >Thread-10239::DEBUG::2012-12-02 23:54:29,395::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10239::DEBUG::2012-12-02 23:54:29,395::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10239::DEBUG::2012-12-02 23:54:29,395::task::957::TaskManager.Task::(_decref) Task=`88c689f4-83b5-4516-a6a1-ca5f81c10300`::ref 0 aborting False >Thread-10240::DEBUG::2012-12-02 23:54:29,539::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10240::DEBUG::2012-12-02 23:54:29,539::task::568::TaskManager.Task::(_updateState) Task=`6b6ac27d-f0f8-4e30-b1f0-caf2c991c67b`::moving from state init -> state preparing >Thread-10240::INFO::2012-12-02 23:54:29,539::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10240::DEBUG::2012-12-02 23:54:29,539::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`0dbbc2e5-8f3b-4f21-a0e0-64de46619499`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10240::DEBUG::2012-12-02 23:54:29,540::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10240::DEBUG::2012-12-02 23:54:29,540::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10240::DEBUG::2012-12-02 23:54:29,540::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`0dbbc2e5-8f3b-4f21-a0e0-64de46619499`::Granted request >Thread-10240::DEBUG::2012-12-02 23:54:29,540::task::794::TaskManager.Task::(resourceAcquired) Task=`6b6ac27d-f0f8-4e30-b1f0-caf2c991c67b`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10240::DEBUG::2012-12-02 23:54:29,540::task::957::TaskManager.Task::(_decref) Task=`6b6ac27d-f0f8-4e30-b1f0-caf2c991c67b`::ref 1 aborting False >Thread-10240::DEBUG::2012-12-02 23:54:29,546::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10240::DEBUG::2012-12-02 23:54:29,546::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10240::DEBUG::2012-12-02 23:54:29,748::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10240::WARNING::2012-12-02 23:54:29,750::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10240::DEBUG::2012-12-02 23:54:29,750::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10240::WARNING::2012-12-02 23:54:29,780::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10240::DEBUG::2012-12-02 23:54:29,780::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10240::DEBUG::2012-12-02 23:54:29,780::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10240::DEBUG::2012-12-02 23:54:29,986::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10240::WARNING::2012-12-02 23:54:29,987::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10240::DEBUG::2012-12-02 23:54:29,988::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10240::WARNING::2012-12-02 23:54:30,017::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10240::INFO::2012-12-02 23:54:30,018::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10240::DEBUG::2012-12-02 23:54:30,018::task::1151::TaskManager.Task::(prepare) Task=`6b6ac27d-f0f8-4e30-b1f0-caf2c991c67b`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10240::DEBUG::2012-12-02 23:54:30,018::task::568::TaskManager.Task::(_updateState) Task=`6b6ac27d-f0f8-4e30-b1f0-caf2c991c67b`::moving from state preparing -> state finished >Thread-10240::DEBUG::2012-12-02 23:54:30,018::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10240::DEBUG::2012-12-02 23:54:30,019::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10240::DEBUG::2012-12-02 23:54:30,019::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10240::DEBUG::2012-12-02 23:54:30,019::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10240::DEBUG::2012-12-02 23:54:30,019::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10240::DEBUG::2012-12-02 23:54:30,019::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10240::DEBUG::2012-12-02 23:54:30,019::task::957::TaskManager.Task::(_decref) Task=`6b6ac27d-f0f8-4e30-b1f0-caf2c991c67b`::ref 0 aborting False >Thread-10363::DEBUG::2012-12-02 23:54:30,873::task::568::TaskManager.Task::(_updateState) Task=`1429a93b-12ca-40fb-a78b-6438ed62f33b`::moving from state init -> state preparing >Thread-10363::INFO::2012-12-02 23:54:30,873::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10363::INFO::2012-12-02 23:54:30,873::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00783681869507', 'lastCheck': 1354485267.535485, 'code': 0, 'valid': True}} >Thread-10363::DEBUG::2012-12-02 23:54:30,873::task::1151::TaskManager.Task::(prepare) Task=`1429a93b-12ca-40fb-a78b-6438ed62f33b`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00783681869507', 'lastCheck': 1354485267.535485, 'code': 0, 'valid': True}} >Thread-10363::DEBUG::2012-12-02 23:54:30,873::task::568::TaskManager.Task::(_updateState) Task=`1429a93b-12ca-40fb-a78b-6438ed62f33b`::moving from state preparing -> state finished >Thread-10363::DEBUG::2012-12-02 23:54:30,873::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10363::DEBUG::2012-12-02 23:54:30,873::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10363::DEBUG::2012-12-02 23:54:30,874::task::957::TaskManager.Task::(_decref) Task=`1429a93b-12ca-40fb-a78b-6438ed62f33b`::ref 0 aborting False >Thread-10369::DEBUG::2012-12-02 23:54:40,254::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10369::DEBUG::2012-12-02 23:54:40,255::task::568::TaskManager.Task::(_updateState) Task=`7aece915-a703-4c34-b77b-30a6dfbb434e`::moving from state init -> state preparing >Thread-10369::INFO::2012-12-02 23:54:40,255::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10369::INFO::2012-12-02 23:54:40,255::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10369::DEBUG::2012-12-02 23:54:40,255::task::1151::TaskManager.Task::(prepare) Task=`7aece915-a703-4c34-b77b-30a6dfbb434e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10369::DEBUG::2012-12-02 23:54:40,255::task::568::TaskManager.Task::(_updateState) Task=`7aece915-a703-4c34-b77b-30a6dfbb434e`::moving from state preparing -> state finished >Thread-10369::DEBUG::2012-12-02 23:54:40,256::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10369::DEBUG::2012-12-02 23:54:40,256::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10369::DEBUG::2012-12-02 23:54:40,256::task::957::TaskManager.Task::(_decref) Task=`7aece915-a703-4c34-b77b-30a6dfbb434e`::ref 0 aborting False >Thread-10370::DEBUG::2012-12-02 23:54:40,408::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10370::DEBUG::2012-12-02 23:54:40,408::task::568::TaskManager.Task::(_updateState) Task=`59e31d0a-8ca0-4e00-bad9-87d936858fab`::moving from state init -> state preparing >Thread-10370::INFO::2012-12-02 23:54:40,409::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10370::DEBUG::2012-12-02 23:54:40,409::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f5e40156-695a-4f29-b8f9-34cd6a3fc0c4`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10370::DEBUG::2012-12-02 23:54:40,409::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10370::DEBUG::2012-12-02 23:54:40,409::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10370::DEBUG::2012-12-02 23:54:40,409::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`f5e40156-695a-4f29-b8f9-34cd6a3fc0c4`::Granted request >Thread-10370::DEBUG::2012-12-02 23:54:40,410::task::794::TaskManager.Task::(resourceAcquired) Task=`59e31d0a-8ca0-4e00-bad9-87d936858fab`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10370::DEBUG::2012-12-02 23:54:40,410::task::957::TaskManager.Task::(_decref) Task=`59e31d0a-8ca0-4e00-bad9-87d936858fab`::ref 1 aborting False >Thread-10370::DEBUG::2012-12-02 23:54:40,416::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10370::DEBUG::2012-12-02 23:54:40,416::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10370::DEBUG::2012-12-02 23:54:40,620::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10370::WARNING::2012-12-02 23:54:40,622::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10370::DEBUG::2012-12-02 23:54:40,622::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10370::WARNING::2012-12-02 23:54:40,653::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10370::DEBUG::2012-12-02 23:54:40,653::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10370::DEBUG::2012-12-02 23:54:40,654::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10370::DEBUG::2012-12-02 23:54:40,859::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10370::WARNING::2012-12-02 23:54:40,861::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10370::DEBUG::2012-12-02 23:54:40,861::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10370::WARNING::2012-12-02 23:54:40,890::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10370::INFO::2012-12-02 23:54:40,890::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10370::DEBUG::2012-12-02 23:54:40,891::task::1151::TaskManager.Task::(prepare) Task=`59e31d0a-8ca0-4e00-bad9-87d936858fab`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10370::DEBUG::2012-12-02 23:54:40,891::task::568::TaskManager.Task::(_updateState) Task=`59e31d0a-8ca0-4e00-bad9-87d936858fab`::moving from state preparing -> state finished >Thread-10370::DEBUG::2012-12-02 23:54:40,891::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10370::DEBUG::2012-12-02 23:54:40,891::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10370::DEBUG::2012-12-02 23:54:40,891::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10370::DEBUG::2012-12-02 23:54:40,891::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10370::DEBUG::2012-12-02 23:54:40,892::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10370::DEBUG::2012-12-02 23:54:40,892::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10370::DEBUG::2012-12-02 23:54:40,892::task::957::TaskManager.Task::(_decref) Task=`59e31d0a-8ca0-4e00-bad9-87d936858fab`::ref 0 aborting False >Thread-10493::DEBUG::2012-12-02 23:54:41,778::task::568::TaskManager.Task::(_updateState) Task=`b10e88dd-8dc4-4539-916d-dd2bdcd5078e`::moving from state init -> state preparing >Thread-10493::INFO::2012-12-02 23:54:41,778::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10493::INFO::2012-12-02 23:54:41,778::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00792813301086', 'lastCheck': 1354485277.5536489, 'code': 0, 'valid': True}} >Thread-10493::DEBUG::2012-12-02 23:54:41,778::task::1151::TaskManager.Task::(prepare) Task=`b10e88dd-8dc4-4539-916d-dd2bdcd5078e`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00792813301086', 'lastCheck': 1354485277.5536489, 'code': 0, 'valid': True}} >Thread-10493::DEBUG::2012-12-02 23:54:41,778::task::568::TaskManager.Task::(_updateState) Task=`b10e88dd-8dc4-4539-916d-dd2bdcd5078e`::moving from state preparing -> state finished >Thread-10493::DEBUG::2012-12-02 23:54:41,779::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10493::DEBUG::2012-12-02 23:54:41,779::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10493::DEBUG::2012-12-02 23:54:41,779::task::957::TaskManager.Task::(_decref) Task=`b10e88dd-8dc4-4539-916d-dd2bdcd5078e`::ref 0 aborting False >Thread-10499::DEBUG::2012-12-02 23:54:51,119::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10499::DEBUG::2012-12-02 23:54:51,119::task::568::TaskManager.Task::(_updateState) Task=`5669e954-5812-48ad-bf69-5a219f461781`::moving from state init -> state preparing >Thread-10499::INFO::2012-12-02 23:54:51,120::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10499::INFO::2012-12-02 23:54:51,120::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10499::DEBUG::2012-12-02 23:54:51,120::task::1151::TaskManager.Task::(prepare) Task=`5669e954-5812-48ad-bf69-5a219f461781`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10499::DEBUG::2012-12-02 23:54:51,120::task::568::TaskManager.Task::(_updateState) Task=`5669e954-5812-48ad-bf69-5a219f461781`::moving from state preparing -> state finished >Thread-10499::DEBUG::2012-12-02 23:54:51,120::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10499::DEBUG::2012-12-02 23:54:51,120::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10499::DEBUG::2012-12-02 23:54:51,121::task::957::TaskManager.Task::(_decref) Task=`5669e954-5812-48ad-bf69-5a219f461781`::ref 0 aborting False >Thread-10500::DEBUG::2012-12-02 23:54:51,266::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10500::DEBUG::2012-12-02 23:54:51,266::task::568::TaskManager.Task::(_updateState) Task=`f54ad304-80db-4042-a441-abae13c3e34a`::moving from state init -> state preparing >Thread-10500::INFO::2012-12-02 23:54:51,266::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10500::DEBUG::2012-12-02 23:54:51,267::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d08ec140-139a-4240-ada0-9bbd1e6d31b1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10500::DEBUG::2012-12-02 23:54:51,267::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10500::DEBUG::2012-12-02 23:54:51,267::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10500::DEBUG::2012-12-02 23:54:51,267::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d08ec140-139a-4240-ada0-9bbd1e6d31b1`::Granted request >Thread-10500::DEBUG::2012-12-02 23:54:51,267::task::794::TaskManager.Task::(resourceAcquired) Task=`f54ad304-80db-4042-a441-abae13c3e34a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10500::DEBUG::2012-12-02 23:54:51,268::task::957::TaskManager.Task::(_decref) Task=`f54ad304-80db-4042-a441-abae13c3e34a`::ref 1 aborting False >Thread-10500::DEBUG::2012-12-02 23:54:51,274::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10500::DEBUG::2012-12-02 23:54:51,274::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10500::DEBUG::2012-12-02 23:54:51,478::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10500::WARNING::2012-12-02 23:54:51,479::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10500::DEBUG::2012-12-02 23:54:51,480::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10500::WARNING::2012-12-02 23:54:51,509::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10500::DEBUG::2012-12-02 23:54:51,510::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10500::DEBUG::2012-12-02 23:54:51,510::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10500::DEBUG::2012-12-02 23:54:51,716::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10500::WARNING::2012-12-02 23:54:51,718::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10500::DEBUG::2012-12-02 23:54:51,718::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10500::WARNING::2012-12-02 23:54:51,748::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10500::INFO::2012-12-02 23:54:51,748::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10500::DEBUG::2012-12-02 23:54:51,748::task::1151::TaskManager.Task::(prepare) Task=`f54ad304-80db-4042-a441-abae13c3e34a`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10500::DEBUG::2012-12-02 23:54:51,749::task::568::TaskManager.Task::(_updateState) Task=`f54ad304-80db-4042-a441-abae13c3e34a`::moving from state preparing -> state finished >Thread-10500::DEBUG::2012-12-02 23:54:51,749::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10500::DEBUG::2012-12-02 23:54:51,749::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10500::DEBUG::2012-12-02 23:54:51,749::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10500::DEBUG::2012-12-02 23:54:51,749::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10500::DEBUG::2012-12-02 23:54:51,749::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10500::DEBUG::2012-12-02 23:54:51,750::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10500::DEBUG::2012-12-02 23:54:51,750::task::957::TaskManager.Task::(_decref) Task=`f54ad304-80db-4042-a441-abae13c3e34a`::ref 0 aborting False >Thread-10623::DEBUG::2012-12-02 23:54:52,811::task::568::TaskManager.Task::(_updateState) Task=`94c39c94-a80c-469b-b281-1b2ddae7852a`::moving from state init -> state preparing >Thread-10623::INFO::2012-12-02 23:54:52,811::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10623::INFO::2012-12-02 23:54:52,811::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00812315940857', 'lastCheck': 1354485287.5732651, 'code': 0, 'valid': True}} >Thread-10623::DEBUG::2012-12-02 23:54:52,811::task::1151::TaskManager.Task::(prepare) Task=`94c39c94-a80c-469b-b281-1b2ddae7852a`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00812315940857', 'lastCheck': 1354485287.5732651, 'code': 0, 'valid': True}} >Thread-10623::DEBUG::2012-12-02 23:54:52,811::task::568::TaskManager.Task::(_updateState) Task=`94c39c94-a80c-469b-b281-1b2ddae7852a`::moving from state preparing -> state finished >Thread-10623::DEBUG::2012-12-02 23:54:52,811::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10623::DEBUG::2012-12-02 23:54:52,812::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10623::DEBUG::2012-12-02 23:54:52,812::task::957::TaskManager.Task::(_decref) Task=`94c39c94-a80c-469b-b281-1b2ddae7852a`::ref 0 aborting False >Thread-10629::DEBUG::2012-12-02 23:55:01,982::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10629::DEBUG::2012-12-02 23:55:01,983::task::568::TaskManager.Task::(_updateState) Task=`a9d40a53-4bf4-49d0-aa9e-c941520927e4`::moving from state init -> state preparing >Thread-10629::INFO::2012-12-02 23:55:01,983::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10629::INFO::2012-12-02 23:55:01,983::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10629::DEBUG::2012-12-02 23:55:01,984::task::1151::TaskManager.Task::(prepare) Task=`a9d40a53-4bf4-49d0-aa9e-c941520927e4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10629::DEBUG::2012-12-02 23:55:01,984::task::568::TaskManager.Task::(_updateState) Task=`a9d40a53-4bf4-49d0-aa9e-c941520927e4`::moving from state preparing -> state finished >Thread-10629::DEBUG::2012-12-02 23:55:01,984::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10629::DEBUG::2012-12-02 23:55:01,984::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10629::DEBUG::2012-12-02 23:55:01,984::task::957::TaskManager.Task::(_decref) Task=`a9d40a53-4bf4-49d0-aa9e-c941520927e4`::ref 0 aborting False >Thread-10630::DEBUG::2012-12-02 23:55:02,128::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10630::DEBUG::2012-12-02 23:55:02,128::task::568::TaskManager.Task::(_updateState) Task=`64bb024e-8ee1-4fe7-b5e6-45683738e35a`::moving from state init -> state preparing >Thread-10630::INFO::2012-12-02 23:55:02,128::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10630::DEBUG::2012-12-02 23:55:02,129::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a10b5257-3dd9-4167-9d9b-94cddeda270d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10630::DEBUG::2012-12-02 23:55:02,129::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10630::DEBUG::2012-12-02 23:55:02,129::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10630::DEBUG::2012-12-02 23:55:02,129::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`a10b5257-3dd9-4167-9d9b-94cddeda270d`::Granted request >Thread-10630::DEBUG::2012-12-02 23:55:02,129::task::794::TaskManager.Task::(resourceAcquired) Task=`64bb024e-8ee1-4fe7-b5e6-45683738e35a`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10630::DEBUG::2012-12-02 23:55:02,130::task::957::TaskManager.Task::(_decref) Task=`64bb024e-8ee1-4fe7-b5e6-45683738e35a`::ref 1 aborting False >Thread-10630::DEBUG::2012-12-02 23:55:02,137::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10630::DEBUG::2012-12-02 23:55:02,137::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10630::DEBUG::2012-12-02 23:55:02,345::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10630::WARNING::2012-12-02 23:55:02,346::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10630::DEBUG::2012-12-02 23:55:02,346::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10630::WARNING::2012-12-02 23:55:02,375::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10630::DEBUG::2012-12-02 23:55:02,376::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10630::DEBUG::2012-12-02 23:55:02,376::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10630::DEBUG::2012-12-02 23:55:02,584::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10630::WARNING::2012-12-02 23:55:02,585::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10630::DEBUG::2012-12-02 23:55:02,585::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10630::WARNING::2012-12-02 23:55:02,615::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10630::INFO::2012-12-02 23:55:02,615::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10630::DEBUG::2012-12-02 23:55:02,615::task::1151::TaskManager.Task::(prepare) Task=`64bb024e-8ee1-4fe7-b5e6-45683738e35a`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}}} >Thread-10630::DEBUG::2012-12-02 23:55:02,615::task::568::TaskManager.Task::(_updateState) Task=`64bb024e-8ee1-4fe7-b5e6-45683738e35a`::moving from state preparing -> state finished >Thread-10630::DEBUG::2012-12-02 23:55:02,616::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10630::DEBUG::2012-12-02 23:55:02,616::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10630::DEBUG::2012-12-02 23:55:02,616::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10630::DEBUG::2012-12-02 23:55:02,616::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10630::DEBUG::2012-12-02 23:55:02,616::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10630::DEBUG::2012-12-02 23:55:02,616::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10630::DEBUG::2012-12-02 23:55:02,617::task::957::TaskManager.Task::(_decref) Task=`64bb024e-8ee1-4fe7-b5e6-45683738e35a`::ref 0 aborting False >Thread-10753::DEBUG::2012-12-02 23:55:03,728::task::568::TaskManager.Task::(_updateState) Task=`7c6cbaf5-0809-4eb8-b5e3-5345e51677d1`::moving from state init -> state preparing >Thread-10753::INFO::2012-12-02 23:55:03,728::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10753::INFO::2012-12-02 23:55:03,728::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00905084609985', 'lastCheck': 1354485297.593019, 'code': 0, 'valid': True}} >Thread-10753::DEBUG::2012-12-02 23:55:03,728::task::1151::TaskManager.Task::(prepare) Task=`7c6cbaf5-0809-4eb8-b5e3-5345e51677d1`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0.00905084609985', 'lastCheck': 1354485297.593019, 'code': 0, 'valid': True}} >Thread-10753::DEBUG::2012-12-02 23:55:03,728::task::568::TaskManager.Task::(_updateState) Task=`7c6cbaf5-0809-4eb8-b5e3-5345e51677d1`::moving from state preparing -> state finished >Thread-10753::DEBUG::2012-12-02 23:55:03,728::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10753::DEBUG::2012-12-02 23:55:03,729::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10753::DEBUG::2012-12-02 23:55:03,729::task::957::TaskManager.Task::(_decref) Task=`7c6cbaf5-0809-4eb8-b5e3-5345e51677d1`::ref 0 aborting False >Thread-2443::DEBUG::2012-12-02 23:55:07,601::fileSD::405::Storage.StorageDomain::(selftest) Unmounting stale file system /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328 >Thread-2443::DEBUG::2012-12-02 23:55:07,607::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/umount /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328' (cwd None) >Thread-2443::ERROR::2012-12-02 23:55:07,622::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 406, in selftest > mount.getMountFromTarget(self.mountpoint).umount() > File "/usr/share/vdsm/storage/mount.py", line 218, in umount > return self._runcmd(cmd, timeout) > File "/usr/share/vdsm/storage/mount.py", line 206, in _runcmd > raise MountError(rc, ";".join((out, err))) >MountError: (16, ';umount.nfs: /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328: device is busy\numount.nfs: /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328: device is busy\n') >Thread-2443::DEBUG::2012-12-02 23:55:07,623::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `5971512c-ee51-475e-b956-3872c38ce193` changed its status to Invalid >Thread-10756::DEBUG::2012-12-02 23:55:07,623::misc::1183::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Emitting event >Thread-10756::DEBUG::2012-12-02 23:55:07,624::misc::1194::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Calling registered method `<functools.partial object at 0x7fed44451d08>` >Thread-10756::DEBUG::2012-12-02 23:55:07,624::misc::1204::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Event emitted >Thread-10761::DEBUG::2012-12-02 23:55:13,119::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10761::DEBUG::2012-12-02 23:55:13,119::task::568::TaskManager.Task::(_updateState) Task=`0fd5b00d-99bf-4fa1-bfd8-a517588a112a`::moving from state init -> state preparing >Thread-10761::INFO::2012-12-02 23:55:13,120::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10761::INFO::2012-12-02 23:55:13,120::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10761::DEBUG::2012-12-02 23:55:13,120::task::1151::TaskManager.Task::(prepare) Task=`0fd5b00d-99bf-4fa1-bfd8-a517588a112a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10761::DEBUG::2012-12-02 23:55:13,120::task::568::TaskManager.Task::(_updateState) Task=`0fd5b00d-99bf-4fa1-bfd8-a517588a112a`::moving from state preparing -> state finished >Thread-10761::DEBUG::2012-12-02 23:55:13,120::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10761::DEBUG::2012-12-02 23:55:13,120::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10761::DEBUG::2012-12-02 23:55:13,120::task::957::TaskManager.Task::(_decref) Task=`0fd5b00d-99bf-4fa1-bfd8-a517588a112a`::ref 0 aborting False >Thread-10762::DEBUG::2012-12-02 23:55:13,256::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10762::DEBUG::2012-12-02 23:55:13,257::task::568::TaskManager.Task::(_updateState) Task=`d41c72e6-0da4-4208-be9e-246bfbc363ca`::moving from state init -> state preparing >Thread-10762::INFO::2012-12-02 23:55:13,257::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10762::DEBUG::2012-12-02 23:55:13,257::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d97b24c7-b654-4df4-bcf1-52b7ff6e15ea`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10762::DEBUG::2012-12-02 23:55:13,257::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10762::DEBUG::2012-12-02 23:55:13,257::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10762::DEBUG::2012-12-02 23:55:13,258::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`d97b24c7-b654-4df4-bcf1-52b7ff6e15ea`::Granted request >Thread-10762::DEBUG::2012-12-02 23:55:13,258::task::794::TaskManager.Task::(resourceAcquired) Task=`d41c72e6-0da4-4208-be9e-246bfbc363ca`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10762::DEBUG::2012-12-02 23:55:13,258::task::957::TaskManager.Task::(_decref) Task=`d41c72e6-0da4-4208-be9e-246bfbc363ca`::ref 1 aborting False >Thread-10762::DEBUG::2012-12-02 23:55:13,265::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10762::DEBUG::2012-12-02 23:55:13,265::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10762::DEBUG::2012-12-02 23:55:13,473::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10762::WARNING::2012-12-02 23:55:13,475::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10762::DEBUG::2012-12-02 23:55:13,475::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10762::WARNING::2012-12-02 23:55:13,504::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10762::DEBUG::2012-12-02 23:55:13,505::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10762::DEBUG::2012-12-02 23:55:13,505::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10762::DEBUG::2012-12-02 23:55:13,711::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10762::WARNING::2012-12-02 23:55:13,712::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10762::DEBUG::2012-12-02 23:55:13,713::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10762::WARNING::2012-12-02 23:55:13,742::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10762::INFO::2012-12-02 23:55:13,743::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-10762::DEBUG::2012-12-02 23:55:13,743::task::1151::TaskManager.Task::(prepare) Task=`d41c72e6-0da4-4208-be9e-246bfbc363ca`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-10762::DEBUG::2012-12-02 23:55:13,743::task::568::TaskManager.Task::(_updateState) Task=`d41c72e6-0da4-4208-be9e-246bfbc363ca`::moving from state preparing -> state finished >Thread-10762::DEBUG::2012-12-02 23:55:13,743::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10762::DEBUG::2012-12-02 23:55:13,744::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10762::DEBUG::2012-12-02 23:55:13,744::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10762::DEBUG::2012-12-02 23:55:13,744::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10762::DEBUG::2012-12-02 23:55:13,744::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10762::DEBUG::2012-12-02 23:55:13,744::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10762::DEBUG::2012-12-02 23:55:13,744::task::957::TaskManager.Task::(_decref) Task=`d41c72e6-0da4-4208-be9e-246bfbc363ca`::ref 0 aborting False >Thread-10885::DEBUG::2012-12-02 23:55:14,636::task::568::TaskManager.Task::(_updateState) Task=`f7406a5d-91a4-4d8f-8fd6-22ae909d52f7`::moving from state init -> state preparing >Thread-10885::INFO::2012-12-02 23:55:14,636::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10885::INFO::2012-12-02 23:55:14,636::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485307.6234739, 'code': 200, 'valid': False}} >Thread-10885::DEBUG::2012-12-02 23:55:14,636::task::1151::TaskManager.Task::(prepare) Task=`f7406a5d-91a4-4d8f-8fd6-22ae909d52f7`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485307.6234739, 'code': 200, 'valid': False}} >Thread-10885::DEBUG::2012-12-02 23:55:14,637::task::568::TaskManager.Task::(_updateState) Task=`f7406a5d-91a4-4d8f-8fd6-22ae909d52f7`::moving from state preparing -> state finished >Thread-10885::DEBUG::2012-12-02 23:55:14,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10885::DEBUG::2012-12-02 23:55:14,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10885::DEBUG::2012-12-02 23:55:14,637::task::957::TaskManager.Task::(_decref) Task=`f7406a5d-91a4-4d8f-8fd6-22ae909d52f7`::ref 0 aborting False >Thread-2443::ERROR::2012-12-02 23:55:17,633::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193' >Thread-10891::DEBUG::2012-12-02 23:55:23,980::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10891::DEBUG::2012-12-02 23:55:23,980::task::568::TaskManager.Task::(_updateState) Task=`30e6fd55-a127-4b16-8b99-7e631d548044`::moving from state init -> state preparing >Thread-10891::INFO::2012-12-02 23:55:23,980::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10891::INFO::2012-12-02 23:55:23,980::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10891::DEBUG::2012-12-02 23:55:23,980::task::1151::TaskManager.Task::(prepare) Task=`30e6fd55-a127-4b16-8b99-7e631d548044`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-10891::DEBUG::2012-12-02 23:55:23,981::task::568::TaskManager.Task::(_updateState) Task=`30e6fd55-a127-4b16-8b99-7e631d548044`::moving from state preparing -> state finished >Thread-10891::DEBUG::2012-12-02 23:55:23,981::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10891::DEBUG::2012-12-02 23:55:23,981::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10891::DEBUG::2012-12-02 23:55:23,981::task::957::TaskManager.Task::(_decref) Task=`30e6fd55-a127-4b16-8b99-7e631d548044`::ref 0 aborting False >Thread-10892::DEBUG::2012-12-02 23:55:24,116::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-10892::DEBUG::2012-12-02 23:55:24,116::task::568::TaskManager.Task::(_updateState) Task=`b943cedf-55e4-4e54-a7da-52d14bd17bc1`::moving from state init -> state preparing >Thread-10892::INFO::2012-12-02 23:55:24,116::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-10892::DEBUG::2012-12-02 23:55:24,117::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9fdb91a5-63be-4061-a154-cd47f171e40a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-10892::DEBUG::2012-12-02 23:55:24,117::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-10892::DEBUG::2012-12-02 23:55:24,117::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-10892::DEBUG::2012-12-02 23:55:24,117::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`9fdb91a5-63be-4061-a154-cd47f171e40a`::Granted request >Thread-10892::DEBUG::2012-12-02 23:55:24,117::task::794::TaskManager.Task::(resourceAcquired) Task=`b943cedf-55e4-4e54-a7da-52d14bd17bc1`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-10892::DEBUG::2012-12-02 23:55:24,118::task::957::TaskManager.Task::(_decref) Task=`b943cedf-55e4-4e54-a7da-52d14bd17bc1`::ref 1 aborting False >Thread-10892::DEBUG::2012-12-02 23:55:24,124::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10892::DEBUG::2012-12-02 23:55:24,124::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-10892::DEBUG::2012-12-02 23:55:24,330::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-10892::WARNING::2012-12-02 23:55:24,331::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-10892::DEBUG::2012-12-02 23:55:24,331::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10892::WARNING::2012-12-02 23:55:24,366::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-10892::DEBUG::2012-12-02 23:55:24,366::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-10892::DEBUG::2012-12-02 23:55:24,367::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-10892::DEBUG::2012-12-02 23:55:24,572::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-10892::WARNING::2012-12-02 23:55:24,573::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-10892::DEBUG::2012-12-02 23:55:24,574::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10892::WARNING::2012-12-02 23:55:24,603::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-10892::INFO::2012-12-02 23:55:24,603::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-10892::DEBUG::2012-12-02 23:55:24,603::task::1151::TaskManager.Task::(prepare) Task=`b943cedf-55e4-4e54-a7da-52d14bd17bc1`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-10892::DEBUG::2012-12-02 23:55:24,603::task::568::TaskManager.Task::(_updateState) Task=`b943cedf-55e4-4e54-a7da-52d14bd17bc1`::moving from state preparing -> state finished >Thread-10892::DEBUG::2012-12-02 23:55:24,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-10892::DEBUG::2012-12-02 23:55:24,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10892::DEBUG::2012-12-02 23:55:24,604::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-10892::DEBUG::2012-12-02 23:55:24,604::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-10892::DEBUG::2012-12-02 23:55:24,604::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-10892::DEBUG::2012-12-02 23:55:24,604::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-10892::DEBUG::2012-12-02 23:55:24,605::task::957::TaskManager.Task::(_decref) Task=`b943cedf-55e4-4e54-a7da-52d14bd17bc1`::ref 0 aborting False >Thread-11015::DEBUG::2012-12-02 23:55:25,547::task::568::TaskManager.Task::(_updateState) Task=`a1d66e71-035b-4323-8a83-53fb739c6b31`::moving from state init -> state preparing >Thread-11015::INFO::2012-12-02 23:55:25,548::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11015::INFO::2012-12-02 23:55:25,548::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485317.6339059, 'code': 200, 'valid': False}} >Thread-11015::DEBUG::2012-12-02 23:55:25,548::task::1151::TaskManager.Task::(prepare) Task=`a1d66e71-035b-4323-8a83-53fb739c6b31`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485317.6339059, 'code': 200, 'valid': False}} >Thread-11015::DEBUG::2012-12-02 23:55:25,548::task::568::TaskManager.Task::(_updateState) Task=`a1d66e71-035b-4323-8a83-53fb739c6b31`::moving from state preparing -> state finished >Thread-11015::DEBUG::2012-12-02 23:55:25,548::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11015::DEBUG::2012-12-02 23:55:25,548::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11015::DEBUG::2012-12-02 23:55:25,548::task::957::TaskManager.Task::(_decref) Task=`a1d66e71-035b-4323-8a83-53fb739c6b31`::ref 0 aborting False >Thread-2443::ERROR::2012-12-02 23:55:27,640::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193' >Thread-11021::DEBUG::2012-12-02 23:55:34,839::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11021::DEBUG::2012-12-02 23:55:34,840::task::568::TaskManager.Task::(_updateState) Task=`07489188-4970-4321-bed4-4675a69f0129`::moving from state init -> state preparing >Thread-11021::INFO::2012-12-02 23:55:34,840::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11021::INFO::2012-12-02 23:55:34,840::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11021::DEBUG::2012-12-02 23:55:34,840::task::1151::TaskManager.Task::(prepare) Task=`07489188-4970-4321-bed4-4675a69f0129`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11021::DEBUG::2012-12-02 23:55:34,840::task::568::TaskManager.Task::(_updateState) Task=`07489188-4970-4321-bed4-4675a69f0129`::moving from state preparing -> state finished >Thread-11021::DEBUG::2012-12-02 23:55:34,840::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11021::DEBUG::2012-12-02 23:55:34,841::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11021::DEBUG::2012-12-02 23:55:34,841::task::957::TaskManager.Task::(_decref) Task=`07489188-4970-4321-bed4-4675a69f0129`::ref 0 aborting False >Thread-11022::DEBUG::2012-12-02 23:55:34,977::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11022::DEBUG::2012-12-02 23:55:34,978::task::568::TaskManager.Task::(_updateState) Task=`1ffccbc5-9d52-4631-a63e-5f576fb170c3`::moving from state init -> state preparing >Thread-11022::INFO::2012-12-02 23:55:34,978::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11022::DEBUG::2012-12-02 23:55:34,978::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`da12a4c4-1234-4c3d-ad07-47d6ebf393b8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-11022::DEBUG::2012-12-02 23:55:34,978::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-11022::DEBUG::2012-12-02 23:55:34,979::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-11022::DEBUG::2012-12-02 23:55:34,979::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`da12a4c4-1234-4c3d-ad07-47d6ebf393b8`::Granted request >Thread-11022::DEBUG::2012-12-02 23:55:34,979::task::794::TaskManager.Task::(resourceAcquired) Task=`1ffccbc5-9d52-4631-a63e-5f576fb170c3`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-11022::DEBUG::2012-12-02 23:55:34,979::task::957::TaskManager.Task::(_decref) Task=`1ffccbc5-9d52-4631-a63e-5f576fb170c3`::ref 1 aborting False >Thread-11022::DEBUG::2012-12-02 23:55:34,985::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11022::DEBUG::2012-12-02 23:55:34,986::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-11022::DEBUG::2012-12-02 23:55:35,191::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-11022::WARNING::2012-12-02 23:55:35,192::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-11022::DEBUG::2012-12-02 23:55:35,192::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11022::WARNING::2012-12-02 23:55:35,222::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-11022::DEBUG::2012-12-02 23:55:35,222::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11022::DEBUG::2012-12-02 23:55:35,222::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-11022::DEBUG::2012-12-02 23:55:35,428::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-11022::WARNING::2012-12-02 23:55:35,429::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-11022::DEBUG::2012-12-02 23:55:35,430::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11022::WARNING::2012-12-02 23:55:35,459::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-11022::INFO::2012-12-02 23:55:35,459::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11022::DEBUG::2012-12-02 23:55:35,459::task::1151::TaskManager.Task::(prepare) Task=`1ffccbc5-9d52-4631-a63e-5f576fb170c3`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11022::DEBUG::2012-12-02 23:55:35,459::task::568::TaskManager.Task::(_updateState) Task=`1ffccbc5-9d52-4631-a63e-5f576fb170c3`::moving from state preparing -> state finished >Thread-11022::DEBUG::2012-12-02 23:55:35,460::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-11022::DEBUG::2012-12-02 23:55:35,460::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11022::DEBUG::2012-12-02 23:55:35,460::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-11022::DEBUG::2012-12-02 23:55:35,460::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-11022::DEBUG::2012-12-02 23:55:35,460::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-11022::DEBUG::2012-12-02 23:55:35,460::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-11022::DEBUG::2012-12-02 23:55:35,461::task::957::TaskManager.Task::(_decref) Task=`1ffccbc5-9d52-4631-a63e-5f576fb170c3`::ref 0 aborting False >Thread-11145::DEBUG::2012-12-02 23:55:36,458::task::568::TaskManager.Task::(_updateState) Task=`20ee3e01-f34d-482b-82c8-f00521cd331a`::moving from state init -> state preparing >Thread-11145::INFO::2012-12-02 23:55:36,458::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11145::INFO::2012-12-02 23:55:36,459::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485327.6407421, 'code': 200, 'valid': False}} >Thread-11145::DEBUG::2012-12-02 23:55:36,459::task::1151::TaskManager.Task::(prepare) Task=`20ee3e01-f34d-482b-82c8-f00521cd331a`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485327.6407421, 'code': 200, 'valid': False}} >Thread-11145::DEBUG::2012-12-02 23:55:36,459::task::568::TaskManager.Task::(_updateState) Task=`20ee3e01-f34d-482b-82c8-f00521cd331a`::moving from state preparing -> state finished >Thread-11145::DEBUG::2012-12-02 23:55:36,459::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11145::DEBUG::2012-12-02 23:55:36,459::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11145::DEBUG::2012-12-02 23:55:36,459::task::957::TaskManager.Task::(_decref) Task=`20ee3e01-f34d-482b-82c8-f00521cd331a`::ref 0 aborting False >Thread-2443::ERROR::2012-12-02 23:55:37,647::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193' >Thread-11151::DEBUG::2012-12-02 23:55:45,691::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11151::DEBUG::2012-12-02 23:55:45,692::task::568::TaskManager.Task::(_updateState) Task=`3efd35e3-4127-425b-a2b5-8f1affb343bc`::moving from state init -> state preparing >Thread-11151::INFO::2012-12-02 23:55:45,692::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11151::INFO::2012-12-02 23:55:45,692::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11151::DEBUG::2012-12-02 23:55:45,692::task::1151::TaskManager.Task::(prepare) Task=`3efd35e3-4127-425b-a2b5-8f1affb343bc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11151::DEBUG::2012-12-02 23:55:45,692::task::568::TaskManager.Task::(_updateState) Task=`3efd35e3-4127-425b-a2b5-8f1affb343bc`::moving from state preparing -> state finished >Thread-11151::DEBUG::2012-12-02 23:55:45,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11151::DEBUG::2012-12-02 23:55:45,692::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11151::DEBUG::2012-12-02 23:55:45,693::task::957::TaskManager.Task::(_decref) Task=`3efd35e3-4127-425b-a2b5-8f1affb343bc`::ref 0 aborting False >Thread-11152::DEBUG::2012-12-02 23:55:45,844::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11152::DEBUG::2012-12-02 23:55:45,844::task::568::TaskManager.Task::(_updateState) Task=`35b0f299-cc83-4410-9e35-2ab53d9bf0ee`::moving from state init -> state preparing >Thread-11152::INFO::2012-12-02 23:55:45,845::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11152::DEBUG::2012-12-02 23:55:45,845::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e141ed0d-1032-47b5-ade7-122d709fbdb1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-11152::DEBUG::2012-12-02 23:55:45,845::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-11152::DEBUG::2012-12-02 23:55:45,845::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-11152::DEBUG::2012-12-02 23:55:45,845::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e141ed0d-1032-47b5-ade7-122d709fbdb1`::Granted request >Thread-11152::DEBUG::2012-12-02 23:55:45,845::task::794::TaskManager.Task::(resourceAcquired) Task=`35b0f299-cc83-4410-9e35-2ab53d9bf0ee`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-11152::DEBUG::2012-12-02 23:55:45,846::task::957::TaskManager.Task::(_decref) Task=`35b0f299-cc83-4410-9e35-2ab53d9bf0ee`::ref 1 aborting False >Thread-11152::DEBUG::2012-12-02 23:55:45,850::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11152::DEBUG::2012-12-02 23:55:45,850::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-11152::DEBUG::2012-12-02 23:55:46,056::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-11152::WARNING::2012-12-02 23:55:46,057::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-11152::DEBUG::2012-12-02 23:55:46,057::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11152::WARNING::2012-12-02 23:55:46,088::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-11152::DEBUG::2012-12-02 23:55:46,089::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11152::DEBUG::2012-12-02 23:55:46,089::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-11152::DEBUG::2012-12-02 23:55:46,296::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-11152::WARNING::2012-12-02 23:55:46,297::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-11152::DEBUG::2012-12-02 23:55:46,297::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11152::WARNING::2012-12-02 23:55:46,327::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-11152::INFO::2012-12-02 23:55:46,327::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11152::DEBUG::2012-12-02 23:55:46,327::task::1151::TaskManager.Task::(prepare) Task=`35b0f299-cc83-4410-9e35-2ab53d9bf0ee`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11152::DEBUG::2012-12-02 23:55:46,328::task::568::TaskManager.Task::(_updateState) Task=`35b0f299-cc83-4410-9e35-2ab53d9bf0ee`::moving from state preparing -> state finished >Thread-11152::DEBUG::2012-12-02 23:55:46,328::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-11152::DEBUG::2012-12-02 23:55:46,328::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11152::DEBUG::2012-12-02 23:55:46,328::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-11152::DEBUG::2012-12-02 23:55:46,328::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-11152::DEBUG::2012-12-02 23:55:46,328::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-11152::DEBUG::2012-12-02 23:55:46,329::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-11152::DEBUG::2012-12-02 23:55:46,329::task::957::TaskManager.Task::(_decref) Task=`35b0f299-cc83-4410-9e35-2ab53d9bf0ee`::ref 0 aborting False >Thread-11275::DEBUG::2012-12-02 23:55:47,373::task::568::TaskManager.Task::(_updateState) Task=`2d54533f-a291-4b3f-ac9e-9638609f628e`::moving from state init -> state preparing >Thread-11275::INFO::2012-12-02 23:55:47,373::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11275::INFO::2012-12-02 23:55:47,373::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485337.647624, 'code': 200, 'valid': False}} >Thread-11275::DEBUG::2012-12-02 23:55:47,373::task::1151::TaskManager.Task::(prepare) Task=`2d54533f-a291-4b3f-ac9e-9638609f628e`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485337.647624, 'code': 200, 'valid': False}} >Thread-11275::DEBUG::2012-12-02 23:55:47,373::task::568::TaskManager.Task::(_updateState) Task=`2d54533f-a291-4b3f-ac9e-9638609f628e`::moving from state preparing -> state finished >Thread-11275::DEBUG::2012-12-02 23:55:47,373::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11275::DEBUG::2012-12-02 23:55:47,373::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11275::DEBUG::2012-12-02 23:55:47,374::task::957::TaskManager.Task::(_decref) Task=`2d54533f-a291-4b3f-ac9e-9638609f628e`::ref 0 aborting False >Thread-2443::ERROR::2012-12-02 23:55:47,654::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193' >Thread-11281::DEBUG::2012-12-02 23:55:56,551::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11281::DEBUG::2012-12-02 23:55:56,552::task::568::TaskManager.Task::(_updateState) Task=`41cc3df5-6b1a-4f50-84ce-f6610e185399`::moving from state init -> state preparing >Thread-11281::INFO::2012-12-02 23:55:56,552::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11281::INFO::2012-12-02 23:55:56,552::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11281::DEBUG::2012-12-02 23:55:56,552::task::1151::TaskManager.Task::(prepare) Task=`41cc3df5-6b1a-4f50-84ce-f6610e185399`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11281::DEBUG::2012-12-02 23:55:56,552::task::568::TaskManager.Task::(_updateState) Task=`41cc3df5-6b1a-4f50-84ce-f6610e185399`::moving from state preparing -> state finished >Thread-11281::DEBUG::2012-12-02 23:55:56,552::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11281::DEBUG::2012-12-02 23:55:56,552::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11281::DEBUG::2012-12-02 23:55:56,553::task::957::TaskManager.Task::(_decref) Task=`41cc3df5-6b1a-4f50-84ce-f6610e185399`::ref 0 aborting False >Thread-11282::DEBUG::2012-12-02 23:55:56,691::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11282::DEBUG::2012-12-02 23:55:56,691::task::568::TaskManager.Task::(_updateState) Task=`5a4e2f03-b34b-4d80-97d5-a1f02f4dce39`::moving from state init -> state preparing >Thread-11282::INFO::2012-12-02 23:55:56,691::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11282::DEBUG::2012-12-02 23:55:56,691::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ddea7e55-8ae2-4e30-80f4-dec6e380d713`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-11282::DEBUG::2012-12-02 23:55:56,692::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-11282::DEBUG::2012-12-02 23:55:56,692::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-11282::DEBUG::2012-12-02 23:55:56,692::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ddea7e55-8ae2-4e30-80f4-dec6e380d713`::Granted request >Thread-11282::DEBUG::2012-12-02 23:55:56,692::task::794::TaskManager.Task::(resourceAcquired) Task=`5a4e2f03-b34b-4d80-97d5-a1f02f4dce39`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-11282::DEBUG::2012-12-02 23:55:56,692::task::957::TaskManager.Task::(_decref) Task=`5a4e2f03-b34b-4d80-97d5-a1f02f4dce39`::ref 1 aborting False >Thread-11282::DEBUG::2012-12-02 23:55:56,698::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11282::DEBUG::2012-12-02 23:55:56,698::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-11282::DEBUG::2012-12-02 23:55:56,903::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-11282::WARNING::2012-12-02 23:55:56,905::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-11282::DEBUG::2012-12-02 23:55:56,905::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11282::WARNING::2012-12-02 23:55:56,934::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-11282::DEBUG::2012-12-02 23:55:56,935::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11282::DEBUG::2012-12-02 23:55:56,935::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-11282::DEBUG::2012-12-02 23:55:57,143::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-11282::WARNING::2012-12-02 23:55:57,144::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-11282::DEBUG::2012-12-02 23:55:57,145::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11282::WARNING::2012-12-02 23:55:57,172::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-11282::INFO::2012-12-02 23:55:57,173::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11282::DEBUG::2012-12-02 23:55:57,173::task::1151::TaskManager.Task::(prepare) Task=`5a4e2f03-b34b-4d80-97d5-a1f02f4dce39`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11282::DEBUG::2012-12-02 23:55:57,173::task::568::TaskManager.Task::(_updateState) Task=`5a4e2f03-b34b-4d80-97d5-a1f02f4dce39`::moving from state preparing -> state finished >Thread-11282::DEBUG::2012-12-02 23:55:57,173::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-11282::DEBUG::2012-12-02 23:55:57,173::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11282::DEBUG::2012-12-02 23:55:57,174::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-11282::DEBUG::2012-12-02 23:55:57,174::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-11282::DEBUG::2012-12-02 23:55:57,174::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-11282::DEBUG::2012-12-02 23:55:57,174::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-11282::DEBUG::2012-12-02 23:55:57,174::task::957::TaskManager.Task::(_decref) Task=`5a4e2f03-b34b-4d80-97d5-a1f02f4dce39`::ref 0 aborting False >Thread-2443::ERROR::2012-12-02 23:55:57,660::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193' >Thread-11405::DEBUG::2012-12-02 23:55:58,276::task::568::TaskManager.Task::(_updateState) Task=`ebf6c278-f103-4265-af43-156db169fa8e`::moving from state init -> state preparing >Thread-11405::INFO::2012-12-02 23:55:58,276::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11405::INFO::2012-12-02 23:55:58,277::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485357.6612101, 'code': 200, 'valid': False}} >Thread-11405::DEBUG::2012-12-02 23:55:58,277::task::1151::TaskManager.Task::(prepare) Task=`ebf6c278-f103-4265-af43-156db169fa8e`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485357.6612101, 'code': 200, 'valid': False}} >Thread-11405::DEBUG::2012-12-02 23:55:58,277::task::568::TaskManager.Task::(_updateState) Task=`ebf6c278-f103-4265-af43-156db169fa8e`::moving from state preparing -> state finished >Thread-11405::DEBUG::2012-12-02 23:55:58,277::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11405::DEBUG::2012-12-02 23:55:58,277::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11405::DEBUG::2012-12-02 23:55:58,277::task::957::TaskManager.Task::(_decref) Task=`ebf6c278-f103-4265-af43-156db169fa8e`::ref 0 aborting False >Thread-11411::DEBUG::2012-12-02 23:56:07,406::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11411::DEBUG::2012-12-02 23:56:07,407::task::568::TaskManager.Task::(_updateState) Task=`7200b1ce-efc8-4665-8cb5-47663cf3de65`::moving from state init -> state preparing >Thread-11411::INFO::2012-12-02 23:56:07,407::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11411::INFO::2012-12-02 23:56:07,407::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11411::DEBUG::2012-12-02 23:56:07,407::task::1151::TaskManager.Task::(prepare) Task=`7200b1ce-efc8-4665-8cb5-47663cf3de65`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 0}} >Thread-11411::DEBUG::2012-12-02 23:56:07,407::task::568::TaskManager.Task::(_updateState) Task=`7200b1ce-efc8-4665-8cb5-47663cf3de65`::moving from state preparing -> state finished >Thread-11411::DEBUG::2012-12-02 23:56:07,407::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11411::DEBUG::2012-12-02 23:56:07,408::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11411::DEBUG::2012-12-02 23:56:07,408::task::957::TaskManager.Task::(_decref) Task=`7200b1ce-efc8-4665-8cb5-47663cf3de65`::ref 0 aborting False >Thread-11412::DEBUG::2012-12-02 23:56:07,561::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-11412::DEBUG::2012-12-02 23:56:07,561::task::568::TaskManager.Task::(_updateState) Task=`31c8af6b-4714-45e1-9f90-9cb6eb9a1ca6`::moving from state init -> state preparing >Thread-11412::INFO::2012-12-02 23:56:07,561::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-11412::DEBUG::2012-12-02 23:56:07,562::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`64030e3a-0e62-40b5-9da7-ac2cdd3af540`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-11412::DEBUG::2012-12-02 23:56:07,562::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-11412::DEBUG::2012-12-02 23:56:07,562::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-11412::DEBUG::2012-12-02 23:56:07,562::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`64030e3a-0e62-40b5-9da7-ac2cdd3af540`::Granted request >Thread-11412::DEBUG::2012-12-02 23:56:07,562::task::794::TaskManager.Task::(resourceAcquired) Task=`31c8af6b-4714-45e1-9f90-9cb6eb9a1ca6`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-11412::DEBUG::2012-12-02 23:56:07,563::task::957::TaskManager.Task::(_decref) Task=`31c8af6b-4714-45e1-9f90-9cb6eb9a1ca6`::ref 1 aborting False >Thread-11412::DEBUG::2012-12-02 23:56:07,568::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11412::DEBUG::2012-12-02 23:56:07,568::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-11412::DEBUG::2012-12-02 23:56:07,773::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-11412::WARNING::2012-12-02 23:56:07,775::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-11412::DEBUG::2012-12-02 23:56:07,775::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11412::WARNING::2012-12-02 23:56:07,804::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-11412::DEBUG::2012-12-02 23:56:07,805::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-11412::DEBUG::2012-12-02 23:56:07,807::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-11412::DEBUG::2012-12-02 23:56:08,015::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-11412::WARNING::2012-12-02 23:56:08,016::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-11412::DEBUG::2012-12-02 23:56:08,016::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11412::WARNING::2012-12-02 23:56:08,047::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-11412::INFO::2012-12-02 23:56:08,048::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-2443::ERROR::2012-12-02 23:56:08,047::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `5971512c-ee51-475e-b956-3872c38ce193` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/nfsSD.py", line 103, in selftest > fileSD.FileStorageDomain.selftest(self) > File "/usr/share/vdsm/storage/fileSD.py", line 398, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 270, in callCrabRPCFunction > *args, **kwargs) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 188, in callCrabRPCFunction > raise err >OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328/5971512c-ee51-475e-b956-3872c38ce193' >Thread-11412::DEBUG::2012-12-02 23:56:08,048::task::1151::TaskManager.Task::(prepare) Task=`31c8af6b-4714-45e1-9f90-9cb6eb9a1ca6`::finished: {'info': {'spm_id': 2, 'master_uuid': '5971512c-ee51-475e-b956-3872c38ce193', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active,5971512c-ee51-475e-b956-3872c38ce193:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 3, 'lver': 0}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active', 'alerts': []}}} >Thread-11412::DEBUG::2012-12-02 23:56:08,048::task::568::TaskManager.Task::(_updateState) Task=`31c8af6b-4714-45e1-9f90-9cb6eb9a1ca6`::moving from state preparing -> state finished >Thread-11412::DEBUG::2012-12-02 23:56:08,048::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-11412::DEBUG::2012-12-02 23:56:08,048::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11412::DEBUG::2012-12-02 23:56:08,049::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-11412::DEBUG::2012-12-02 23:56:08,049::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-11412::DEBUG::2012-12-02 23:56:08,049::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-11412::DEBUG::2012-12-02 23:56:08,049::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-11412::DEBUG::2012-12-02 23:56:08,049::task::957::TaskManager.Task::(_decref) Task=`31c8af6b-4714-45e1-9f90-9cb6eb9a1ca6`::ref 0 aborting False >Thread-11535::DEBUG::2012-12-02 23:56:09,219::task::568::TaskManager.Task::(_updateState) Task=`a72814a2-9fd8-484a-aa92-979ab75b5ace`::moving from state init -> state preparing >Thread-11535::INFO::2012-12-02 23:56:09,220::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-11535::INFO::2012-12-02 23:56:09,220::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485368.048439, 'code': 200, 'valid': False}} >Thread-11535::DEBUG::2012-12-02 23:56:09,220::task::1151::TaskManager.Task::(prepare) Task=`a72814a2-9fd8-484a-aa92-979ab75b5ace`::finished: {'5971512c-ee51-475e-b956-3872c38ce193': {'delay': '0', 'lastCheck': 1354485368.048439, 'code': 200, 'valid': False}} >Thread-11535::DEBUG::2012-12-02 23:56:09,220::task::568::TaskManager.Task::(_updateState) Task=`a72814a2-9fd8-484a-aa92-979ab75b5ace`::moving from state preparing -> state finished >Thread-11535::DEBUG::2012-12-02 23:56:09,220::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-11535::DEBUG::2012-12-02 23:56:09,220::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-11535::DEBUG::2012-12-02 23:56:09,220::task::957::TaskManager.Task::(_decref) Task=`a72814a2-9fd8-484a-aa92-979ab75b5ace`::ref 0 aborting False >MainThread::INFO::2012-12-02 23:56:15,718::vmChannels::135::vds::(stop) VM channels listener was stopped. >MainThread::DEBUG::2012-12-02 23:56:15,718::task::568::TaskManager.Task::(_updateState) Task=`203a72f7-5ddd-486e-aadc-733cf58c6f41`::moving from state init -> state preparing >MainThread::INFO::2012-12-02 23:56:15,718::logUtils::37::dispatcher::(wrapper) Run and protect: prepareForShutdown(options=None) >Thread-11::DEBUG::2012-12-02 23:56:15,719::storageServer::617::ConnectionMonitor::(_monitorConnections) Monitoring stopped >MainThread::DEBUG::2012-12-02 23:56:15,725::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >Thread-2443::DEBUG::2012-12-02 23:56:15,776::domainMonitor::182::Storage.DomainMonitor::(_monitorDomain) Monitorg for domain 5971512c-ee51-475e-b956-3872c38ce193 is stopping >Thread-2443::INFO::2012-12-02 23:56:15,776::safelease::183::SANLock::(releaseHostId) Releasing host id for domain 5971512c-ee51-475e-b956-3872c38ce193 (id: 2) >Thread-2443::DEBUG::2012-12-02 23:56:15,777::domainMonitor::191::Storage.DomainMonitor::(_monitorDomain) Unable to release the host id 2 for the domain 5971512c-ee51-475e-b956-3872c38ce193 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 188, in _monitorDomain > domain.releaseHostId(hostId, unused=True) > File "/usr/share/vdsm/storage/sd.py", line 426, in releaseHostId > self._clusterLock.releaseHostId(hostId, async, unused) > File "/usr/share/vdsm/storage/safelease.py", line 190, in releaseHostId > raise se.ReleaseHostIdFailure(self._sdUUID, e) >ReleaseHostIdFailure: Cannot release host id: ('5971512c-ee51-475e-b956-3872c38ce193', SanlockException(16, 'Sanlock lockspace remove failure', 'Device or resource busy')) >MainThread::DEBUG::2012-12-02 23:56:15,777::taskManager::80::TaskManager::(prepareForShutdown) Request to stop all tasks >MainThread::INFO::2012-12-02 23:56:15,777::logUtils::39::dispatcher::(wrapper) Run and protect: prepareForShutdown, Return response: None >MainThread::DEBUG::2012-12-02 23:56:15,777::task::1151::TaskManager.Task::(prepare) Task=`203a72f7-5ddd-486e-aadc-733cf58c6f41`::finished: None >MainThread::DEBUG::2012-12-02 23:56:15,777::task::568::TaskManager.Task::(_updateState) Task=`203a72f7-5ddd-486e-aadc-733cf58c6f41`::moving from state preparing -> state finished >MainThread::DEBUG::2012-12-02 23:56:15,778::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >MainThread::DEBUG::2012-12-02 23:56:15,778::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >MainThread::DEBUG::2012-12-02 23:56:15,778::task::957::TaskManager.Task::(_decref) Task=`203a72f7-5ddd-486e-aadc-733cf58c6f41`::ref 0 aborting False >MainThread::INFO::2012-12-02 23:56:15,778::vdsm::76::vds::(run) VDSM main thread ended. Waiting for 13 other threads... >MainThread::INFO::2012-12-02 23:56:15,778::vdsm::79::vds::(run) <_MainThread(MainThread, started 140658209314560)> >MainThread::INFO::2012-12-02 23:56:15,778::vdsm::79::vds::(run) <Thread(libvirtEventLoop, started daemon 140658130458368)> >MainThread::INFO::2012-12-02 23:56:15,778::vdsm::79::vds::(run) <WorkerThread(Thread-5, started daemon 140657978857216)> >MainThread::INFO::2012-12-02 23:56:15,778::vdsm::79::vds::(run) <WorkerThread(Thread-4, started daemon 140657989347072)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <WorkerThread(Thread-3, started daemon 140657999836928)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <WorkerThread(Thread-2, started daemon 140658010326784)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <WorkerThread(fec5421a-317e-429c-a472-8bebef790541, started daemon 140658020816640)> >MainThread::INFO::2012-12-02 23:56:15,779::vmChannels::135::vds::(stop) VM channels listener was stopped. >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <Listener(VM Channels Listener, started daemon 140657184192256)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <WorkerThread(Thread-9, started daemon 140657597335296)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <KsmMonitorThread(KsmMonitor, started daemon 140657205171968)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <WorkerThread(Thread-6, started daemon 140657628804864)> >MainThread::INFO::2012-12-02 23:56:15,779::vdsm::79::vds::(run) <WorkerThread(Thread-8, started daemon 140657607825152)> >MainThread::INFO::2012-12-02 23:56:15,780::vdsm::79::vds::(run) <WorkerThread(Thread-7, started daemon 140657618315008)> >MainThread::INFO::2012-12-02 23:56:15,780::vdsm::79::vds::(run) <WorkerThread(Thread-10, started daemon 140657586845440)> >MainThread::INFO::2012-12-02 23:56:15,847::vdsm::71::vds::(run) I am the actual vdsm 4.10-0.239 >MainThread::DEBUG::2012-12-02 23:56:15,978::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'Storage' >MainThread::DEBUG::2012-12-02 23:56:15,979::threadPool::45::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 >MainThread::DEBUG::2012-12-02 23:56:15,986::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >MainThread::DEBUG::2012-12-02 23:56:16,009::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/kill -9 1911' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:16,016::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:56:16,017::supervdsm::91::SuperVdsmProxy::(_launchSupervdsm) Launching Super Vdsm >MainThread::DEBUG::2012-12-02 23:56:16,017::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /usr/bin/python /usr/share/vdsm/supervdsmServer.py 78f999a9-cc9b-429d-b30c-6d0b2e638967 14902' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:16,137::supervdsmServer::306::SuperVdsm.Server::(main) Making sure I'm root >MainThread::DEBUG::2012-12-02 23:56:16,137::supervdsmServer::310::SuperVdsm.Server::(main) Parsing cmd args >MainThread::DEBUG::2012-12-02 23:56:16,137::supervdsmServer::313::SuperVdsm.Server::(main) Creating PID file >MainThread::DEBUG::2012-12-02 23:56:16,137::supervdsmServer::317::SuperVdsm.Server::(main) Cleaning old socket >MainThread::DEBUG::2012-12-02 23:56:16,137::supervdsmServer::321::SuperVdsm.Server::(main) Setting up keep alive thread >MainThread::DEBUG::2012-12-02 23:56:16,137::supervdsmServer::327::SuperVdsm.Server::(main) Creating remote object manager >MainThread::DEBUG::2012-12-02 23:56:16,138::supervdsmServer::338::SuperVdsm.Server::(main) Started serving super vdsm object >MainThread::DEBUG::2012-12-02 23:56:18,022::supervdsm::113::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm >MainThread::DEBUG::2012-12-02 23:56:18,027::supervdsm::84::SuperVdsmProxy::(__init__) Connected to Super Vdsm >MainThread::DEBUG::2012-12-02 23:56:18,027::multipath::106::Storage.Multipath::(isEnabled) Current revision of multipath.conf detected, preserving >MainThread::DEBUG::2012-12-02 23:56:18,028::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm dumpconfig global/locking_type' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:18,099::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:56:18,100::lvm::334::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2012-12-02 23:56:18,102::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:18,416::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:56:18,417::lvm::359::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex >MainThread::DEBUG::2012-12-02 23:56:18,417::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >MainThread::DEBUG::2012-12-02 23:56:18,417::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:18,804::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >MainThread::DEBUG::2012-12-02 23:56:18,807::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >MainThread::DEBUG::2012-12-02 23:56:18,807::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:19,148::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = " Couldn't find device with uuid 0s668w-dBq1-VPa0-V67O-keb6-v7bc-cWXe6u.\n"; <rc> = 0 >Thread-12::DEBUG::2012-12-02 23:56:19,164::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >MainThread::INFO::2012-12-02 23:56:19,165::dispatcher::95::Storage.Dispatcher::(__init__) Starting StorageDispatcher... >Thread-12::DEBUG::2012-12-02 23:56:19,165::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-12::DEBUG::2012-12-02 23:56:19,165::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-12::DEBUG::2012-12-02 23:56:19,166::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-12::DEBUG::2012-12-02 23:56:19,166::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:19,173::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/pgrep -xf ksmd' (cwd None) >MainThread::DEBUG::2012-12-02 23:56:19,186::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >MainThread::INFO::2012-12-02 23:56:19,187::vmChannels::139::vds::(settimeout) Setting channels' timeout to 30 seconds. >VM Channels Listener::INFO::2012-12-02 23:56:19,190::vmChannels::127::vds::(run) Starting VM channels listener thread. >Thread-12::DEBUG::2012-12-02 23:56:19,191::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-12::DEBUG::2012-12-02 23:56:19,191::misc::1071::SamplingMethod::(__call__) Returning last result >MainThread::WARNING::2012-12-02 23:56:19,194::clientIF::169::vds::(_prepareBindings) Unable to load the rest server module. Please make sure it is installed. >Thread-12::DEBUG::2012-12-02 23:56:19,824::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-12::DEBUG::2012-12-02 23:56:19,922::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-12::DEBUG::2012-12-02 23:56:19,923::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 23:56:19,923::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 23:56:19,924::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 23:56:19,924::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 23:56:19,924::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-12::DEBUG::2012-12-02 23:56:19,924::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-12::DEBUG::2012-12-02 23:56:19,924::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-12::DEBUG::2012-12-02 23:56:19,924::hsm::355::Storage.HSM::(__cleanStorageRepository) Started cleaning storage repository at '/rhev/data-center' >Thread-12::DEBUG::2012-12-02 23:56:19,932::hsm::387::Storage.HSM::(__cleanStorageRepository) White list: ['/rhev/data-center/hsm-tasks', '/rhev/data-center/hsm-tasks/*', '/rhev/data-center/mnt'] >Thread-12::DEBUG::2012-12-02 23:56:19,932::hsm::388::Storage.HSM::(__cleanStorageRepository) Mount list: ['/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_101', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0941', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0851', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sdds1', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sdds2', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_export1', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_data3', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1446', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510', '/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328'] >Thread-12::DEBUG::2012-12-02 23:56:19,932::hsm::390::Storage.HSM::(__cleanStorageRepository) Cleaning leftovers >Thread-12::DEBUG::2012-12-02 23:56:19,934::hsm::433::Storage.HSM::(__cleanStorageRepository) Finished cleaning storage repository at '/rhev/data-center' >Thread-14::DEBUG::2012-12-02 23:56:20,518::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-14::DEBUG::2012-12-02 23:56:20,615::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-14::DEBUG::2012-12-02 23:56:20,627::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-14::DEBUG::2012-12-02 23:56:20,627::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-14::DEBUG::2012-12-02 23:56:20,636::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-14::DEBUG::2012-12-02 23:56:20,637::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-14::DEBUG::2012-12-02 23:56:20,647::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 23:56:20,647::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-14::DEBUG::2012-12-02 23:56:20,659::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 23:56:20,659::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-14::DEBUG::2012-12-02 23:56:20,671::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-14::DEBUG::2012-12-02 23:56:20,887::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '0.0.0.0'}} >Thread-16::DEBUG::2012-12-02 23:56:21,470::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-16::DEBUG::2012-12-02 23:56:21,470::task::568::TaskManager.Task::(_updateState) Task=`56a032c7-6cc0-4f51-9960-0442cfcc6694`::moving from state init -> state preparing >Thread-16::INFO::2012-12-02 23:56:21,471::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-16::INFO::2012-12-02 23:56:21,471::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-16::DEBUG::2012-12-02 23:56:21,471::task::1151::TaskManager.Task::(prepare) Task=`56a032c7-6cc0-4f51-9960-0442cfcc6694`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-16::DEBUG::2012-12-02 23:56:21,471::task::568::TaskManager.Task::(_updateState) Task=`56a032c7-6cc0-4f51-9960-0442cfcc6694`::moving from state preparing -> state finished >Thread-16::DEBUG::2012-12-02 23:56:21,471::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-16::DEBUG::2012-12-02 23:56:21,471::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-16::DEBUG::2012-12-02 23:56:21,472::task::957::TaskManager.Task::(_decref) Task=`56a032c7-6cc0-4f51-9960-0442cfcc6694`::ref 0 aborting False >Thread-17::DEBUG::2012-12-02 23:56:21,611::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-17::DEBUG::2012-12-02 23:56:21,611::task::568::TaskManager.Task::(_updateState) Task=`5a7d9d8f-8764-42e3-bade-384806313424`::moving from state init -> state preparing >Thread-17::INFO::2012-12-02 23:56:21,611::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2328', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3', 'port': ''}], options=None) >Thread-17::DEBUG::2012-12-02 23:56:21,617::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:56:21,617::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:56:21,617::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:56:21,617::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::DEBUG::2012-12-02 23:56:21,617::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-17::DEBUG::2012-12-02 23:56:21,617::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-17::INFO::2012-12-02 23:56:21,617::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-17::DEBUG::2012-12-02 23:56:21,617::task::1151::TaskManager.Task::(prepare) Task=`5a7d9d8f-8764-42e3-bade-384806313424`::finished: {'statuslist': [{'status': 0, 'id': '97f85d2d-301f-4265-97ba-b6c5c7dc65e3'}]} >Thread-17::DEBUG::2012-12-02 23:56:21,618::task::568::TaskManager.Task::(_updateState) Task=`5a7d9d8f-8764-42e3-bade-384806313424`::moving from state preparing -> state finished >Thread-17::DEBUG::2012-12-02 23:56:21,618::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-17::DEBUG::2012-12-02 23:56:21,618::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-17::DEBUG::2012-12-02 23:56:21,618::task::957::TaskManager.Task::(_decref) Task=`5a7d9d8f-8764-42e3-bade-384806313424`::ref 0 aborting False >Thread-18::DEBUG::2012-12-02 23:56:21,816::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-18::DEBUG::2012-12-02 23:56:21,816::task::568::TaskManager.Task::(_updateState) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::moving from state init -> state preparing >Thread-18::INFO::2012-12-02 23:56:21,816::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='5971512c-ee51-475e-b956-3872c38ce193', masterVersion=3, options=None) >Thread-18::DEBUG::2012-12-02 23:56:21,817::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7c1853a8-a715-4099-a011-fd2d570ec124`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-18::DEBUG::2012-12-02 23:56:21,817::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-18::DEBUG::2012-12-02 23:56:21,817::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-18::DEBUG::2012-12-02 23:56:21,817::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`7c1853a8-a715-4099-a011-fd2d570ec124`::Granted request >Thread-18::INFO::2012-12-02 23:56:21,818::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: 5971512c-ee51-475e-b956-3872c38ce193 (ver = 3) >Thread-18::DEBUG::2012-12-02 23:56:21,818::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:21,818::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:21,818::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:21,819::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:21,819::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:21,819::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:21,819::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-18::DEBUG::2012-12-02 23:56:21,819::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-18::DEBUG::2012-12-02 23:56:21,819::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-18::DEBUG::2012-12-02 23:56:21,819::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-18::DEBUG::2012-12-02 23:56:21,820::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-18::DEBUG::2012-12-02 23:56:21,846::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-18::DEBUG::2012-12-02 23:56:21,846::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-18::DEBUG::2012-12-02 23:56:22,480::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-18::DEBUG::2012-12-02 23:56:22,574::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-18::DEBUG::2012-12-02 23:56:22,574::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,575::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,575::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,575::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,575::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,575::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,575::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-18::DEBUG::2012-12-02 23:56:22,576::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,577::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-18::DEBUG::2012-12-02 23:56:22,784::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-18::WARNING::2012-12-02 23:56:22,785::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-18::DEBUG::2012-12-02 23:56:22,785::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18::DEBUG::2012-12-02 23:56:22,909::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-18::DEBUG::2012-12-02 23:56:22,909::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-18::DEBUG::2012-12-02 23:56:22,909::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-18::DEBUG::2012-12-02 23:56:22,909::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-18::ERROR::2012-12-02 23:56:22,910::task::833::TaskManager.Task::(_setError) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=5971512c-ee51-475e-b956-3872c38ce193' >Thread-18::DEBUG::2012-12-02 23:56:22,912::task::852::TaskManager.Task::(_run) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::Task._run: 96c72fab-8345-4008-b858-b900dc7b8bef ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', '5971512c-ee51-475e-b956-3872c38ce193', 3) {} failed - stopping task >Thread-18::DEBUG::2012-12-02 23:56:22,912::task::1177::TaskManager.Task::(stop) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::stopping in state preparing (force False) >Thread-18::DEBUG::2012-12-02 23:56:22,912::task::957::TaskManager.Task::(_decref) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::ref 1 aborting True >Thread-18::INFO::2012-12-02 23:56:22,912::task::1134::TaskManager.Task::(prepare) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-18::DEBUG::2012-12-02 23:56:22,912::task::1139::TaskManager.Task::(prepare) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::Prepare: aborted: Cannot find master domain >Thread-18::DEBUG::2012-12-02 23:56:22,912::task::957::TaskManager.Task::(_decref) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::ref 0 aborting True >Thread-18::DEBUG::2012-12-02 23:56:22,912::task::892::TaskManager.Task::(_doAbort) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::Task._doAbort: force False >Thread-18::DEBUG::2012-12-02 23:56:22,913::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-18::DEBUG::2012-12-02 23:56:22,913::task::568::TaskManager.Task::(_updateState) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::moving from state preparing -> state aborting >Thread-18::DEBUG::2012-12-02 23:56:22,913::task::523::TaskManager.Task::(__state_aborting) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::_aborting: recover policy none >Thread-18::DEBUG::2012-12-02 23:56:22,913::task::568::TaskManager.Task::(_updateState) Task=`96c72fab-8345-4008-b858-b900dc7b8bef`::moving from state aborting -> state failed >Thread-18::DEBUG::2012-12-02 23:56:22,913::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-18::DEBUG::2012-12-02 23:56:22,913::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-18::ERROR::2012-12-02 23:56:22,913::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=5971512c-ee51-475e-b956-3872c38ce193'", 'code': 304}} >Thread-80::DEBUG::2012-12-02 23:56:23,187::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-80::DEBUG::2012-12-02 23:56:23,187::task::568::TaskManager.Task::(_updateState) Task=`9cad01ae-b233-4b20-9c4c-d084be5ec197`::moving from state init -> state preparing >Thread-80::INFO::2012-12-02 23:56:23,188::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-80::DEBUG::2012-12-02 23:56:23,188::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-80::DEBUG::2012-12-02 23:56:23,188::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-80::INFO::2012-12-02 23:56:23,188::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-80::DEBUG::2012-12-02 23:56:23,188::task::1151::TaskManager.Task::(prepare) Task=`9cad01ae-b233-4b20-9c4c-d084be5ec197`::finished: {'allTasksStatus': {}} >Thread-80::DEBUG::2012-12-02 23:56:23,188::task::568::TaskManager.Task::(_updateState) Task=`9cad01ae-b233-4b20-9c4c-d084be5ec197`::moving from state preparing -> state finished >Thread-80::DEBUG::2012-12-02 23:56:23,188::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-80::DEBUG::2012-12-02 23:56:23,188::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-80::DEBUG::2012-12-02 23:56:23,189::task::957::TaskManager.Task::(_decref) Task=`9cad01ae-b233-4b20-9c4c-d084be5ec197`::ref 0 aborting False >Thread-81::DEBUG::2012-12-02 23:56:23,335::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-81::DEBUG::2012-12-02 23:56:23,335::task::568::TaskManager.Task::(_updateState) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::moving from state init -> state preparing >Thread-81::INFO::2012-12-02 23:56:23,336::logUtils::37::dispatcher::(wrapper) Run and protect: spmStop(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-81::DEBUG::2012-12-02 23:56:23,336::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`60acf696-5ec7-473e-8da3-8c6a259af6ef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-81::DEBUG::2012-12-02 23:56:23,336::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-81::DEBUG::2012-12-02 23:56:23,336::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-81::DEBUG::2012-12-02 23:56:23,336::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`60acf696-5ec7-473e-8da3-8c6a259af6ef`::Granted request >Thread-81::DEBUG::2012-12-02 23:56:23,337::task::794::TaskManager.Task::(resourceAcquired) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-81::DEBUG::2012-12-02 23:56:23,337::task::957::TaskManager.Task::(_decref) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::ref 1 aborting False >Thread-81::ERROR::2012-12-02 23:56:23,337::task::833::TaskManager.Task::(_setError) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 516, in spmStop > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-81::DEBUG::2012-12-02 23:56:23,337::task::852::TaskManager.Task::(_run) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::Task._run: 1f3df3c1-96ed-41cc-b44e-64e56c351207 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-81::DEBUG::2012-12-02 23:56:23,337::task::1177::TaskManager.Task::(stop) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::stopping in state preparing (force False) >Thread-81::DEBUG::2012-12-02 23:56:23,337::task::957::TaskManager.Task::(_decref) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::ref 1 aborting True >Thread-81::INFO::2012-12-02 23:56:23,338::task::1134::TaskManager.Task::(prepare) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-81::DEBUG::2012-12-02 23:56:23,338::task::1139::TaskManager.Task::(prepare) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::Prepare: aborted: Unknown pool id, pool not connected >Thread-81::DEBUG::2012-12-02 23:56:23,338::task::957::TaskManager.Task::(_decref) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::ref 0 aborting True >Thread-81::DEBUG::2012-12-02 23:56:23,338::task::892::TaskManager.Task::(_doAbort) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::Task._doAbort: force False >Thread-81::DEBUG::2012-12-02 23:56:23,338::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-81::DEBUG::2012-12-02 23:56:23,338::task::568::TaskManager.Task::(_updateState) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::moving from state preparing -> state aborting >Thread-81::DEBUG::2012-12-02 23:56:23,338::task::523::TaskManager.Task::(__state_aborting) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::_aborting: recover policy none >Thread-81::DEBUG::2012-12-02 23:56:23,338::task::568::TaskManager.Task::(_updateState) Task=`1f3df3c1-96ed-41cc-b44e-64e56c351207`::moving from state aborting -> state failed >Thread-81::DEBUG::2012-12-02 23:56:23,339::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-81::DEBUG::2012-12-02 23:56:23,339::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-81::DEBUG::2012-12-02 23:56:23,339::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-81::DEBUG::2012-12-02 23:56:23,339::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-81::DEBUG::2012-12-02 23:56:23,339::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-81::DEBUG::2012-12-02 23:56:23,339::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-81::ERROR::2012-12-02 23:56:23,339::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-82::DEBUG::2012-12-02 23:56:29,631::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-82::DEBUG::2012-12-02 23:56:29,632::task::568::TaskManager.Task::(_updateState) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::moving from state init -> state preparing >Thread-82::INFO::2012-12-02 23:56:29,632::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-82::ERROR::2012-12-02 23:56:29,632::task::833::TaskManager.Task::(_setError) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 521, in getSpmStatus > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-82::DEBUG::2012-12-02 23:56:29,633::task::852::TaskManager.Task::(_run) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::Task._run: d4678837-81ac-40a8-85b8-29be177e1e99 ('479cf195-bb99-45c5-b400-433d810eaa48',) {} failed - stopping task >Thread-82::DEBUG::2012-12-02 23:56:29,633::task::1177::TaskManager.Task::(stop) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::stopping in state preparing (force False) >Thread-82::DEBUG::2012-12-02 23:56:29,633::task::957::TaskManager.Task::(_decref) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::ref 1 aborting True >Thread-82::INFO::2012-12-02 23:56:29,633::task::1134::TaskManager.Task::(prepare) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-82::DEBUG::2012-12-02 23:56:29,633::task::1139::TaskManager.Task::(prepare) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::Prepare: aborted: Unknown pool id, pool not connected >Thread-82::DEBUG::2012-12-02 23:56:29,633::task::957::TaskManager.Task::(_decref) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::ref 0 aborting True >Thread-82::DEBUG::2012-12-02 23:56:29,633::task::892::TaskManager.Task::(_doAbort) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::Task._doAbort: force False >Thread-82::DEBUG::2012-12-02 23:56:29,633::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-82::DEBUG::2012-12-02 23:56:29,634::task::568::TaskManager.Task::(_updateState) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::moving from state preparing -> state aborting >Thread-82::DEBUG::2012-12-02 23:56:29,634::task::523::TaskManager.Task::(__state_aborting) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::_aborting: recover policy none >Thread-82::DEBUG::2012-12-02 23:56:29,634::task::568::TaskManager.Task::(_updateState) Task=`d4678837-81ac-40a8-85b8-29be177e1e99`::moving from state aborting -> state failed >Thread-82::DEBUG::2012-12-02 23:56:29,634::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-82::DEBUG::2012-12-02 23:56:29,634::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-82::ERROR::2012-12-02 23:56:29,634::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-83::DEBUG::2012-12-03 00:03:39,023::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-83::DEBUG::2012-12-03 00:03:39,027::task::568::TaskManager.Task::(_updateState) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::moving from state init -> state preparing >Thread-83::INFO::2012-12-03 00:03:39,027::logUtils::37::dispatcher::(wrapper) Run and protect: refreshStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='a00efcc1-5685-4dbe-ae70-440e9f403b76', masterVersion=4, options=None) >Thread-83::DEBUG::2012-12-03 00:03:39,027::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b541fc2b-8e0c-4b4c-a228-3038de4a07fa`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-83::DEBUG::2012-12-03 00:03:39,027::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-83::DEBUG::2012-12-03 00:03:39,027::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-83::DEBUG::2012-12-03 00:03:39,027::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b541fc2b-8e0c-4b4c-a228-3038de4a07fa`::Granted request >Thread-83::DEBUG::2012-12-03 00:03:39,027::task::794::TaskManager.Task::(resourceAcquired) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-83::DEBUG::2012-12-03 00:03:39,027::task::957::TaskManager.Task::(_decref) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::ref 1 aborting False >Thread-83::ERROR::2012-12-03 00:03:39,028::task::833::TaskManager.Task::(_setError) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 693, in refreshStoragePool > pool = self.getPool(spUUID) > File "/usr/share/vdsm/storage/hsm.py", line 258, in getPool > raise se.StoragePoolUnknown(spUUID) >StoragePoolUnknown: Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',) >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::852::TaskManager.Task::(_run) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::Task._run: 681b0387-59b9-4884-ada9-ce3c87463e2e ('479cf195-bb99-45c5-b400-433d810eaa48', 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 4) {} failed - stopping task >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::1177::TaskManager.Task::(stop) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::stopping in state preparing (force False) >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::957::TaskManager.Task::(_decref) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::ref 1 aborting True >Thread-83::INFO::2012-12-03 00:03:39,030::task::1134::TaskManager.Task::(prepare) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::1139::TaskManager.Task::(prepare) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::Prepare: aborted: Unknown pool id, pool not connected >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::957::TaskManager.Task::(_decref) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::ref 0 aborting True >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::892::TaskManager.Task::(_doAbort) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::Task._doAbort: force False >Thread-83::DEBUG::2012-12-03 00:03:39,030::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::568::TaskManager.Task::(_updateState) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::moving from state preparing -> state aborting >Thread-83::DEBUG::2012-12-03 00:03:39,030::task::523::TaskManager.Task::(__state_aborting) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::_aborting: recover policy none >Thread-83::DEBUG::2012-12-03 00:03:39,031::task::568::TaskManager.Task::(_updateState) Task=`681b0387-59b9-4884-ada9-ce3c87463e2e`::moving from state aborting -> state failed >Thread-83::DEBUG::2012-12-03 00:03:39,031::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-83::DEBUG::2012-12-03 00:03:39,031::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-83::DEBUG::2012-12-03 00:03:39,031::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-83::DEBUG::2012-12-03 00:03:39,031::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-83::DEBUG::2012-12-03 00:03:39,031::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-83::DEBUG::2012-12-03 00:03:39,031::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-83::ERROR::2012-12-03 00:03:39,031::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('479cf195-bb99-45c5-b400-433d810eaa48',)", 'code': 309}} >Thread-84::DEBUG::2012-12-03 00:03:55,605::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-84::DEBUG::2012-12-03 00:03:55,605::task::568::TaskManager.Task::(_updateState) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::moving from state init -> state preparing >Thread-84::INFO::2012-12-03 00:03:55,605::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='a00efcc1-5685-4dbe-ae70-440e9f403b76', masterVersion=4, options=None) >Thread-84::DEBUG::2012-12-03 00:03:55,606::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ed73cc0e-771a-40ff-a47a-fc8aca56adc0`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-84::DEBUG::2012-12-03 00:03:55,606::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-84::DEBUG::2012-12-03 00:03:55,606::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-84::DEBUG::2012-12-03 00:03:55,606::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`ed73cc0e-771a-40ff-a47a-fc8aca56adc0`::Granted request >Thread-84::INFO::2012-12-03 00:03:55,606::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: a00efcc1-5685-4dbe-ae70-440e9f403b76 (ver = 4) >Thread-84::DEBUG::2012-12-03 00:03:55,607::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:55,607::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:55,607::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:55,607::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:55,607::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:55,608::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:55,608::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-84::DEBUG::2012-12-03 00:03:55,608::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-84::DEBUG::2012-12-03 00:03:55,608::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-84::DEBUG::2012-12-03 00:03:55,608::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-84::DEBUG::2012-12-03 00:03:55,608::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-84::DEBUG::2012-12-03 00:03:55,642::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-84::DEBUG::2012-12-03 00:03:55,643::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-84::DEBUG::2012-12-03 00:03:56,276::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-84::DEBUG::2012-12-03 00:03:56,415::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-84::DEBUG::2012-12-03 00:03:56,415::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,416::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,416::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,416::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,416::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,416::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,417::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-84::DEBUG::2012-12-03 00:03:56,417::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,419::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-84::DEBUG::2012-12-03 00:03:56,639::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-84::WARNING::2012-12-03 00:03:56,641::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-84::DEBUG::2012-12-03 00:03:56,641::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-84::DEBUG::2012-12-03 00:03:56,673::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-84::DEBUG::2012-12-03 00:03:56,673::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-84::DEBUG::2012-12-03 00:03:56,673::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-84::DEBUG::2012-12-03 00:03:56,673::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-84::ERROR::2012-12-03 00:03:56,674::task::833::TaskManager.Task::(_setError) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 840, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 38, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 812, in connectStoragePool > return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) > File "/usr/share/vdsm/storage/hsm.py", line 854, in _connectStoragePool > res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 638, in connect > self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1163, in __rebuild > self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) > File "/usr/share/vdsm/storage/sp.py", line 1502, in getMasterDomain > raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) >StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76' >Thread-84::DEBUG::2012-12-03 00:03:56,674::task::852::TaskManager.Task::(_run) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::Task._run: 57af6725-51bf-4914-aecf-af46e2b9ceca ('479cf195-bb99-45c5-b400-433d810eaa48', 2, '479cf195-bb99-45c5-b400-433d810eaa48', 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 4) {} failed - stopping task >Thread-84::DEBUG::2012-12-03 00:03:56,674::task::1177::TaskManager.Task::(stop) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::stopping in state preparing (force False) >Thread-84::DEBUG::2012-12-03 00:03:56,674::task::957::TaskManager.Task::(_decref) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::ref 1 aborting True >Thread-84::INFO::2012-12-03 00:03:56,675::task::1134::TaskManager.Task::(prepare) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::aborting: Task is aborted: 'Cannot find master domain' - code 304 >Thread-84::DEBUG::2012-12-03 00:03:56,675::task::1139::TaskManager.Task::(prepare) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::Prepare: aborted: Cannot find master domain >Thread-84::DEBUG::2012-12-03 00:03:56,675::task::957::TaskManager.Task::(_decref) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::ref 0 aborting True >Thread-84::DEBUG::2012-12-03 00:03:56,675::task::892::TaskManager.Task::(_doAbort) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::Task._doAbort: force False >Thread-84::DEBUG::2012-12-03 00:03:56,675::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-84::DEBUG::2012-12-03 00:03:56,675::task::568::TaskManager.Task::(_updateState) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::moving from state preparing -> state aborting >Thread-84::DEBUG::2012-12-03 00:03:56,675::task::523::TaskManager.Task::(__state_aborting) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::_aborting: recover policy none >Thread-84::DEBUG::2012-12-03 00:03:56,675::task::568::TaskManager.Task::(_updateState) Task=`57af6725-51bf-4914-aecf-af46e2b9ceca`::moving from state aborting -> state failed >Thread-84::DEBUG::2012-12-03 00:03:56,676::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-84::DEBUG::2012-12-03 00:03:56,676::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-84::ERROR::2012-12-03 00:03:56,676::dispatcher::66::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=479cf195-bb99-45c5-b400-433d810eaa48, msdUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76'", 'code': 304}} >Thread-146::DEBUG::2012-12-03 00:09:46,220::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} flowID [22cf7d22] >Thread-146::DEBUG::2012-12-03 00:09:46,234::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-146::DEBUG::2012-12-03 00:09:46,246::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-146::DEBUG::2012-12-03 00:09:46,246::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-146::DEBUG::2012-12-03 00:09:46,255::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-146::DEBUG::2012-12-03 00:09:46,256::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-146::DEBUG::2012-12-03 00:09:46,267::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-146::DEBUG::2012-12-03 00:09:46,267::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-146::DEBUG::2012-12-03 00:09:46,279::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-146::DEBUG::2012-12-03 00:09:46,280::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-146::DEBUG::2012-12-03 00:09:46,292::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-146::DEBUG::2012-12-03 00:09:46,293::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.202.48'}} >Thread-147::DEBUG::2012-12-03 00:09:46,800::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.48]::call getCapabilities with () {} >Thread-147::DEBUG::2012-12-03 00:09:46,809::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-147::DEBUG::2012-12-03 00:09:46,819::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-147::DEBUG::2012-12-03 00:09:46,820::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-147::DEBUG::2012-12-03 00:09:46,829::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-147::DEBUG::2012-12-03 00:09:46,829::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-147::DEBUG::2012-12-03 00:09:46,841::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-147::DEBUG::2012-12-03 00:09:46,841::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-147::DEBUG::2012-12-03 00:09:46,853::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-147::DEBUG::2012-12-03 00:09:46,854::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-147::DEBUG::2012-12-03 00:09:46,863::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-147::DEBUG::2012-12-03 00:09:46,864::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.202.48'}} >Thread-149::DEBUG::2012-12-03 00:09:47,421::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-149::DEBUG::2012-12-03 00:09:47,421::task::568::TaskManager.Task::(_updateState) Task=`2ecfa03e-a9c8-407d-a176-4c54bacfddb0`::moving from state init -> state preparing >Thread-149::INFO::2012-12-03 00:09:47,421::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-149::INFO::2012-12-03 00:09:47,422::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-149::DEBUG::2012-12-03 00:09:47,422::task::1151::TaskManager.Task::(prepare) Task=`2ecfa03e-a9c8-407d-a176-4c54bacfddb0`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-149::DEBUG::2012-12-03 00:09:47,422::task::568::TaskManager.Task::(_updateState) Task=`2ecfa03e-a9c8-407d-a176-4c54bacfddb0`::moving from state preparing -> state finished >Thread-149::DEBUG::2012-12-03 00:09:47,422::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-149::DEBUG::2012-12-03 00:09:47,422::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-149::DEBUG::2012-12-03 00:09:47,422::task::957::TaskManager.Task::(_decref) Task=`2ecfa03e-a9c8-407d-a176-4c54bacfddb0`::ref 0 aborting False >Thread-150::DEBUG::2012-12-03 00:09:47,602::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-150::DEBUG::2012-12-03 00:09:47,603::task::568::TaskManager.Task::(_updateState) Task=`76cb1efd-a620-4710-987e-d6139b4aafdd`::moving from state init -> state preparing >Thread-150::INFO::2012-12-03 00:09:47,603::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-150::DEBUG::2012-12-03 00:09:47,608::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348 /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348' (cwd None) >Thread-150::DEBUG::2012-12-03 00:09:47,732::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-150::DEBUG::2012-12-03 00:09:47,733::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-150::DEBUG::2012-12-03 00:09:47,733::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-150::DEBUG::2012-12-03 00:09:47,733::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-150::DEBUG::2012-12-03 00:09:47,733::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-150::DEBUG::2012-12-03 00:09:47,733::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-150::INFO::2012-12-03 00:09:47,734::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-150::DEBUG::2012-12-03 00:09:47,734::task::1151::TaskManager.Task::(prepare) Task=`76cb1efd-a620-4710-987e-d6139b4aafdd`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-150::DEBUG::2012-12-03 00:09:47,734::task::568::TaskManager.Task::(_updateState) Task=`76cb1efd-a620-4710-987e-d6139b4aafdd`::moving from state preparing -> state finished >Thread-150::DEBUG::2012-12-03 00:09:47,734::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-150::DEBUG::2012-12-03 00:09:47,734::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-150::DEBUG::2012-12-03 00:09:47,734::task::957::TaskManager.Task::(_decref) Task=`76cb1efd-a620-4710-987e-d6139b4aafdd`::ref 0 aborting False >Thread-151::DEBUG::2012-12-03 00:09:47,926::BindingXMLRPC::160::vds::(wrapper) [10.35.202.48] >Thread-151::DEBUG::2012-12-03 00:09:47,927::task::568::TaskManager.Task::(_updateState) Task=`1ad40f35-df76-4404-aa1d-b0e1010de18d`::moving from state init -> state preparing >Thread-151::INFO::2012-12-03 00:09:47,927::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='a00efcc1-5685-4dbe-ae70-440e9f403b76', masterVersion=4, options=None) >Thread-151::DEBUG::2012-12-03 00:09:47,927::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8dd40f6e-8c49-4092-b7b0-a5bf2a49c292`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-151::DEBUG::2012-12-03 00:09:47,927::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-151::DEBUG::2012-12-03 00:09:47,928::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-151::DEBUG::2012-12-03 00:09:47,928::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`8dd40f6e-8c49-4092-b7b0-a5bf2a49c292`::Granted request >Thread-151::INFO::2012-12-03 00:09:47,928::sp::622::Storage.StoragePool::(connect) Connect host #2 to the storage pool 479cf195-bb99-45c5-b400-433d810eaa48 with master domain: a00efcc1-5685-4dbe-ae70-440e9f403b76 (ver = 4) >Thread-151::DEBUG::2012-12-03 00:09:47,928::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:47,929::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:47,929::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:47,929::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:47,929::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:47,929::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:47,929::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-151::DEBUG::2012-12-03 00:09:47,930::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-151::DEBUG::2012-12-03 00:09:47,930::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-151::DEBUG::2012-12-03 00:09:47,930::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-151::DEBUG::2012-12-03 00:09:47,930::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-151::DEBUG::2012-12-03 00:09:47,954::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-151::DEBUG::2012-12-03 00:09:47,954::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-151::DEBUG::2012-12-03 00:09:48,588::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-151::DEBUG::2012-12-03 00:09:48,668::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-151::DEBUG::2012-12-03 00:09:48,668::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,668::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,668::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,669::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,669::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,669::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,669::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-151::DEBUG::2012-12-03 00:09:48,669::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,671::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-151::DEBUG::2012-12-03 00:09:48,883::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-151::WARNING::2012-12-03 00:09:48,884::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-151::DEBUG::2012-12-03 00:09:48,885::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:48,926::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-151::DEBUG::2012-12-03 00:09:48,926::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-151::DEBUG::2012-12-03 00:09:48,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f88715a0e40bd7b8575a75e585467485b3ce9fc4'] >Thread-151::DEBUG::2012-12-03 00:09:49,009::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-151::DEBUG::2012-12-03 00:09:49,010::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS' >Thread-151::DEBUG::2012-12-03 00:09:49,010::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS' >Thread-151::DEBUG::2012-12-03 00:09:49,010::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain a00efcc1-5685-4dbe-ae70-440e9f403b76 verified, version 4 >Thread-151::DEBUG::2012-12-03 00:09:49,010::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-151::DEBUG::2012-12-03 00:09:49,010::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-151::DEBUG::2012-12-03 00:09:49,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f88715a0e40bd7b8575a75e585467485b3ce9fc4'] >Thread-151::DEBUG::2012-12-03 00:09:49,019::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:49,020::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-151::DEBUG::2012-12-03 00:09:49,224::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-151::WARNING::2012-12-03 00:09:49,226::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-151::DEBUG::2012-12-03 00:09:49,226::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-151::ERROR::2012-12-03 00:09:49,256::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-151::DEBUG::2012-12-03 00:09:49,257::sp::1554::Storage.StoragePool::(updateMonitoringThreads) Storage Pool `479cf195-bb99-45c5-b400-433d810eaa48` started monitoring domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` >Thread-151::DEBUG::2012-12-03 00:09:49,257::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:49,258::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-151::DEBUG::2012-12-03 00:09:49,462::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-151::WARNING::2012-12-03 00:09:49,464::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-151::DEBUG::2012-12-03 00:09:49,464::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-151::ERROR::2012-12-03 00:09:49,493::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `5971512c-ee51-475e-b956-3872c38ce193` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('5971512c-ee51-475e-b956-3872c38ce193',) >Thread-151::DEBUG::2012-12-03 00:09:49,494::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:49,495::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-151::DEBUG::2012-12-03 00:09:49,700::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-151::WARNING::2012-12-03 00:09:49,702::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-151::DEBUG::2012-12-03 00:09:49,702::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-151::ERROR::2012-12-03 00:09:49,732::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-151::DEBUG::2012-12-03 00:09:49,732::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-151::DEBUG::2012-12-03 00:09:49,733::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-151::DEBUG::2012-12-03 00:09:49,734::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4 5971512c-ee51-475e-b956-3872c38ce193 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-276::INFO::2012-12-03 00:09:49,744::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 (id: 2) >Thread-276::DEBUG::2012-12-03 00:09:49,744::safelease::178::SANLock::(acquireHostId) Host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully acquired (id: 2) >Thread-151::DEBUG::2012-12-03 00:09:50,032::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-151::WARNING::2012-12-03 00:09:50,033::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found', ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found', ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-151::DEBUG::2012-12-03 00:09:50,033::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-151::INFO::2012-12-03 00:09:50,063::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-151::INFO::2012-12-03 00:09:50,064::sp::1118::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 to /rhev/data-center/479cf195-bb99-45c5-b400-433d810eaa48/mastersd >Thread-151::DEBUG::2012-12-03 00:09:50,064::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-151::DEBUG::2012-12-03 00:09:50,064::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-151::DEBUG::2012-12-03 00:09:50,065::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-151::DEBUG::2012-12-03 00:09:50,065::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-151::INFO::2012-12-03 00:09:50,065::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-151::DEBUG::2012-12-03 00:09:50,065::task::1151::TaskManager.Task::(prepare) Task=`1ad40f35-df76-4404-aa1d-b0e1010de18d`::finished: True >Thread-151::DEBUG::2012-12-03 00:09:50,065::task::568::TaskManager.Task::(_updateState) Task=`1ad40f35-df76-4404-aa1d-b0e1010de18d`::moving from state preparing -> state finished >Thread-151::DEBUG::2012-12-03 00:09:50,065::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-151::DEBUG::2012-12-03 00:09:50,065::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-151::DEBUG::2012-12-03 00:09:50,066::task::957::TaskManager.Task::(_decref) Task=`1ad40f35-df76-4404-aa1d-b0e1010de18d`::ref 0 aborting False >Thread-462::DEBUG::2012-12-03 00:09:50,210::task::568::TaskManager.Task::(_updateState) Task=`67f8e26d-293e-4307-a63f-3cc9fc8d0010`::moving from state init -> state preparing >Thread-462::INFO::2012-12-03 00:09:50,211::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-462::INFO::2012-12-03 00:09:50,211::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00668597221375', 'lastCheck': 1354486189.7440569, 'code': 0, 'valid': True}} >Thread-462::DEBUG::2012-12-03 00:09:50,211::task::1151::TaskManager.Task::(prepare) Task=`67f8e26d-293e-4307-a63f-3cc9fc8d0010`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00668597221375', 'lastCheck': 1354486189.7440569, 'code': 0, 'valid': True}} >Thread-462::DEBUG::2012-12-03 00:09:50,211::task::568::TaskManager.Task::(_updateState) Task=`67f8e26d-293e-4307-a63f-3cc9fc8d0010`::moving from state preparing -> state finished >Thread-462::DEBUG::2012-12-03 00:09:50,211::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-462::DEBUG::2012-12-03 00:09:50,211::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-462::DEBUG::2012-12-03 00:09:50,211::task::957::TaskManager.Task::(_decref) Task=`67f8e26d-293e-4307-a63f-3cc9fc8d0010`::ref 0 aborting False >Thread-466::DEBUG::2012-12-03 00:09:58,832::task::568::TaskManager.Task::(_updateState) Task=`52718219-ec12-4cc4-8bdb-8201b2168fcc`::moving from state init -> state preparing >Thread-466::INFO::2012-12-03 00:09:58,832::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-466::INFO::2012-12-03 00:09:58,832::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00668597221375', 'lastCheck': 1354486189.7440569, 'code': 0, 'valid': True}} >Thread-466::DEBUG::2012-12-03 00:09:58,833::task::1151::TaskManager.Task::(prepare) Task=`52718219-ec12-4cc4-8bdb-8201b2168fcc`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00668597221375', 'lastCheck': 1354486189.7440569, 'code': 0, 'valid': True}} >Thread-466::DEBUG::2012-12-03 00:09:58,833::task::568::TaskManager.Task::(_updateState) Task=`52718219-ec12-4cc4-8bdb-8201b2168fcc`::moving from state preparing -> state finished >Thread-466::DEBUG::2012-12-03 00:09:58,833::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-466::DEBUG::2012-12-03 00:09:58,833::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-466::DEBUG::2012-12-03 00:09:58,833::task::957::TaskManager.Task::(_decref) Task=`52718219-ec12-4cc4-8bdb-8201b2168fcc`::ref 0 aborting False >Thread-472::DEBUG::2012-12-03 00:10:09,752::task::568::TaskManager.Task::(_updateState) Task=`0aed4f58-ced1-4f63-b9e5-fc1fe6d672a2`::moving from state init -> state preparing >Thread-472::INFO::2012-12-03 00:10:09,753::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-472::INFO::2012-12-03 00:10:09,753::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00933599472046', 'lastCheck': 1354486199.7657781, 'code': 0, 'valid': True}} >Thread-472::DEBUG::2012-12-03 00:10:09,753::task::1151::TaskManager.Task::(prepare) Task=`0aed4f58-ced1-4f63-b9e5-fc1fe6d672a2`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00933599472046', 'lastCheck': 1354486199.7657781, 'code': 0, 'valid': True}} >Thread-472::DEBUG::2012-12-03 00:10:09,753::task::568::TaskManager.Task::(_updateState) Task=`0aed4f58-ced1-4f63-b9e5-fc1fe6d672a2`::moving from state preparing -> state finished >Thread-472::DEBUG::2012-12-03 00:10:09,753::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-472::DEBUG::2012-12-03 00:10:09,753::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-472::DEBUG::2012-12-03 00:10:09,753::task::957::TaskManager.Task::(_decref) Task=`0aed4f58-ced1-4f63-b9e5-fc1fe6d672a2`::ref 0 aborting False >Thread-478::DEBUG::2012-12-03 00:10:20,696::task::568::TaskManager.Task::(_updateState) Task=`254c6068-6758-49a2-8ba8-57c3292c2a37`::moving from state init -> state preparing >Thread-478::INFO::2012-12-03 00:10:20,696::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-478::INFO::2012-12-03 00:10:20,697::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00856709480286', 'lastCheck': 1354486219.8113739, 'code': 0, 'valid': True}} >Thread-478::DEBUG::2012-12-03 00:10:20,697::task::1151::TaskManager.Task::(prepare) Task=`254c6068-6758-49a2-8ba8-57c3292c2a37`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00856709480286', 'lastCheck': 1354486219.8113739, 'code': 0, 'valid': True}} >Thread-478::DEBUG::2012-12-03 00:10:20,697::task::568::TaskManager.Task::(_updateState) Task=`254c6068-6758-49a2-8ba8-57c3292c2a37`::moving from state preparing -> state finished >Thread-478::DEBUG::2012-12-03 00:10:20,697::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-478::DEBUG::2012-12-03 00:10:20,697::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-478::DEBUG::2012-12-03 00:10:20,697::task::957::TaskManager.Task::(_decref) Task=`254c6068-6758-49a2-8ba8-57c3292c2a37`::ref 0 aborting False >Thread-484::DEBUG::2012-12-03 00:10:31,636::task::568::TaskManager.Task::(_updateState) Task=`be3a10da-adcb-4e9e-aa3f-8da36fcad2c1`::moving from state init -> state preparing >Thread-484::INFO::2012-12-03 00:10:31,636::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-484::INFO::2012-12-03 00:10:31,636::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00836181640625', 'lastCheck': 1354486229.8306341, 'code': 0, 'valid': True}} >Thread-484::DEBUG::2012-12-03 00:10:31,636::task::1151::TaskManager.Task::(prepare) Task=`be3a10da-adcb-4e9e-aa3f-8da36fcad2c1`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00836181640625', 'lastCheck': 1354486229.8306341, 'code': 0, 'valid': True}} >Thread-484::DEBUG::2012-12-03 00:10:31,636::task::568::TaskManager.Task::(_updateState) Task=`be3a10da-adcb-4e9e-aa3f-8da36fcad2c1`::moving from state preparing -> state finished >Thread-484::DEBUG::2012-12-03 00:10:31,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-484::DEBUG::2012-12-03 00:10:31,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-484::DEBUG::2012-12-03 00:10:31,637::task::957::TaskManager.Task::(_decref) Task=`be3a10da-adcb-4e9e-aa3f-8da36fcad2c1`::ref 0 aborting False >Thread-490::DEBUG::2012-12-03 00:10:42,548::task::568::TaskManager.Task::(_updateState) Task=`47701ceb-2f3c-407b-8765-0686d5a18c72`::moving from state init -> state preparing >Thread-490::INFO::2012-12-03 00:10:42,549::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-490::INFO::2012-12-03 00:10:42,549::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00836515426636', 'lastCheck': 1354486239.8480761, 'code': 0, 'valid': True}} >Thread-490::DEBUG::2012-12-03 00:10:42,549::task::1151::TaskManager.Task::(prepare) Task=`47701ceb-2f3c-407b-8765-0686d5a18c72`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00836515426636', 'lastCheck': 1354486239.8480761, 'code': 0, 'valid': True}} >Thread-490::DEBUG::2012-12-03 00:10:42,549::task::568::TaskManager.Task::(_updateState) Task=`47701ceb-2f3c-407b-8765-0686d5a18c72`::moving from state preparing -> state finished >Thread-490::DEBUG::2012-12-03 00:10:42,549::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-490::DEBUG::2012-12-03 00:10:42,549::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-490::DEBUG::2012-12-03 00:10:42,550::task::957::TaskManager.Task::(_decref) Task=`47701ceb-2f3c-407b-8765-0686d5a18c72`::ref 0 aborting False >Thread-496::DEBUG::2012-12-03 00:10:53,479::task::568::TaskManager.Task::(_updateState) Task=`5f120a68-0d52-40c6-b03c-1b41711eabe8`::moving from state init -> state preparing >Thread-496::INFO::2012-12-03 00:10:53,479::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-496::INFO::2012-12-03 00:10:53,479::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00940108299255', 'lastCheck': 1354486249.8682871, 'code': 0, 'valid': True}} >Thread-496::DEBUG::2012-12-03 00:10:53,479::task::1151::TaskManager.Task::(prepare) Task=`5f120a68-0d52-40c6-b03c-1b41711eabe8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00940108299255', 'lastCheck': 1354486249.8682871, 'code': 0, 'valid': True}} >Thread-496::DEBUG::2012-12-03 00:10:53,480::task::568::TaskManager.Task::(_updateState) Task=`5f120a68-0d52-40c6-b03c-1b41711eabe8`::moving from state preparing -> state finished >Thread-496::DEBUG::2012-12-03 00:10:53,480::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-496::DEBUG::2012-12-03 00:10:53,480::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-496::DEBUG::2012-12-03 00:10:53,480::task::957::TaskManager.Task::(_decref) Task=`5f120a68-0d52-40c6-b03c-1b41711eabe8`::ref 0 aborting False >Thread-502::DEBUG::2012-12-03 00:11:04,370::task::568::TaskManager.Task::(_updateState) Task=`93269094-a01f-4aac-9273-2ae55c327098`::moving from state init -> state preparing >Thread-502::INFO::2012-12-03 00:11:04,370::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-502::INFO::2012-12-03 00:11:04,370::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00814390182495', 'lastCheck': 1354486259.8878839, 'code': 0, 'valid': True}} >Thread-502::DEBUG::2012-12-03 00:11:04,370::task::1151::TaskManager.Task::(prepare) Task=`93269094-a01f-4aac-9273-2ae55c327098`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00814390182495', 'lastCheck': 1354486259.8878839, 'code': 0, 'valid': True}} >Thread-502::DEBUG::2012-12-03 00:11:04,371::task::568::TaskManager.Task::(_updateState) Task=`93269094-a01f-4aac-9273-2ae55c327098`::moving from state preparing -> state finished >Thread-502::DEBUG::2012-12-03 00:11:04,371::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-502::DEBUG::2012-12-03 00:11:04,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-502::DEBUG::2012-12-03 00:11:04,371::task::957::TaskManager.Task::(_decref) Task=`93269094-a01f-4aac-9273-2ae55c327098`::ref 0 aborting False >Thread-508::DEBUG::2012-12-03 00:11:15,256::task::568::TaskManager.Task::(_updateState) Task=`449804c5-e7dc-45a4-8929-5b4ba6ee30eb`::moving from state init -> state preparing >Thread-508::INFO::2012-12-03 00:11:15,256::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-508::INFO::2012-12-03 00:11:15,257::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00785493850708', 'lastCheck': 1354486269.906204, 'code': 0, 'valid': True}} >Thread-508::DEBUG::2012-12-03 00:11:15,257::task::1151::TaskManager.Task::(prepare) Task=`449804c5-e7dc-45a4-8929-5b4ba6ee30eb`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00785493850708', 'lastCheck': 1354486269.906204, 'code': 0, 'valid': True}} >Thread-508::DEBUG::2012-12-03 00:11:15,257::task::568::TaskManager.Task::(_updateState) Task=`449804c5-e7dc-45a4-8929-5b4ba6ee30eb`::moving from state preparing -> state finished >Thread-508::DEBUG::2012-12-03 00:11:15,257::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-508::DEBUG::2012-12-03 00:11:15,257::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-508::DEBUG::2012-12-03 00:11:15,257::task::957::TaskManager.Task::(_decref) Task=`449804c5-e7dc-45a4-8929-5b4ba6ee30eb`::ref 0 aborting False >Thread-514::DEBUG::2012-12-03 00:11:26,209::task::568::TaskManager.Task::(_updateState) Task=`b7199264-c3f5-4a1f-9b3d-aec8c21db77e`::moving from state init -> state preparing >Thread-514::INFO::2012-12-03 00:11:26,209::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-514::INFO::2012-12-03 00:11:26,209::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00944399833679', 'lastCheck': 1354486279.93168, 'code': 0, 'valid': True}} >Thread-514::DEBUG::2012-12-03 00:11:26,209::task::1151::TaskManager.Task::(prepare) Task=`b7199264-c3f5-4a1f-9b3d-aec8c21db77e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00944399833679', 'lastCheck': 1354486279.93168, 'code': 0, 'valid': True}} >Thread-514::DEBUG::2012-12-03 00:11:26,209::task::568::TaskManager.Task::(_updateState) Task=`b7199264-c3f5-4a1f-9b3d-aec8c21db77e`::moving from state preparing -> state finished >Thread-514::DEBUG::2012-12-03 00:11:26,209::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-514::DEBUG::2012-12-03 00:11:26,210::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-514::DEBUG::2012-12-03 00:11:26,210::task::957::TaskManager.Task::(_decref) Task=`b7199264-c3f5-4a1f-9b3d-aec8c21db77e`::ref 0 aborting False >Thread-520::DEBUG::2012-12-03 00:11:37,133::task::568::TaskManager.Task::(_updateState) Task=`beff2501-d0b4-4682-b38b-f02227eb8a11`::moving from state init -> state preparing >Thread-520::INFO::2012-12-03 00:11:37,134::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-520::INFO::2012-12-03 00:11:37,134::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0130739212036', 'lastCheck': 1354486289.9562731, 'code': 0, 'valid': True}} >Thread-520::DEBUG::2012-12-03 00:11:37,134::task::1151::TaskManager.Task::(prepare) Task=`beff2501-d0b4-4682-b38b-f02227eb8a11`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0130739212036', 'lastCheck': 1354486289.9562731, 'code': 0, 'valid': True}} >Thread-520::DEBUG::2012-12-03 00:11:37,134::task::568::TaskManager.Task::(_updateState) Task=`beff2501-d0b4-4682-b38b-f02227eb8a11`::moving from state preparing -> state finished >Thread-520::DEBUG::2012-12-03 00:11:37,134::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-520::DEBUG::2012-12-03 00:11:37,134::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-520::DEBUG::2012-12-03 00:11:37,135::task::957::TaskManager.Task::(_decref) Task=`beff2501-d0b4-4682-b38b-f02227eb8a11`::ref 0 aborting False >Thread-526::DEBUG::2012-12-03 00:11:48,101::task::568::TaskManager.Task::(_updateState) Task=`abb2a5f4-675c-41e0-9363-eb6686faa990`::moving from state init -> state preparing >Thread-526::INFO::2012-12-03 00:11:48,101::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-526::INFO::2012-12-03 00:11:48,101::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00928616523743', 'lastCheck': 1354486299.975482, 'code': 0, 'valid': True}} >Thread-526::DEBUG::2012-12-03 00:11:48,101::task::1151::TaskManager.Task::(prepare) Task=`abb2a5f4-675c-41e0-9363-eb6686faa990`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00928616523743', 'lastCheck': 1354486299.975482, 'code': 0, 'valid': True}} >Thread-526::DEBUG::2012-12-03 00:11:48,102::task::568::TaskManager.Task::(_updateState) Task=`abb2a5f4-675c-41e0-9363-eb6686faa990`::moving from state preparing -> state finished >Thread-526::DEBUG::2012-12-03 00:11:48,102::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-526::DEBUG::2012-12-03 00:11:48,102::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-526::DEBUG::2012-12-03 00:11:48,102::task::957::TaskManager.Task::(_decref) Task=`abb2a5f4-675c-41e0-9363-eb6686faa990`::ref 0 aborting False >Thread-532::DEBUG::2012-12-03 00:11:59,052::task::568::TaskManager.Task::(_updateState) Task=`cfcd9af8-bfe6-44b9-b0a4-78bb12d3559e`::moving from state init -> state preparing >Thread-532::INFO::2012-12-03 00:11:59,053::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-532::INFO::2012-12-03 00:11:59,053::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00965881347656', 'lastCheck': 1354486309.9976029, 'code': 0, 'valid': True}} >Thread-532::DEBUG::2012-12-03 00:11:59,053::task::1151::TaskManager.Task::(prepare) Task=`cfcd9af8-bfe6-44b9-b0a4-78bb12d3559e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00965881347656', 'lastCheck': 1354486309.9976029, 'code': 0, 'valid': True}} >Thread-532::DEBUG::2012-12-03 00:11:59,053::task::568::TaskManager.Task::(_updateState) Task=`cfcd9af8-bfe6-44b9-b0a4-78bb12d3559e`::moving from state preparing -> state finished >Thread-532::DEBUG::2012-12-03 00:11:59,053::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-532::DEBUG::2012-12-03 00:11:59,053::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-532::DEBUG::2012-12-03 00:11:59,054::task::957::TaskManager.Task::(_decref) Task=`cfcd9af8-bfe6-44b9-b0a4-78bb12d3559e`::ref 0 aborting False >Thread-538::DEBUG::2012-12-03 00:12:10,061::task::568::TaskManager.Task::(_updateState) Task=`8af862fa-9313-47ef-8dc8-94f1e3bcf9d2`::moving from state init -> state preparing >Thread-538::INFO::2012-12-03 00:12:10,062::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-538::INFO::2012-12-03 00:12:10,062::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00348401069641', 'lastCheck': 1354486330.03426, 'code': 0, 'valid': True}} >Thread-538::DEBUG::2012-12-03 00:12:10,062::task::1151::TaskManager.Task::(prepare) Task=`8af862fa-9313-47ef-8dc8-94f1e3bcf9d2`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00348401069641', 'lastCheck': 1354486330.03426, 'code': 0, 'valid': True}} >Thread-538::DEBUG::2012-12-03 00:12:10,062::task::568::TaskManager.Task::(_updateState) Task=`8af862fa-9313-47ef-8dc8-94f1e3bcf9d2`::moving from state preparing -> state finished >Thread-538::DEBUG::2012-12-03 00:12:10,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-538::DEBUG::2012-12-03 00:12:10,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-538::DEBUG::2012-12-03 00:12:10,062::task::957::TaskManager.Task::(_decref) Task=`8af862fa-9313-47ef-8dc8-94f1e3bcf9d2`::ref 0 aborting False >Thread-544::DEBUG::2012-12-03 00:12:21,061::task::568::TaskManager.Task::(_updateState) Task=`d0a8aca1-27f4-4140-ba36-ecfbe118deb5`::moving from state init -> state preparing >Thread-544::INFO::2012-12-03 00:12:21,061::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-544::INFO::2012-12-03 00:12:21,061::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00845408439636', 'lastCheck': 1354486340.056895, 'code': 0, 'valid': True}} >Thread-544::DEBUG::2012-12-03 00:12:21,062::task::1151::TaskManager.Task::(prepare) Task=`d0a8aca1-27f4-4140-ba36-ecfbe118deb5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00845408439636', 'lastCheck': 1354486340.056895, 'code': 0, 'valid': True}} >Thread-544::DEBUG::2012-12-03 00:12:21,062::task::568::TaskManager.Task::(_updateState) Task=`d0a8aca1-27f4-4140-ba36-ecfbe118deb5`::moving from state preparing -> state finished >Thread-544::DEBUG::2012-12-03 00:12:21,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-544::DEBUG::2012-12-03 00:12:21,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-544::DEBUG::2012-12-03 00:12:21,062::task::957::TaskManager.Task::(_decref) Task=`d0a8aca1-27f4-4140-ba36-ecfbe118deb5`::ref 0 aborting False >Thread-550::DEBUG::2012-12-03 00:12:31,978::task::568::TaskManager.Task::(_updateState) Task=`0a01ff1c-2261-459d-96a0-81df27cf6f60`::moving from state init -> state preparing >Thread-550::INFO::2012-12-03 00:12:31,978::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-550::INFO::2012-12-03 00:12:31,978::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0101618766785', 'lastCheck': 1354486350.0781989, 'code': 0, 'valid': True}} >Thread-550::DEBUG::2012-12-03 00:12:31,979::task::1151::TaskManager.Task::(prepare) Task=`0a01ff1c-2261-459d-96a0-81df27cf6f60`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0101618766785', 'lastCheck': 1354486350.0781989, 'code': 0, 'valid': True}} >Thread-550::DEBUG::2012-12-03 00:12:31,979::task::568::TaskManager.Task::(_updateState) Task=`0a01ff1c-2261-459d-96a0-81df27cf6f60`::moving from state preparing -> state finished >Thread-550::DEBUG::2012-12-03 00:12:31,979::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-550::DEBUG::2012-12-03 00:12:31,979::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-550::DEBUG::2012-12-03 00:12:31,979::task::957::TaskManager.Task::(_decref) Task=`0a01ff1c-2261-459d-96a0-81df27cf6f60`::ref 0 aborting False >Thread-556::DEBUG::2012-12-03 00:12:42,852::task::568::TaskManager.Task::(_updateState) Task=`c5fe1767-6225-4bcc-9c7f-44a25c790475`::moving from state init -> state preparing >Thread-556::INFO::2012-12-03 00:12:42,852::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-556::INFO::2012-12-03 00:12:42,852::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00826096534729', 'lastCheck': 1354486360.098393, 'code': 0, 'valid': True}} >Thread-556::DEBUG::2012-12-03 00:12:42,852::task::1151::TaskManager.Task::(prepare) Task=`c5fe1767-6225-4bcc-9c7f-44a25c790475`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00826096534729', 'lastCheck': 1354486360.098393, 'code': 0, 'valid': True}} >Thread-556::DEBUG::2012-12-03 00:12:42,852::task::568::TaskManager.Task::(_updateState) Task=`c5fe1767-6225-4bcc-9c7f-44a25c790475`::moving from state preparing -> state finished >Thread-556::DEBUG::2012-12-03 00:12:42,852::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-556::DEBUG::2012-12-03 00:12:42,853::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-556::DEBUG::2012-12-03 00:12:42,853::task::957::TaskManager.Task::(_decref) Task=`c5fe1767-6225-4bcc-9c7f-44a25c790475`::ref 0 aborting False >Thread-562::DEBUG::2012-12-03 00:12:53,752::task::568::TaskManager.Task::(_updateState) Task=`ee11cb39-cceb-4152-80d4-9cdb56342189`::moving from state init -> state preparing >Thread-562::INFO::2012-12-03 00:12:53,752::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-562::INFO::2012-12-03 00:12:53,752::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00916981697083', 'lastCheck': 1354486370.1197879, 'code': 0, 'valid': True}} >Thread-562::DEBUG::2012-12-03 00:12:53,752::task::1151::TaskManager.Task::(prepare) Task=`ee11cb39-cceb-4152-80d4-9cdb56342189`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00916981697083', 'lastCheck': 1354486370.1197879, 'code': 0, 'valid': True}} >Thread-562::DEBUG::2012-12-03 00:12:53,753::task::568::TaskManager.Task::(_updateState) Task=`ee11cb39-cceb-4152-80d4-9cdb56342189`::moving from state preparing -> state finished >Thread-562::DEBUG::2012-12-03 00:12:53,753::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-562::DEBUG::2012-12-03 00:12:53,753::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-562::DEBUG::2012-12-03 00:12:53,753::task::957::TaskManager.Task::(_decref) Task=`ee11cb39-cceb-4152-80d4-9cdb56342189`::ref 0 aborting False >Thread-568::DEBUG::2012-12-03 00:13:04,675::task::568::TaskManager.Task::(_updateState) Task=`617396b4-4ec4-4392-9555-c9f5a6d39ab1`::moving from state init -> state preparing >Thread-568::INFO::2012-12-03 00:13:04,675::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-568::INFO::2012-12-03 00:13:04,675::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0108320713043', 'lastCheck': 1354486380.1407681, 'code': 0, 'valid': True}} >Thread-568::DEBUG::2012-12-03 00:13:04,675::task::1151::TaskManager.Task::(prepare) Task=`617396b4-4ec4-4392-9555-c9f5a6d39ab1`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0108320713043', 'lastCheck': 1354486380.1407681, 'code': 0, 'valid': True}} >Thread-568::DEBUG::2012-12-03 00:13:04,675::task::568::TaskManager.Task::(_updateState) Task=`617396b4-4ec4-4392-9555-c9f5a6d39ab1`::moving from state preparing -> state finished >Thread-568::DEBUG::2012-12-03 00:13:04,675::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-568::DEBUG::2012-12-03 00:13:04,675::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-568::DEBUG::2012-12-03 00:13:04,676::task::957::TaskManager.Task::(_decref) Task=`617396b4-4ec4-4392-9555-c9f5a6d39ab1`::ref 0 aborting False >Thread-574::DEBUG::2012-12-03 00:13:15,573::task::568::TaskManager.Task::(_updateState) Task=`b7276546-3713-4a3b-b094-0a4328dd0921`::moving from state init -> state preparing >Thread-574::INFO::2012-12-03 00:13:15,573::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-574::INFO::2012-12-03 00:13:15,574::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00944995880127', 'lastCheck': 1354486390.160332, 'code': 0, 'valid': True}} >Thread-574::DEBUG::2012-12-03 00:13:15,574::task::1151::TaskManager.Task::(prepare) Task=`b7276546-3713-4a3b-b094-0a4328dd0921`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00944995880127', 'lastCheck': 1354486390.160332, 'code': 0, 'valid': True}} >Thread-574::DEBUG::2012-12-03 00:13:15,574::task::568::TaskManager.Task::(_updateState) Task=`b7276546-3713-4a3b-b094-0a4328dd0921`::moving from state preparing -> state finished >Thread-574::DEBUG::2012-12-03 00:13:15,574::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-574::DEBUG::2012-12-03 00:13:15,574::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-574::DEBUG::2012-12-03 00:13:15,574::task::957::TaskManager.Task::(_decref) Task=`b7276546-3713-4a3b-b094-0a4328dd0921`::ref 0 aborting False >Thread-580::DEBUG::2012-12-03 00:13:26,501::task::568::TaskManager.Task::(_updateState) Task=`2201f3e3-5571-4f3e-af48-f13c5d9f29f0`::moving from state init -> state preparing >Thread-580::INFO::2012-12-03 00:13:26,501::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-580::INFO::2012-12-03 00:13:26,501::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00953197479248', 'lastCheck': 1354486400.1858399, 'code': 0, 'valid': True}} >Thread-580::DEBUG::2012-12-03 00:13:26,501::task::1151::TaskManager.Task::(prepare) Task=`2201f3e3-5571-4f3e-af48-f13c5d9f29f0`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00953197479248', 'lastCheck': 1354486400.1858399, 'code': 0, 'valid': True}} >Thread-580::DEBUG::2012-12-03 00:13:26,502::task::568::TaskManager.Task::(_updateState) Task=`2201f3e3-5571-4f3e-af48-f13c5d9f29f0`::moving from state preparing -> state finished >Thread-580::DEBUG::2012-12-03 00:13:26,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-580::DEBUG::2012-12-03 00:13:26,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-580::DEBUG::2012-12-03 00:13:26,502::task::957::TaskManager.Task::(_decref) Task=`2201f3e3-5571-4f3e-af48-f13c5d9f29f0`::ref 0 aborting False >Thread-586::DEBUG::2012-12-03 00:13:37,374::task::568::TaskManager.Task::(_updateState) Task=`e77cf828-898e-4407-8d31-e9b02d0b5698`::moving from state init -> state preparing >Thread-586::INFO::2012-12-03 00:13:37,374::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-586::INFO::2012-12-03 00:13:37,375::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00771713256836', 'lastCheck': 1354486410.203573, 'code': 0, 'valid': True}} >Thread-586::DEBUG::2012-12-03 00:13:37,375::task::1151::TaskManager.Task::(prepare) Task=`e77cf828-898e-4407-8d31-e9b02d0b5698`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00771713256836', 'lastCheck': 1354486410.203573, 'code': 0, 'valid': True}} >Thread-586::DEBUG::2012-12-03 00:13:37,375::task::568::TaskManager.Task::(_updateState) Task=`e77cf828-898e-4407-8d31-e9b02d0b5698`::moving from state preparing -> state finished >Thread-586::DEBUG::2012-12-03 00:13:37,375::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-586::DEBUG::2012-12-03 00:13:37,375::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-586::DEBUG::2012-12-03 00:13:37,375::task::957::TaskManager.Task::(_decref) Task=`e77cf828-898e-4407-8d31-e9b02d0b5698`::ref 0 aborting False >Thread-592::DEBUG::2012-12-03 00:13:48,294::task::568::TaskManager.Task::(_updateState) Task=`a866dc45-11be-4dfb-8207-e16ae19ef618`::moving from state init -> state preparing >Thread-592::INFO::2012-12-03 00:13:48,294::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-592::INFO::2012-12-03 00:13:48,294::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00824189186096', 'lastCheck': 1354486420.221204, 'code': 0, 'valid': True}} >Thread-592::DEBUG::2012-12-03 00:13:48,295::task::1151::TaskManager.Task::(prepare) Task=`a866dc45-11be-4dfb-8207-e16ae19ef618`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00824189186096', 'lastCheck': 1354486420.221204, 'code': 0, 'valid': True}} >Thread-592::DEBUG::2012-12-03 00:13:48,295::task::568::TaskManager.Task::(_updateState) Task=`a866dc45-11be-4dfb-8207-e16ae19ef618`::moving from state preparing -> state finished >Thread-592::DEBUG::2012-12-03 00:13:48,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-592::DEBUG::2012-12-03 00:13:48,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-592::DEBUG::2012-12-03 00:13:48,295::task::957::TaskManager.Task::(_decref) Task=`a866dc45-11be-4dfb-8207-e16ae19ef618`::ref 0 aborting False >Thread-598::DEBUG::2012-12-03 00:13:59,258::task::568::TaskManager.Task::(_updateState) Task=`ab207c12-7918-4fbd-adca-f85fbe8fd7c4`::moving from state init -> state preparing >Thread-598::INFO::2012-12-03 00:13:59,258::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-598::INFO::2012-12-03 00:13:59,258::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00915598869324', 'lastCheck': 1354486430.241137, 'code': 0, 'valid': True}} >Thread-598::DEBUG::2012-12-03 00:13:59,258::task::1151::TaskManager.Task::(prepare) Task=`ab207c12-7918-4fbd-adca-f85fbe8fd7c4`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00915598869324', 'lastCheck': 1354486430.241137, 'code': 0, 'valid': True}} >Thread-598::DEBUG::2012-12-03 00:13:59,258::task::568::TaskManager.Task::(_updateState) Task=`ab207c12-7918-4fbd-adca-f85fbe8fd7c4`::moving from state preparing -> state finished >Thread-598::DEBUG::2012-12-03 00:13:59,258::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-598::DEBUG::2012-12-03 00:13:59,259::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-598::DEBUG::2012-12-03 00:13:59,259::task::957::TaskManager.Task::(_decref) Task=`ab207c12-7918-4fbd-adca-f85fbe8fd7c4`::ref 0 aborting False >Thread-604::DEBUG::2012-12-03 00:14:10,247::task::568::TaskManager.Task::(_updateState) Task=`91694a72-e9d1-430d-9e4d-62a05f0ba1fd`::moving from state init -> state preparing >Thread-604::INFO::2012-12-03 00:14:10,247::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-604::INFO::2012-12-03 00:14:10,247::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00941514968872', 'lastCheck': 1354486440.2604561, 'code': 0, 'valid': True}} >Thread-604::DEBUG::2012-12-03 00:14:10,248::task::1151::TaskManager.Task::(prepare) Task=`91694a72-e9d1-430d-9e4d-62a05f0ba1fd`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00941514968872', 'lastCheck': 1354486440.2604561, 'code': 0, 'valid': True}} >Thread-604::DEBUG::2012-12-03 00:14:10,248::task::568::TaskManager.Task::(_updateState) Task=`91694a72-e9d1-430d-9e4d-62a05f0ba1fd`::moving from state preparing -> state finished >Thread-604::DEBUG::2012-12-03 00:14:10,248::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-604::DEBUG::2012-12-03 00:14:10,248::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-604::DEBUG::2012-12-03 00:14:10,248::task::957::TaskManager.Task::(_decref) Task=`91694a72-e9d1-430d-9e4d-62a05f0ba1fd`::ref 0 aborting False >Thread-610::DEBUG::2012-12-03 00:14:21,204::task::568::TaskManager.Task::(_updateState) Task=`abbe7c3c-d670-43bf-88d7-f6c24b67476d`::moving from state init -> state preparing >Thread-610::INFO::2012-12-03 00:14:21,204::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-610::INFO::2012-12-03 00:14:21,205::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00423789024353', 'lastCheck': 1354486460.29461, 'code': 0, 'valid': True}} >Thread-610::DEBUG::2012-12-03 00:14:21,205::task::1151::TaskManager.Task::(prepare) Task=`abbe7c3c-d670-43bf-88d7-f6c24b67476d`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00423789024353', 'lastCheck': 1354486460.29461, 'code': 0, 'valid': True}} >Thread-610::DEBUG::2012-12-03 00:14:21,205::task::568::TaskManager.Task::(_updateState) Task=`abbe7c3c-d670-43bf-88d7-f6c24b67476d`::moving from state preparing -> state finished >Thread-610::DEBUG::2012-12-03 00:14:21,205::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-610::DEBUG::2012-12-03 00:14:21,205::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-610::DEBUG::2012-12-03 00:14:21,205::task::957::TaskManager.Task::(_decref) Task=`abbe7c3c-d670-43bf-88d7-f6c24b67476d`::ref 0 aborting False >Thread-616::DEBUG::2012-12-03 00:14:32,260::task::568::TaskManager.Task::(_updateState) Task=`d52e3bfc-b655-4dd7-b314-52a015b8a96c`::moving from state init -> state preparing >Thread-616::INFO::2012-12-03 00:14:32,260::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-616::INFO::2012-12-03 00:14:32,261::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0080738067627', 'lastCheck': 1354486470.3147049, 'code': 0, 'valid': True}} >Thread-616::DEBUG::2012-12-03 00:14:32,261::task::1151::TaskManager.Task::(prepare) Task=`d52e3bfc-b655-4dd7-b314-52a015b8a96c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0080738067627', 'lastCheck': 1354486470.3147049, 'code': 0, 'valid': True}} >Thread-616::DEBUG::2012-12-03 00:14:32,261::task::568::TaskManager.Task::(_updateState) Task=`d52e3bfc-b655-4dd7-b314-52a015b8a96c`::moving from state preparing -> state finished >Thread-616::DEBUG::2012-12-03 00:14:32,261::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-616::DEBUG::2012-12-03 00:14:32,261::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-616::DEBUG::2012-12-03 00:14:32,261::task::957::TaskManager.Task::(_decref) Task=`d52e3bfc-b655-4dd7-b314-52a015b8a96c`::ref 0 aborting False >Thread-622::DEBUG::2012-12-03 00:14:43,201::task::568::TaskManager.Task::(_updateState) Task=`4ca5f05e-1fcd-4359-89ae-a7959bd8adf8`::moving from state init -> state preparing >Thread-622::INFO::2012-12-03 00:14:43,202::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-622::INFO::2012-12-03 00:14:43,202::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00888681411743', 'lastCheck': 1354486480.3336301, 'code': 0, 'valid': True}} >Thread-622::DEBUG::2012-12-03 00:14:43,202::task::1151::TaskManager.Task::(prepare) Task=`4ca5f05e-1fcd-4359-89ae-a7959bd8adf8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00888681411743', 'lastCheck': 1354486480.3336301, 'code': 0, 'valid': True}} >Thread-622::DEBUG::2012-12-03 00:14:43,202::task::568::TaskManager.Task::(_updateState) Task=`4ca5f05e-1fcd-4359-89ae-a7959bd8adf8`::moving from state preparing -> state finished >Thread-622::DEBUG::2012-12-03 00:14:43,202::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-622::DEBUG::2012-12-03 00:14:43,202::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-622::DEBUG::2012-12-03 00:14:43,202::task::957::TaskManager.Task::(_decref) Task=`4ca5f05e-1fcd-4359-89ae-a7959bd8adf8`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 00:14:50,333::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:14:50,334::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:14:50,334::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:14:50,543::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:14:50,546::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:14:50,547::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:14:50,584::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:14:50,584::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:14:50,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f88715a0e40bd7b8575a75e585467485b3ce9fc4'] >Thread-276::DEBUG::2012-12-03 00:14:50,595::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:14:50,595::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:14:50,595::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-690::DEBUG::2012-12-03 00:14:54,193::task::568::TaskManager.Task::(_updateState) Task=`cd2fbf60-a34b-453c-a9bb-8f1bb4ad0390`::moving from state init -> state preparing >Thread-690::INFO::2012-12-03 00:14:54,193::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-690::INFO::2012-12-03 00:14:54,193::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00699901580811', 'lastCheck': 1354486490.6125929, 'code': 0, 'valid': True}} >Thread-690::DEBUG::2012-12-03 00:14:54,193::task::1151::TaskManager.Task::(prepare) Task=`cd2fbf60-a34b-453c-a9bb-8f1bb4ad0390`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00699901580811', 'lastCheck': 1354486490.6125929, 'code': 0, 'valid': True}} >Thread-690::DEBUG::2012-12-03 00:14:54,193::task::568::TaskManager.Task::(_updateState) Task=`cd2fbf60-a34b-453c-a9bb-8f1bb4ad0390`::moving from state preparing -> state finished >Thread-690::DEBUG::2012-12-03 00:14:54,193::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-690::DEBUG::2012-12-03 00:14:54,194::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-690::DEBUG::2012-12-03 00:14:54,194::task::957::TaskManager.Task::(_decref) Task=`cd2fbf60-a34b-453c-a9bb-8f1bb4ad0390`::ref 0 aborting False >Thread-696::DEBUG::2012-12-03 00:15:05,110::task::568::TaskManager.Task::(_updateState) Task=`4da45f53-13be-46dc-884e-591b8297b1a3`::moving from state init -> state preparing >Thread-696::INFO::2012-12-03 00:15:05,110::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-696::INFO::2012-12-03 00:15:05,110::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0090160369873', 'lastCheck': 1354486500.6314211, 'code': 0, 'valid': True}} >Thread-696::DEBUG::2012-12-03 00:15:05,110::task::1151::TaskManager.Task::(prepare) Task=`4da45f53-13be-46dc-884e-591b8297b1a3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0090160369873', 'lastCheck': 1354486500.6314211, 'code': 0, 'valid': True}} >Thread-696::DEBUG::2012-12-03 00:15:05,110::task::568::TaskManager.Task::(_updateState) Task=`4da45f53-13be-46dc-884e-591b8297b1a3`::moving from state preparing -> state finished >Thread-696::DEBUG::2012-12-03 00:15:05,110::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-696::DEBUG::2012-12-03 00:15:05,111::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-696::DEBUG::2012-12-03 00:15:05,111::task::957::TaskManager.Task::(_decref) Task=`4da45f53-13be-46dc-884e-591b8297b1a3`::ref 0 aborting False >Thread-702::DEBUG::2012-12-03 00:15:16,080::task::568::TaskManager.Task::(_updateState) Task=`c254e1bc-d09d-4fd2-9584-f46aabcfe01d`::moving from state init -> state preparing >Thread-702::INFO::2012-12-03 00:15:16,080::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-702::INFO::2012-12-03 00:15:16,080::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00357699394226', 'lastCheck': 1354486510.643661, 'code': 0, 'valid': True}} >Thread-702::DEBUG::2012-12-03 00:15:16,080::task::1151::TaskManager.Task::(prepare) Task=`c254e1bc-d09d-4fd2-9584-f46aabcfe01d`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00357699394226', 'lastCheck': 1354486510.643661, 'code': 0, 'valid': True}} >Thread-702::DEBUG::2012-12-03 00:15:16,080::task::568::TaskManager.Task::(_updateState) Task=`c254e1bc-d09d-4fd2-9584-f46aabcfe01d`::moving from state preparing -> state finished >Thread-702::DEBUG::2012-12-03 00:15:16,080::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-702::DEBUG::2012-12-03 00:15:16,080::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-702::DEBUG::2012-12-03 00:15:16,081::task::957::TaskManager.Task::(_decref) Task=`c254e1bc-d09d-4fd2-9584-f46aabcfe01d`::ref 0 aborting False >Thread-708::DEBUG::2012-12-03 00:15:26,980::task::568::TaskManager.Task::(_updateState) Task=`b96ab617-b60d-4680-a5b1-9cfd03cb6aac`::moving from state init -> state preparing >Thread-708::INFO::2012-12-03 00:15:26,981::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-708::INFO::2012-12-03 00:15:26,981::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00854396820068', 'lastCheck': 1354486520.664284, 'code': 0, 'valid': True}} >Thread-708::DEBUG::2012-12-03 00:15:26,981::task::1151::TaskManager.Task::(prepare) Task=`b96ab617-b60d-4680-a5b1-9cfd03cb6aac`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00854396820068', 'lastCheck': 1354486520.664284, 'code': 0, 'valid': True}} >Thread-708::DEBUG::2012-12-03 00:15:26,981::task::568::TaskManager.Task::(_updateState) Task=`b96ab617-b60d-4680-a5b1-9cfd03cb6aac`::moving from state preparing -> state finished >Thread-708::DEBUG::2012-12-03 00:15:26,981::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-708::DEBUG::2012-12-03 00:15:26,981::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-708::DEBUG::2012-12-03 00:15:26,982::task::957::TaskManager.Task::(_decref) Task=`b96ab617-b60d-4680-a5b1-9cfd03cb6aac`::ref 0 aborting False >Thread-714::DEBUG::2012-12-03 00:15:37,896::task::568::TaskManager.Task::(_updateState) Task=`8a7f6324-979f-4876-a260-f7b2ad2a61b6`::moving from state init -> state preparing >Thread-714::INFO::2012-12-03 00:15:37,896::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-714::INFO::2012-12-03 00:15:37,896::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00805997848511', 'lastCheck': 1354486530.6806369, 'code': 0, 'valid': True}} >Thread-714::DEBUG::2012-12-03 00:15:37,896::task::1151::TaskManager.Task::(prepare) Task=`8a7f6324-979f-4876-a260-f7b2ad2a61b6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00805997848511', 'lastCheck': 1354486530.6806369, 'code': 0, 'valid': True}} >Thread-714::DEBUG::2012-12-03 00:15:37,896::task::568::TaskManager.Task::(_updateState) Task=`8a7f6324-979f-4876-a260-f7b2ad2a61b6`::moving from state preparing -> state finished >Thread-714::DEBUG::2012-12-03 00:15:37,897::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-714::DEBUG::2012-12-03 00:15:37,897::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-714::DEBUG::2012-12-03 00:15:37,897::task::957::TaskManager.Task::(_decref) Task=`8a7f6324-979f-4876-a260-f7b2ad2a61b6`::ref 0 aborting False >Thread-720::DEBUG::2012-12-03 00:15:48,816::task::568::TaskManager.Task::(_updateState) Task=`c65a0dd4-d362-4f80-9075-81d0a498345e`::moving from state init -> state preparing >Thread-720::INFO::2012-12-03 00:15:48,816::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-720::INFO::2012-12-03 00:15:48,816::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.012806892395', 'lastCheck': 1354486540.703799, 'code': 0, 'valid': True}} >Thread-720::DEBUG::2012-12-03 00:15:48,816::task::1151::TaskManager.Task::(prepare) Task=`c65a0dd4-d362-4f80-9075-81d0a498345e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.012806892395', 'lastCheck': 1354486540.703799, 'code': 0, 'valid': True}} >Thread-720::DEBUG::2012-12-03 00:15:48,817::task::568::TaskManager.Task::(_updateState) Task=`c65a0dd4-d362-4f80-9075-81d0a498345e`::moving from state preparing -> state finished >Thread-720::DEBUG::2012-12-03 00:15:48,817::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-720::DEBUG::2012-12-03 00:15:48,817::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-720::DEBUG::2012-12-03 00:15:48,817::task::957::TaskManager.Task::(_decref) Task=`c65a0dd4-d362-4f80-9075-81d0a498345e`::ref 0 aborting False >Thread-726::DEBUG::2012-12-03 00:15:59,783::task::568::TaskManager.Task::(_updateState) Task=`b96da85e-5fc6-4870-b604-d92369152bfc`::moving from state init -> state preparing >Thread-726::INFO::2012-12-03 00:15:59,783::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-726::INFO::2012-12-03 00:15:59,783::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00810289382935', 'lastCheck': 1354486550.7217009, 'code': 0, 'valid': True}} >Thread-726::DEBUG::2012-12-03 00:15:59,783::task::1151::TaskManager.Task::(prepare) Task=`b96da85e-5fc6-4870-b604-d92369152bfc`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00810289382935', 'lastCheck': 1354486550.7217009, 'code': 0, 'valid': True}} >Thread-726::DEBUG::2012-12-03 00:15:59,784::task::568::TaskManager.Task::(_updateState) Task=`b96da85e-5fc6-4870-b604-d92369152bfc`::moving from state preparing -> state finished >Thread-726::DEBUG::2012-12-03 00:15:59,784::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-726::DEBUG::2012-12-03 00:15:59,784::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-726::DEBUG::2012-12-03 00:15:59,784::task::957::TaskManager.Task::(_decref) Task=`b96da85e-5fc6-4870-b604-d92369152bfc`::ref 0 aborting False >Thread-732::DEBUG::2012-12-03 00:16:10,679::task::568::TaskManager.Task::(_updateState) Task=`a2035d9d-b1fd-4827-a625-29c776971d7c`::moving from state init -> state preparing >Thread-732::INFO::2012-12-03 00:16:10,679::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-732::INFO::2012-12-03 00:16:10,679::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00757193565369', 'lastCheck': 1354486560.7397251, 'code': 0, 'valid': True}} >Thread-732::DEBUG::2012-12-03 00:16:10,679::task::1151::TaskManager.Task::(prepare) Task=`a2035d9d-b1fd-4827-a625-29c776971d7c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00757193565369', 'lastCheck': 1354486560.7397251, 'code': 0, 'valid': True}} >Thread-732::DEBUG::2012-12-03 00:16:10,680::task::568::TaskManager.Task::(_updateState) Task=`a2035d9d-b1fd-4827-a625-29c776971d7c`::moving from state preparing -> state finished >Thread-732::DEBUG::2012-12-03 00:16:10,680::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-732::DEBUG::2012-12-03 00:16:10,680::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-732::DEBUG::2012-12-03 00:16:10,680::task::957::TaskManager.Task::(_decref) Task=`a2035d9d-b1fd-4827-a625-29c776971d7c`::ref 0 aborting False >Thread-738::DEBUG::2012-12-03 00:16:21,619::task::568::TaskManager.Task::(_updateState) Task=`3206ff0d-ad45-4b57-8737-0e2d9d8e36c2`::moving from state init -> state preparing >Thread-738::INFO::2012-12-03 00:16:21,619::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-738::INFO::2012-12-03 00:16:21,620::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00803208351135', 'lastCheck': 1354486580.7837131, 'code': 0, 'valid': True}} >Thread-738::DEBUG::2012-12-03 00:16:21,620::task::1151::TaskManager.Task::(prepare) Task=`3206ff0d-ad45-4b57-8737-0e2d9d8e36c2`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00803208351135', 'lastCheck': 1354486580.7837131, 'code': 0, 'valid': True}} >Thread-738::DEBUG::2012-12-03 00:16:21,620::task::568::TaskManager.Task::(_updateState) Task=`3206ff0d-ad45-4b57-8737-0e2d9d8e36c2`::moving from state preparing -> state finished >Thread-738::DEBUG::2012-12-03 00:16:21,620::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-738::DEBUG::2012-12-03 00:16:21,620::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-738::DEBUG::2012-12-03 00:16:21,620::task::957::TaskManager.Task::(_decref) Task=`3206ff0d-ad45-4b57-8737-0e2d9d8e36c2`::ref 0 aborting False >Thread-744::DEBUG::2012-12-03 00:16:32,519::task::568::TaskManager.Task::(_updateState) Task=`15b61336-7dd0-40fd-98bb-40e372ce2d14`::moving from state init -> state preparing >Thread-744::INFO::2012-12-03 00:16:32,519::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-744::INFO::2012-12-03 00:16:32,519::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00718593597412', 'lastCheck': 1354486590.8010769, 'code': 0, 'valid': True}} >Thread-744::DEBUG::2012-12-03 00:16:32,519::task::1151::TaskManager.Task::(prepare) Task=`15b61336-7dd0-40fd-98bb-40e372ce2d14`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00718593597412', 'lastCheck': 1354486590.8010769, 'code': 0, 'valid': True}} >Thread-744::DEBUG::2012-12-03 00:16:32,520::task::568::TaskManager.Task::(_updateState) Task=`15b61336-7dd0-40fd-98bb-40e372ce2d14`::moving from state preparing -> state finished >Thread-744::DEBUG::2012-12-03 00:16:32,520::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-744::DEBUG::2012-12-03 00:16:32,520::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-744::DEBUG::2012-12-03 00:16:32,520::task::957::TaskManager.Task::(_decref) Task=`15b61336-7dd0-40fd-98bb-40e372ce2d14`::ref 0 aborting False >Thread-750::DEBUG::2012-12-03 00:16:43,451::task::568::TaskManager.Task::(_updateState) Task=`ad4cb62d-b4f7-4f27-89d6-fc3780587528`::moving from state init -> state preparing >Thread-750::INFO::2012-12-03 00:16:43,451::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-750::INFO::2012-12-03 00:16:43,451::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0107109546661', 'lastCheck': 1354486600.8237629, 'code': 0, 'valid': True}} >Thread-750::DEBUG::2012-12-03 00:16:43,451::task::1151::TaskManager.Task::(prepare) Task=`ad4cb62d-b4f7-4f27-89d6-fc3780587528`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0107109546661', 'lastCheck': 1354486600.8237629, 'code': 0, 'valid': True}} >Thread-750::DEBUG::2012-12-03 00:16:43,451::task::568::TaskManager.Task::(_updateState) Task=`ad4cb62d-b4f7-4f27-89d6-fc3780587528`::moving from state preparing -> state finished >Thread-750::DEBUG::2012-12-03 00:16:43,452::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-750::DEBUG::2012-12-03 00:16:43,452::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-750::DEBUG::2012-12-03 00:16:43,452::task::957::TaskManager.Task::(_decref) Task=`ad4cb62d-b4f7-4f27-89d6-fc3780587528`::ref 0 aborting False >Thread-756::DEBUG::2012-12-03 00:16:54,370::task::568::TaskManager.Task::(_updateState) Task=`d0e26419-7317-4388-a0cf-e4c3f54479bc`::moving from state init -> state preparing >Thread-756::INFO::2012-12-03 00:16:54,370::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-756::INFO::2012-12-03 00:16:54,370::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00945997238159', 'lastCheck': 1354486610.8434241, 'code': 0, 'valid': True}} >Thread-756::DEBUG::2012-12-03 00:16:54,370::task::1151::TaskManager.Task::(prepare) Task=`d0e26419-7317-4388-a0cf-e4c3f54479bc`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00945997238159', 'lastCheck': 1354486610.8434241, 'code': 0, 'valid': True}} >Thread-756::DEBUG::2012-12-03 00:16:54,370::task::568::TaskManager.Task::(_updateState) Task=`d0e26419-7317-4388-a0cf-e4c3f54479bc`::moving from state preparing -> state finished >Thread-756::DEBUG::2012-12-03 00:16:54,370::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-756::DEBUG::2012-12-03 00:16:54,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-756::DEBUG::2012-12-03 00:16:54,371::task::957::TaskManager.Task::(_decref) Task=`d0e26419-7317-4388-a0cf-e4c3f54479bc`::ref 0 aborting False >Thread-762::DEBUG::2012-12-03 00:17:05,263::task::568::TaskManager.Task::(_updateState) Task=`106b0913-c506-4aa3-a313-487f78806f53`::moving from state init -> state preparing >Thread-762::INFO::2012-12-03 00:17:05,263::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-762::INFO::2012-12-03 00:17:05,264::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00825095176697', 'lastCheck': 1354486620.8611259, 'code': 0, 'valid': True}} >Thread-762::DEBUG::2012-12-03 00:17:05,264::task::1151::TaskManager.Task::(prepare) Task=`106b0913-c506-4aa3-a313-487f78806f53`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00825095176697', 'lastCheck': 1354486620.8611259, 'code': 0, 'valid': True}} >Thread-762::DEBUG::2012-12-03 00:17:05,264::task::568::TaskManager.Task::(_updateState) Task=`106b0913-c506-4aa3-a313-487f78806f53`::moving from state preparing -> state finished >Thread-762::DEBUG::2012-12-03 00:17:05,264::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-762::DEBUG::2012-12-03 00:17:05,264::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-762::DEBUG::2012-12-03 00:17:05,264::task::957::TaskManager.Task::(_decref) Task=`106b0913-c506-4aa3-a313-487f78806f53`::ref 0 aborting False >Thread-768::DEBUG::2012-12-03 00:17:16,224::task::568::TaskManager.Task::(_updateState) Task=`7eeda3b7-7120-45d8-9c8f-4731d9298601`::moving from state init -> state preparing >Thread-768::INFO::2012-12-03 00:17:16,224::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-768::INFO::2012-12-03 00:17:16,224::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00896596908569', 'lastCheck': 1354486630.8807011, 'code': 0, 'valid': True}} >Thread-768::DEBUG::2012-12-03 00:17:16,224::task::1151::TaskManager.Task::(prepare) Task=`7eeda3b7-7120-45d8-9c8f-4731d9298601`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00896596908569', 'lastCheck': 1354486630.8807011, 'code': 0, 'valid': True}} >Thread-768::DEBUG::2012-12-03 00:17:16,225::task::568::TaskManager.Task::(_updateState) Task=`7eeda3b7-7120-45d8-9c8f-4731d9298601`::moving from state preparing -> state finished >Thread-768::DEBUG::2012-12-03 00:17:16,225::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-768::DEBUG::2012-12-03 00:17:16,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-768::DEBUG::2012-12-03 00:17:16,225::task::957::TaskManager.Task::(_decref) Task=`7eeda3b7-7120-45d8-9c8f-4731d9298601`::ref 0 aborting False >Thread-774::DEBUG::2012-12-03 00:17:27,145::task::568::TaskManager.Task::(_updateState) Task=`0e1b1ea7-c4a0-42e6-95c2-f068859da16c`::moving from state init -> state preparing >Thread-774::INFO::2012-12-03 00:17:27,145::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-774::INFO::2012-12-03 00:17:27,145::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00914883613586', 'lastCheck': 1354486640.9045291, 'code': 0, 'valid': True}} >Thread-774::DEBUG::2012-12-03 00:17:27,146::task::1151::TaskManager.Task::(prepare) Task=`0e1b1ea7-c4a0-42e6-95c2-f068859da16c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00914883613586', 'lastCheck': 1354486640.9045291, 'code': 0, 'valid': True}} >Thread-774::DEBUG::2012-12-03 00:17:27,146::task::568::TaskManager.Task::(_updateState) Task=`0e1b1ea7-c4a0-42e6-95c2-f068859da16c`::moving from state preparing -> state finished >Thread-774::DEBUG::2012-12-03 00:17:27,146::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-774::DEBUG::2012-12-03 00:17:27,146::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-774::DEBUG::2012-12-03 00:17:27,146::task::957::TaskManager.Task::(_decref) Task=`0e1b1ea7-c4a0-42e6-95c2-f068859da16c`::ref 0 aborting False >Thread-780::DEBUG::2012-12-03 00:17:38,102::task::568::TaskManager.Task::(_updateState) Task=`8befbe48-1cf8-4374-af11-38c233377c48`::moving from state init -> state preparing >Thread-780::INFO::2012-12-03 00:17:38,103::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-780::INFO::2012-12-03 00:17:38,103::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00853109359741', 'lastCheck': 1354486650.92377, 'code': 0, 'valid': True}} >Thread-780::DEBUG::2012-12-03 00:17:38,103::task::1151::TaskManager.Task::(prepare) Task=`8befbe48-1cf8-4374-af11-38c233377c48`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00853109359741', 'lastCheck': 1354486650.92377, 'code': 0, 'valid': True}} >Thread-780::DEBUG::2012-12-03 00:17:38,103::task::568::TaskManager.Task::(_updateState) Task=`8befbe48-1cf8-4374-af11-38c233377c48`::moving from state preparing -> state finished >Thread-780::DEBUG::2012-12-03 00:17:38,103::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-780::DEBUG::2012-12-03 00:17:38,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-780::DEBUG::2012-12-03 00:17:38,104::task::957::TaskManager.Task::(_decref) Task=`8befbe48-1cf8-4374-af11-38c233377c48`::ref 0 aborting False >Thread-786::DEBUG::2012-12-03 00:17:49,030::task::568::TaskManager.Task::(_updateState) Task=`ada95451-4976-488c-a093-ffe3eefe782f`::moving from state init -> state preparing >Thread-786::INFO::2012-12-03 00:17:49,030::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-786::INFO::2012-12-03 00:17:49,030::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00908017158508', 'lastCheck': 1354486660.9453449, 'code': 0, 'valid': True}} >Thread-786::DEBUG::2012-12-03 00:17:49,031::task::1151::TaskManager.Task::(prepare) Task=`ada95451-4976-488c-a093-ffe3eefe782f`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00908017158508', 'lastCheck': 1354486660.9453449, 'code': 0, 'valid': True}} >Thread-786::DEBUG::2012-12-03 00:17:49,031::task::568::TaskManager.Task::(_updateState) Task=`ada95451-4976-488c-a093-ffe3eefe782f`::moving from state preparing -> state finished >Thread-786::DEBUG::2012-12-03 00:17:49,031::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-786::DEBUG::2012-12-03 00:17:49,031::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-786::DEBUG::2012-12-03 00:17:49,031::task::957::TaskManager.Task::(_decref) Task=`ada95451-4976-488c-a093-ffe3eefe782f`::ref 0 aborting False >Thread-792::DEBUG::2012-12-03 00:17:59,953::task::568::TaskManager.Task::(_updateState) Task=`e99a25a8-6cae-4a14-962c-b079713b3d45`::moving from state init -> state preparing >Thread-792::INFO::2012-12-03 00:17:59,953::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-792::INFO::2012-12-03 00:17:59,953::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.014190196991', 'lastCheck': 1354486670.973295, 'code': 0, 'valid': True}} >Thread-792::DEBUG::2012-12-03 00:17:59,953::task::1151::TaskManager.Task::(prepare) Task=`e99a25a8-6cae-4a14-962c-b079713b3d45`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.014190196991', 'lastCheck': 1354486670.973295, 'code': 0, 'valid': True}} >Thread-792::DEBUG::2012-12-03 00:17:59,953::task::568::TaskManager.Task::(_updateState) Task=`e99a25a8-6cae-4a14-962c-b079713b3d45`::moving from state preparing -> state finished >Thread-792::DEBUG::2012-12-03 00:17:59,954::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-792::DEBUG::2012-12-03 00:17:59,954::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-792::DEBUG::2012-12-03 00:17:59,954::task::957::TaskManager.Task::(_decref) Task=`e99a25a8-6cae-4a14-962c-b079713b3d45`::ref 0 aborting False >Thread-798::DEBUG::2012-12-03 00:18:10,884::task::568::TaskManager.Task::(_updateState) Task=`c1bbcd90-a511-44e0-a34d-1ff37137a7df`::moving from state init -> state preparing >Thread-798::INFO::2012-12-03 00:18:10,884::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-798::INFO::2012-12-03 00:18:10,884::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00347399711609', 'lastCheck': 1354486680.984772, 'code': 0, 'valid': True}} >Thread-798::DEBUG::2012-12-03 00:18:10,885::task::1151::TaskManager.Task::(prepare) Task=`c1bbcd90-a511-44e0-a34d-1ff37137a7df`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00347399711609', 'lastCheck': 1354486680.984772, 'code': 0, 'valid': True}} >Thread-798::DEBUG::2012-12-03 00:18:10,885::task::568::TaskManager.Task::(_updateState) Task=`c1bbcd90-a511-44e0-a34d-1ff37137a7df`::moving from state preparing -> state finished >Thread-798::DEBUG::2012-12-03 00:18:10,885::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-798::DEBUG::2012-12-03 00:18:10,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-798::DEBUG::2012-12-03 00:18:10,885::task::957::TaskManager.Task::(_decref) Task=`c1bbcd90-a511-44e0-a34d-1ff37137a7df`::ref 0 aborting False >Thread-804::DEBUG::2012-12-03 00:18:21,835::task::568::TaskManager.Task::(_updateState) Task=`23b072af-bc83-446e-9f5e-82bbc192db34`::moving from state init -> state preparing >Thread-804::INFO::2012-12-03 00:18:21,836::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-804::INFO::2012-12-03 00:18:21,836::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00900912284851', 'lastCheck': 1354486701.030323, 'code': 0, 'valid': True}} >Thread-804::DEBUG::2012-12-03 00:18:21,836::task::1151::TaskManager.Task::(prepare) Task=`23b072af-bc83-446e-9f5e-82bbc192db34`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00900912284851', 'lastCheck': 1354486701.030323, 'code': 0, 'valid': True}} >Thread-804::DEBUG::2012-12-03 00:18:21,836::task::568::TaskManager.Task::(_updateState) Task=`23b072af-bc83-446e-9f5e-82bbc192db34`::moving from state preparing -> state finished >Thread-804::DEBUG::2012-12-03 00:18:21,836::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-804::DEBUG::2012-12-03 00:18:21,836::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-804::DEBUG::2012-12-03 00:18:21,836::task::957::TaskManager.Task::(_decref) Task=`23b072af-bc83-446e-9f5e-82bbc192db34`::ref 0 aborting False >Thread-810::DEBUG::2012-12-03 00:18:32,803::task::568::TaskManager.Task::(_updateState) Task=`ef6612c1-effb-4155-8952-6766f80539d7`::moving from state init -> state preparing >Thread-810::INFO::2012-12-03 00:18:32,803::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-810::INFO::2012-12-03 00:18:32,803::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0088050365448', 'lastCheck': 1354486711.049011, 'code': 0, 'valid': True}} >Thread-810::DEBUG::2012-12-03 00:18:32,803::task::1151::TaskManager.Task::(prepare) Task=`ef6612c1-effb-4155-8952-6766f80539d7`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0088050365448', 'lastCheck': 1354486711.049011, 'code': 0, 'valid': True}} >Thread-810::DEBUG::2012-12-03 00:18:32,804::task::568::TaskManager.Task::(_updateState) Task=`ef6612c1-effb-4155-8952-6766f80539d7`::moving from state preparing -> state finished >Thread-810::DEBUG::2012-12-03 00:18:32,804::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-810::DEBUG::2012-12-03 00:18:32,804::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-810::DEBUG::2012-12-03 00:18:32,804::task::957::TaskManager.Task::(_decref) Task=`ef6612c1-effb-4155-8952-6766f80539d7`::ref 0 aborting False >Thread-816::DEBUG::2012-12-03 00:18:43,729::task::568::TaskManager.Task::(_updateState) Task=`e8c2a3f8-e755-4e63-9d9e-bf359b67518a`::moving from state init -> state preparing >Thread-816::INFO::2012-12-03 00:18:43,729::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-816::INFO::2012-12-03 00:18:43,729::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00968790054321', 'lastCheck': 1354486721.0708759, 'code': 0, 'valid': True}} >Thread-816::DEBUG::2012-12-03 00:18:43,729::task::1151::TaskManager.Task::(prepare) Task=`e8c2a3f8-e755-4e63-9d9e-bf359b67518a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00968790054321', 'lastCheck': 1354486721.0708759, 'code': 0, 'valid': True}} >Thread-816::DEBUG::2012-12-03 00:18:43,729::task::568::TaskManager.Task::(_updateState) Task=`e8c2a3f8-e755-4e63-9d9e-bf359b67518a`::moving from state preparing -> state finished >Thread-816::DEBUG::2012-12-03 00:18:43,729::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-816::DEBUG::2012-12-03 00:18:43,730::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-816::DEBUG::2012-12-03 00:18:43,730::task::957::TaskManager.Task::(_decref) Task=`e8c2a3f8-e755-4e63-9d9e-bf359b67518a`::ref 0 aborting False >Thread-822::DEBUG::2012-12-03 00:18:54,690::task::568::TaskManager.Task::(_updateState) Task=`6a17d667-9ec7-47c9-bf44-6e5a820f2c75`::moving from state init -> state preparing >Thread-822::INFO::2012-12-03 00:18:54,690::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-822::INFO::2012-12-03 00:18:54,690::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00839591026306', 'lastCheck': 1354486731.0925419, 'code': 0, 'valid': True}} >Thread-822::DEBUG::2012-12-03 00:18:54,690::task::1151::TaskManager.Task::(prepare) Task=`6a17d667-9ec7-47c9-bf44-6e5a820f2c75`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00839591026306', 'lastCheck': 1354486731.0925419, 'code': 0, 'valid': True}} >Thread-822::DEBUG::2012-12-03 00:18:54,691::task::568::TaskManager.Task::(_updateState) Task=`6a17d667-9ec7-47c9-bf44-6e5a820f2c75`::moving from state preparing -> state finished >Thread-822::DEBUG::2012-12-03 00:18:54,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-822::DEBUG::2012-12-03 00:18:54,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-822::DEBUG::2012-12-03 00:18:54,691::task::957::TaskManager.Task::(_decref) Task=`6a17d667-9ec7-47c9-bf44-6e5a820f2c75`::ref 0 aborting False >Thread-828::DEBUG::2012-12-03 00:19:05,594::task::568::TaskManager.Task::(_updateState) Task=`85514171-90f5-4d1d-bee1-ea2d33d37bf5`::moving from state init -> state preparing >Thread-828::INFO::2012-12-03 00:19:05,594::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-828::INFO::2012-12-03 00:19:05,594::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00349903106689', 'lastCheck': 1354486741.1046121, 'code': 0, 'valid': True}} >Thread-828::DEBUG::2012-12-03 00:19:05,595::task::1151::TaskManager.Task::(prepare) Task=`85514171-90f5-4d1d-bee1-ea2d33d37bf5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00349903106689', 'lastCheck': 1354486741.1046121, 'code': 0, 'valid': True}} >Thread-828::DEBUG::2012-12-03 00:19:05,595::task::568::TaskManager.Task::(_updateState) Task=`85514171-90f5-4d1d-bee1-ea2d33d37bf5`::moving from state preparing -> state finished >Thread-828::DEBUG::2012-12-03 00:19:05,595::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-828::DEBUG::2012-12-03 00:19:05,595::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-828::DEBUG::2012-12-03 00:19:05,595::task::957::TaskManager.Task::(_decref) Task=`85514171-90f5-4d1d-bee1-ea2d33d37bf5`::ref 0 aborting False >Thread-834::DEBUG::2012-12-03 00:19:16,513::task::568::TaskManager.Task::(_updateState) Task=`1e0d4485-62df-4e9c-a445-a61eba4ee298`::moving from state init -> state preparing >Thread-834::INFO::2012-12-03 00:19:16,513::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-834::INFO::2012-12-03 00:19:16,514::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00930094718933', 'lastCheck': 1354486751.1247129, 'code': 0, 'valid': True}} >Thread-834::DEBUG::2012-12-03 00:19:16,514::task::1151::TaskManager.Task::(prepare) Task=`1e0d4485-62df-4e9c-a445-a61eba4ee298`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00930094718933', 'lastCheck': 1354486751.1247129, 'code': 0, 'valid': True}} >Thread-834::DEBUG::2012-12-03 00:19:16,514::task::568::TaskManager.Task::(_updateState) Task=`1e0d4485-62df-4e9c-a445-a61eba4ee298`::moving from state preparing -> state finished >Thread-834::DEBUG::2012-12-03 00:19:16,514::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-834::DEBUG::2012-12-03 00:19:16,514::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-834::DEBUG::2012-12-03 00:19:16,514::task::957::TaskManager.Task::(_decref) Task=`1e0d4485-62df-4e9c-a445-a61eba4ee298`::ref 0 aborting False >Thread-840::DEBUG::2012-12-03 00:19:27,423::task::568::TaskManager.Task::(_updateState) Task=`5431e3cf-4f17-4a8e-ba71-f487f2a7b5ab`::moving from state init -> state preparing >Thread-840::INFO::2012-12-03 00:19:27,423::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-840::INFO::2012-12-03 00:19:27,423::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00999712944031', 'lastCheck': 1354486761.1496191, 'code': 0, 'valid': True}} >Thread-840::DEBUG::2012-12-03 00:19:27,423::task::1151::TaskManager.Task::(prepare) Task=`5431e3cf-4f17-4a8e-ba71-f487f2a7b5ab`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00999712944031', 'lastCheck': 1354486761.1496191, 'code': 0, 'valid': True}} >Thread-840::DEBUG::2012-12-03 00:19:27,424::task::568::TaskManager.Task::(_updateState) Task=`5431e3cf-4f17-4a8e-ba71-f487f2a7b5ab`::moving from state preparing -> state finished >Thread-840::DEBUG::2012-12-03 00:19:27,424::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-840::DEBUG::2012-12-03 00:19:27,424::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-840::DEBUG::2012-12-03 00:19:27,424::task::957::TaskManager.Task::(_decref) Task=`5431e3cf-4f17-4a8e-ba71-f487f2a7b5ab`::ref 0 aborting False >Thread-846::DEBUG::2012-12-03 00:19:38,352::task::568::TaskManager.Task::(_updateState) Task=`14a3517a-839d-4c32-8ab8-5f27aa499d06`::moving from state init -> state preparing >Thread-846::INFO::2012-12-03 00:19:38,352::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-846::INFO::2012-12-03 00:19:38,352::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0469708442688', 'lastCheck': 1354486771.220077, 'code': 0, 'valid': True}} >Thread-846::DEBUG::2012-12-03 00:19:38,352::task::1151::TaskManager.Task::(prepare) Task=`14a3517a-839d-4c32-8ab8-5f27aa499d06`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0469708442688', 'lastCheck': 1354486771.220077, 'code': 0, 'valid': True}} >Thread-846::DEBUG::2012-12-03 00:19:38,352::task::568::TaskManager.Task::(_updateState) Task=`14a3517a-839d-4c32-8ab8-5f27aa499d06`::moving from state preparing -> state finished >Thread-846::DEBUG::2012-12-03 00:19:38,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-846::DEBUG::2012-12-03 00:19:38,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-846::DEBUG::2012-12-03 00:19:38,353::task::957::TaskManager.Task::(_decref) Task=`14a3517a-839d-4c32-8ab8-5f27aa499d06`::ref 0 aborting False >Thread-852::DEBUG::2012-12-03 00:19:49,313::task::568::TaskManager.Task::(_updateState) Task=`31a1d8b9-ec1c-4ae6-931f-c05834426fbf`::moving from state init -> state preparing >Thread-852::INFO::2012-12-03 00:19:49,314::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-852::INFO::2012-12-03 00:19:49,314::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00794100761414', 'lastCheck': 1354486781.2386179, 'code': 0, 'valid': True}} >Thread-852::DEBUG::2012-12-03 00:19:49,314::task::1151::TaskManager.Task::(prepare) Task=`31a1d8b9-ec1c-4ae6-931f-c05834426fbf`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00794100761414', 'lastCheck': 1354486781.2386179, 'code': 0, 'valid': True}} >Thread-852::DEBUG::2012-12-03 00:19:49,314::task::568::TaskManager.Task::(_updateState) Task=`31a1d8b9-ec1c-4ae6-931f-c05834426fbf`::moving from state preparing -> state finished >Thread-852::DEBUG::2012-12-03 00:19:49,314::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-852::DEBUG::2012-12-03 00:19:49,314::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-852::DEBUG::2012-12-03 00:19:49,315::task::957::TaskManager.Task::(_decref) Task=`31a1d8b9-ec1c-4ae6-931f-c05834426fbf`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 00:19:51,238::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:19:51,239::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:19:51,239::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:19:51,452::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:19:51,454::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:19:51,454::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:19:51,491::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:19:51,491::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:19:51,499::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f88715a0e40bd7b8575a75e585467485b3ce9fc4'] >Thread-276::DEBUG::2012-12-03 00:19:51,502::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:19:51,502::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:19:51,502::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-920::DEBUG::2012-12-03 00:20:00,238::task::568::TaskManager.Task::(_updateState) Task=`d934da0b-19d8-4593-b51e-0837923b95c7`::moving from state init -> state preparing >Thread-920::INFO::2012-12-03 00:20:00,238::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-920::INFO::2012-12-03 00:20:00,238::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00601696968079', 'lastCheck': 1354486791.5186629, 'code': 0, 'valid': True}} >Thread-920::DEBUG::2012-12-03 00:20:00,238::task::1151::TaskManager.Task::(prepare) Task=`d934da0b-19d8-4593-b51e-0837923b95c7`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00601696968079', 'lastCheck': 1354486791.5186629, 'code': 0, 'valid': True}} >Thread-920::DEBUG::2012-12-03 00:20:00,238::task::568::TaskManager.Task::(_updateState) Task=`d934da0b-19d8-4593-b51e-0837923b95c7`::moving from state preparing -> state finished >Thread-920::DEBUG::2012-12-03 00:20:00,238::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-920::DEBUG::2012-12-03 00:20:00,239::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-920::DEBUG::2012-12-03 00:20:00,239::task::957::TaskManager.Task::(_decref) Task=`d934da0b-19d8-4593-b51e-0837923b95c7`::ref 0 aborting False >Thread-926::DEBUG::2012-12-03 00:20:11,194::task::568::TaskManager.Task::(_updateState) Task=`e8622801-750d-4ea9-a8a8-6155d1b2cb7d`::moving from state init -> state preparing >Thread-926::INFO::2012-12-03 00:20:11,194::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-926::INFO::2012-12-03 00:20:11,194::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00820398330688', 'lastCheck': 1354486801.536988, 'code': 0, 'valid': True}} >Thread-926::DEBUG::2012-12-03 00:20:11,194::task::1151::TaskManager.Task::(prepare) Task=`e8622801-750d-4ea9-a8a8-6155d1b2cb7d`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00820398330688', 'lastCheck': 1354486801.536988, 'code': 0, 'valid': True}} >Thread-926::DEBUG::2012-12-03 00:20:11,195::task::568::TaskManager.Task::(_updateState) Task=`e8622801-750d-4ea9-a8a8-6155d1b2cb7d`::moving from state preparing -> state finished >Thread-926::DEBUG::2012-12-03 00:20:11,195::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-926::DEBUG::2012-12-03 00:20:11,195::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-926::DEBUG::2012-12-03 00:20:11,195::task::957::TaskManager.Task::(_decref) Task=`e8622801-750d-4ea9-a8a8-6155d1b2cb7d`::ref 0 aborting False >Thread-932::DEBUG::2012-12-03 00:20:22,224::task::568::TaskManager.Task::(_updateState) Task=`5537ddc7-53c2-40d3-a37d-4799a79b89f8`::moving from state init -> state preparing >Thread-932::INFO::2012-12-03 00:20:22,224::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-932::INFO::2012-12-03 00:20:22,224::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00827193260193', 'lastCheck': 1354486821.577651, 'code': 0, 'valid': True}} >Thread-932::DEBUG::2012-12-03 00:20:22,224::task::1151::TaskManager.Task::(prepare) Task=`5537ddc7-53c2-40d3-a37d-4799a79b89f8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00827193260193', 'lastCheck': 1354486821.577651, 'code': 0, 'valid': True}} >Thread-932::DEBUG::2012-12-03 00:20:22,224::task::568::TaskManager.Task::(_updateState) Task=`5537ddc7-53c2-40d3-a37d-4799a79b89f8`::moving from state preparing -> state finished >Thread-932::DEBUG::2012-12-03 00:20:22,224::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-932::DEBUG::2012-12-03 00:20:22,225::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-932::DEBUG::2012-12-03 00:20:22,225::task::957::TaskManager.Task::(_decref) Task=`5537ddc7-53c2-40d3-a37d-4799a79b89f8`::ref 0 aborting False >Thread-938::DEBUG::2012-12-03 00:20:33,148::task::568::TaskManager.Task::(_updateState) Task=`5945af65-0839-4544-9f82-fc0bb2b7ae8f`::moving from state init -> state preparing >Thread-938::INFO::2012-12-03 00:20:33,149::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-938::INFO::2012-12-03 00:20:33,149::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0033860206604', 'lastCheck': 1354486831.590461, 'code': 0, 'valid': True}} >Thread-938::DEBUG::2012-12-03 00:20:33,149::task::1151::TaskManager.Task::(prepare) Task=`5945af65-0839-4544-9f82-fc0bb2b7ae8f`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0033860206604', 'lastCheck': 1354486831.590461, 'code': 0, 'valid': True}} >Thread-938::DEBUG::2012-12-03 00:20:33,149::task::568::TaskManager.Task::(_updateState) Task=`5945af65-0839-4544-9f82-fc0bb2b7ae8f`::moving from state preparing -> state finished >Thread-938::DEBUG::2012-12-03 00:20:33,149::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-938::DEBUG::2012-12-03 00:20:33,149::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-938::DEBUG::2012-12-03 00:20:33,150::task::957::TaskManager.Task::(_decref) Task=`5945af65-0839-4544-9f82-fc0bb2b7ae8f`::ref 0 aborting False >Thread-944::DEBUG::2012-12-03 00:20:44,114::task::568::TaskManager.Task::(_updateState) Task=`e9b79525-0edb-4b24-a668-503f196f1974`::moving from state init -> state preparing >Thread-944::INFO::2012-12-03 00:20:44,114::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-944::INFO::2012-12-03 00:20:44,114::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0077121257782', 'lastCheck': 1354486841.608094, 'code': 0, 'valid': True}} >Thread-944::DEBUG::2012-12-03 00:20:44,115::task::1151::TaskManager.Task::(prepare) Task=`e9b79525-0edb-4b24-a668-503f196f1974`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0077121257782', 'lastCheck': 1354486841.608094, 'code': 0, 'valid': True}} >Thread-944::DEBUG::2012-12-03 00:20:44,115::task::568::TaskManager.Task::(_updateState) Task=`e9b79525-0edb-4b24-a668-503f196f1974`::moving from state preparing -> state finished >Thread-944::DEBUG::2012-12-03 00:20:44,115::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-944::DEBUG::2012-12-03 00:20:44,115::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-944::DEBUG::2012-12-03 00:20:44,115::task::957::TaskManager.Task::(_decref) Task=`e9b79525-0edb-4b24-a668-503f196f1974`::ref 0 aborting False >Thread-950::DEBUG::2012-12-03 00:20:55,031::task::568::TaskManager.Task::(_updateState) Task=`2bddad24-c24c-49ac-8f24-7ca66c78afe3`::moving from state init -> state preparing >Thread-950::INFO::2012-12-03 00:20:55,031::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-950::INFO::2012-12-03 00:20:55,031::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00687098503113', 'lastCheck': 1354486851.6269679, 'code': 0, 'valid': True}} >Thread-950::DEBUG::2012-12-03 00:20:55,031::task::1151::TaskManager.Task::(prepare) Task=`2bddad24-c24c-49ac-8f24-7ca66c78afe3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00687098503113', 'lastCheck': 1354486851.6269679, 'code': 0, 'valid': True}} >Thread-950::DEBUG::2012-12-03 00:20:55,031::task::568::TaskManager.Task::(_updateState) Task=`2bddad24-c24c-49ac-8f24-7ca66c78afe3`::moving from state preparing -> state finished >Thread-950::DEBUG::2012-12-03 00:20:55,031::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-950::DEBUG::2012-12-03 00:20:55,031::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-950::DEBUG::2012-12-03 00:20:55,032::task::957::TaskManager.Task::(_decref) Task=`2bddad24-c24c-49ac-8f24-7ca66c78afe3`::ref 0 aborting False >Thread-956::DEBUG::2012-12-03 00:21:05,985::task::568::TaskManager.Task::(_updateState) Task=`f182e9d2-7956-42c8-bde6-ebd7c2d1984c`::moving from state init -> state preparing >Thread-956::INFO::2012-12-03 00:21:05,985::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-956::INFO::2012-12-03 00:21:05,986::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00392699241638', 'lastCheck': 1354486861.6473119, 'code': 0, 'valid': True}} >Thread-956::DEBUG::2012-12-03 00:21:05,986::task::1151::TaskManager.Task::(prepare) Task=`f182e9d2-7956-42c8-bde6-ebd7c2d1984c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00392699241638', 'lastCheck': 1354486861.6473119, 'code': 0, 'valid': True}} >Thread-956::DEBUG::2012-12-03 00:21:05,986::task::568::TaskManager.Task::(_updateState) Task=`f182e9d2-7956-42c8-bde6-ebd7c2d1984c`::moving from state preparing -> state finished >Thread-956::DEBUG::2012-12-03 00:21:05,986::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-956::DEBUG::2012-12-03 00:21:05,986::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-956::DEBUG::2012-12-03 00:21:05,986::task::957::TaskManager.Task::(_decref) Task=`f182e9d2-7956-42c8-bde6-ebd7c2d1984c`::ref 0 aborting False >Thread-962::DEBUG::2012-12-03 00:21:16,985::task::568::TaskManager.Task::(_updateState) Task=`2346bfe9-72ba-4cd3-bae8-a1b66a87da3a`::moving from state init -> state preparing >Thread-962::INFO::2012-12-03 00:21:16,985::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-962::INFO::2012-12-03 00:21:16,985::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00636100769043', 'lastCheck': 1354486871.6637909, 'code': 0, 'valid': True}} >Thread-962::DEBUG::2012-12-03 00:21:16,985::task::1151::TaskManager.Task::(prepare) Task=`2346bfe9-72ba-4cd3-bae8-a1b66a87da3a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00636100769043', 'lastCheck': 1354486871.6637909, 'code': 0, 'valid': True}} >Thread-962::DEBUG::2012-12-03 00:21:16,986::task::568::TaskManager.Task::(_updateState) Task=`2346bfe9-72ba-4cd3-bae8-a1b66a87da3a`::moving from state preparing -> state finished >Thread-962::DEBUG::2012-12-03 00:21:16,986::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-962::DEBUG::2012-12-03 00:21:16,986::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-962::DEBUG::2012-12-03 00:21:16,986::task::957::TaskManager.Task::(_decref) Task=`2346bfe9-72ba-4cd3-bae8-a1b66a87da3a`::ref 0 aborting False >Thread-968::DEBUG::2012-12-03 00:21:27,927::task::568::TaskManager.Task::(_updateState) Task=`132ce659-f36d-4828-b9d0-4470e037f93d`::moving from state init -> state preparing >Thread-968::INFO::2012-12-03 00:21:27,927::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-968::INFO::2012-12-03 00:21:27,927::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00761294364929', 'lastCheck': 1354486881.683589, 'code': 0, 'valid': True}} >Thread-968::DEBUG::2012-12-03 00:21:27,927::task::1151::TaskManager.Task::(prepare) Task=`132ce659-f36d-4828-b9d0-4470e037f93d`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00761294364929', 'lastCheck': 1354486881.683589, 'code': 0, 'valid': True}} >Thread-968::DEBUG::2012-12-03 00:21:27,927::task::568::TaskManager.Task::(_updateState) Task=`132ce659-f36d-4828-b9d0-4470e037f93d`::moving from state preparing -> state finished >Thread-968::DEBUG::2012-12-03 00:21:27,927::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-968::DEBUG::2012-12-03 00:21:27,928::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-968::DEBUG::2012-12-03 00:21:27,928::task::957::TaskManager.Task::(_decref) Task=`132ce659-f36d-4828-b9d0-4470e037f93d`::ref 0 aborting False >Thread-974::DEBUG::2012-12-03 00:21:38,842::task::568::TaskManager.Task::(_updateState) Task=`faa5682b-fc6b-4627-adc0-bb46639af62e`::moving from state init -> state preparing >Thread-974::INFO::2012-12-03 00:21:38,842::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-974::INFO::2012-12-03 00:21:38,842::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.006756067276', 'lastCheck': 1354486891.698843, 'code': 0, 'valid': True}} >Thread-974::DEBUG::2012-12-03 00:21:38,842::task::1151::TaskManager.Task::(prepare) Task=`faa5682b-fc6b-4627-adc0-bb46639af62e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.006756067276', 'lastCheck': 1354486891.698843, 'code': 0, 'valid': True}} >Thread-974::DEBUG::2012-12-03 00:21:38,843::task::568::TaskManager.Task::(_updateState) Task=`faa5682b-fc6b-4627-adc0-bb46639af62e`::moving from state preparing -> state finished >Thread-974::DEBUG::2012-12-03 00:21:38,843::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-974::DEBUG::2012-12-03 00:21:38,843::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-974::DEBUG::2012-12-03 00:21:38,843::task::957::TaskManager.Task::(_decref) Task=`faa5682b-fc6b-4627-adc0-bb46639af62e`::ref 0 aborting False >Thread-980::DEBUG::2012-12-03 00:21:49,902::task::568::TaskManager.Task::(_updateState) Task=`047b5f3f-48ce-4f29-b4e0-feaeff18d979`::moving from state init -> state preparing >Thread-980::INFO::2012-12-03 00:21:49,902::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-980::INFO::2012-12-03 00:21:49,902::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00707101821899', 'lastCheck': 1354486901.7160161, 'code': 0, 'valid': True}} >Thread-980::DEBUG::2012-12-03 00:21:49,902::task::1151::TaskManager.Task::(prepare) Task=`047b5f3f-48ce-4f29-b4e0-feaeff18d979`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00707101821899', 'lastCheck': 1354486901.7160161, 'code': 0, 'valid': True}} >Thread-980::DEBUG::2012-12-03 00:21:49,902::task::568::TaskManager.Task::(_updateState) Task=`047b5f3f-48ce-4f29-b4e0-feaeff18d979`::moving from state preparing -> state finished >Thread-980::DEBUG::2012-12-03 00:21:49,903::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-980::DEBUG::2012-12-03 00:21:49,903::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-980::DEBUG::2012-12-03 00:21:49,903::task::957::TaskManager.Task::(_decref) Task=`047b5f3f-48ce-4f29-b4e0-feaeff18d979`::ref 0 aborting False >Thread-986::DEBUG::2012-12-03 00:22:00,821::task::568::TaskManager.Task::(_updateState) Task=`5ccb3b55-7175-4159-b151-fd0e84253f98`::moving from state init -> state preparing >Thread-986::INFO::2012-12-03 00:22:00,821::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-986::INFO::2012-12-03 00:22:00,821::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00920510292053', 'lastCheck': 1354486911.736644, 'code': 0, 'valid': True}} >Thread-986::DEBUG::2012-12-03 00:22:00,821::task::1151::TaskManager.Task::(prepare) Task=`5ccb3b55-7175-4159-b151-fd0e84253f98`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00920510292053', 'lastCheck': 1354486911.736644, 'code': 0, 'valid': True}} >Thread-986::DEBUG::2012-12-03 00:22:00,821::task::568::TaskManager.Task::(_updateState) Task=`5ccb3b55-7175-4159-b151-fd0e84253f98`::moving from state preparing -> state finished >Thread-986::DEBUG::2012-12-03 00:22:00,821::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-986::DEBUG::2012-12-03 00:22:00,821::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-986::DEBUG::2012-12-03 00:22:00,822::task::957::TaskManager.Task::(_decref) Task=`5ccb3b55-7175-4159-b151-fd0e84253f98`::ref 0 aborting False >Thread-992::DEBUG::2012-12-03 00:22:11,766::task::568::TaskManager.Task::(_updateState) Task=`34f66965-d91e-47a6-9b2a-77d67730a16b`::moving from state init -> state preparing >Thread-992::INFO::2012-12-03 00:22:11,766::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-992::INFO::2012-12-03 00:22:11,767::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00821709632874', 'lastCheck': 1354486921.7548959, 'code': 0, 'valid': True}} >Thread-992::DEBUG::2012-12-03 00:22:11,767::task::1151::TaskManager.Task::(prepare) Task=`34f66965-d91e-47a6-9b2a-77d67730a16b`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00821709632874', 'lastCheck': 1354486921.7548959, 'code': 0, 'valid': True}} >Thread-992::DEBUG::2012-12-03 00:22:11,767::task::568::TaskManager.Task::(_updateState) Task=`34f66965-d91e-47a6-9b2a-77d67730a16b`::moving from state preparing -> state finished >Thread-992::DEBUG::2012-12-03 00:22:11,767::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-992::DEBUG::2012-12-03 00:22:11,767::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-992::DEBUG::2012-12-03 00:22:11,767::task::957::TaskManager.Task::(_decref) Task=`34f66965-d91e-47a6-9b2a-77d67730a16b`::ref 0 aborting False >Thread-998::DEBUG::2012-12-03 00:22:22,706::task::568::TaskManager.Task::(_updateState) Task=`a7d02a74-1ca0-40ac-b843-94585c82f867`::moving from state init -> state preparing >Thread-998::INFO::2012-12-03 00:22:22,706::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-998::INFO::2012-12-03 00:22:22,707::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00858497619629', 'lastCheck': 1354486941.795527, 'code': 0, 'valid': True}} >Thread-998::DEBUG::2012-12-03 00:22:22,707::task::1151::TaskManager.Task::(prepare) Task=`a7d02a74-1ca0-40ac-b843-94585c82f867`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00858497619629', 'lastCheck': 1354486941.795527, 'code': 0, 'valid': True}} >Thread-998::DEBUG::2012-12-03 00:22:22,707::task::568::TaskManager.Task::(_updateState) Task=`a7d02a74-1ca0-40ac-b843-94585c82f867`::moving from state preparing -> state finished >Thread-998::DEBUG::2012-12-03 00:22:22,707::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-998::DEBUG::2012-12-03 00:22:22,707::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-998::DEBUG::2012-12-03 00:22:22,707::task::957::TaskManager.Task::(_decref) Task=`a7d02a74-1ca0-40ac-b843-94585c82f867`::ref 0 aborting False >Thread-1004::DEBUG::2012-12-03 00:22:33,617::task::568::TaskManager.Task::(_updateState) Task=`e52bab5a-d09c-4cba-90c1-f7248b6c9b0c`::moving from state init -> state preparing >Thread-1004::INFO::2012-12-03 00:22:33,617::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1004::INFO::2012-12-03 00:22:33,618::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0082049369812', 'lastCheck': 1354486951.814405, 'code': 0, 'valid': True}} >Thread-1004::DEBUG::2012-12-03 00:22:33,618::task::1151::TaskManager.Task::(prepare) Task=`e52bab5a-d09c-4cba-90c1-f7248b6c9b0c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0082049369812', 'lastCheck': 1354486951.814405, 'code': 0, 'valid': True}} >Thread-1004::DEBUG::2012-12-03 00:22:33,618::task::568::TaskManager.Task::(_updateState) Task=`e52bab5a-d09c-4cba-90c1-f7248b6c9b0c`::moving from state preparing -> state finished >Thread-1004::DEBUG::2012-12-03 00:22:33,618::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1004::DEBUG::2012-12-03 00:22:33,618::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1004::DEBUG::2012-12-03 00:22:33,618::task::957::TaskManager.Task::(_decref) Task=`e52bab5a-d09c-4cba-90c1-f7248b6c9b0c`::ref 0 aborting False >Thread-1010::DEBUG::2012-12-03 00:22:44,528::task::568::TaskManager.Task::(_updateState) Task=`9640b210-7f97-49ba-be4e-8f742bea4656`::moving from state init -> state preparing >Thread-1010::INFO::2012-12-03 00:22:44,528::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1010::INFO::2012-12-03 00:22:44,529::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00933480262756', 'lastCheck': 1354486961.8325529, 'code': 0, 'valid': True}} >Thread-1010::DEBUG::2012-12-03 00:22:44,529::task::1151::TaskManager.Task::(prepare) Task=`9640b210-7f97-49ba-be4e-8f742bea4656`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00933480262756', 'lastCheck': 1354486961.8325529, 'code': 0, 'valid': True}} >Thread-1010::DEBUG::2012-12-03 00:22:44,529::task::568::TaskManager.Task::(_updateState) Task=`9640b210-7f97-49ba-be4e-8f742bea4656`::moving from state preparing -> state finished >Thread-1010::DEBUG::2012-12-03 00:22:44,529::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1010::DEBUG::2012-12-03 00:22:44,529::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1010::DEBUG::2012-12-03 00:22:44,529::task::957::TaskManager.Task::(_decref) Task=`9640b210-7f97-49ba-be4e-8f742bea4656`::ref 0 aborting False >Thread-1016::DEBUG::2012-12-03 00:22:55,457::task::568::TaskManager.Task::(_updateState) Task=`2c11d887-3a3e-4106-a9d2-1cf87604d5a1`::moving from state init -> state preparing >Thread-1016::INFO::2012-12-03 00:22:55,457::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1016::INFO::2012-12-03 00:22:55,457::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00933694839478', 'lastCheck': 1354486971.85218, 'code': 0, 'valid': True}} >Thread-1016::DEBUG::2012-12-03 00:22:55,457::task::1151::TaskManager.Task::(prepare) Task=`2c11d887-3a3e-4106-a9d2-1cf87604d5a1`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00933694839478', 'lastCheck': 1354486971.85218, 'code': 0, 'valid': True}} >Thread-1016::DEBUG::2012-12-03 00:22:55,457::task::568::TaskManager.Task::(_updateState) Task=`2c11d887-3a3e-4106-a9d2-1cf87604d5a1`::moving from state preparing -> state finished >Thread-1016::DEBUG::2012-12-03 00:22:55,457::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1016::DEBUG::2012-12-03 00:22:55,458::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1016::DEBUG::2012-12-03 00:22:55,458::task::957::TaskManager.Task::(_decref) Task=`2c11d887-3a3e-4106-a9d2-1cf87604d5a1`::ref 0 aborting False >Thread-1022::DEBUG::2012-12-03 00:23:06,349::task::568::TaskManager.Task::(_updateState) Task=`72a2678d-1847-47bc-8b02-0961ff211845`::moving from state init -> state preparing >Thread-1022::INFO::2012-12-03 00:23:06,350::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1022::INFO::2012-12-03 00:23:06,350::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00843191146851', 'lastCheck': 1354486981.8700161, 'code': 0, 'valid': True}} >Thread-1022::DEBUG::2012-12-03 00:23:06,350::task::1151::TaskManager.Task::(prepare) Task=`72a2678d-1847-47bc-8b02-0961ff211845`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00843191146851', 'lastCheck': 1354486981.8700161, 'code': 0, 'valid': True}} >Thread-1022::DEBUG::2012-12-03 00:23:06,350::task::568::TaskManager.Task::(_updateState) Task=`72a2678d-1847-47bc-8b02-0961ff211845`::moving from state preparing -> state finished >Thread-1022::DEBUG::2012-12-03 00:23:06,350::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1022::DEBUG::2012-12-03 00:23:06,350::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1022::DEBUG::2012-12-03 00:23:06,351::task::957::TaskManager.Task::(_decref) Task=`72a2678d-1847-47bc-8b02-0961ff211845`::ref 0 aborting False >Thread-1028::DEBUG::2012-12-03 00:23:17,285::task::568::TaskManager.Task::(_updateState) Task=`f0dea23f-ab8b-4b4b-b1c9-901dd755216b`::moving from state init -> state preparing >Thread-1028::INFO::2012-12-03 00:23:17,285::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1028::INFO::2012-12-03 00:23:17,286::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0083019733429', 'lastCheck': 1354486991.8883669, 'code': 0, 'valid': True}} >Thread-1028::DEBUG::2012-12-03 00:23:17,286::task::1151::TaskManager.Task::(prepare) Task=`f0dea23f-ab8b-4b4b-b1c9-901dd755216b`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0083019733429', 'lastCheck': 1354486991.8883669, 'code': 0, 'valid': True}} >Thread-1028::DEBUG::2012-12-03 00:23:17,286::task::568::TaskManager.Task::(_updateState) Task=`f0dea23f-ab8b-4b4b-b1c9-901dd755216b`::moving from state preparing -> state finished >Thread-1028::DEBUG::2012-12-03 00:23:17,286::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1028::DEBUG::2012-12-03 00:23:17,286::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1028::DEBUG::2012-12-03 00:23:17,286::task::957::TaskManager.Task::(_decref) Task=`f0dea23f-ab8b-4b4b-b1c9-901dd755216b`::ref 0 aborting False >Thread-1034::DEBUG::2012-12-03 00:23:28,275::task::568::TaskManager.Task::(_updateState) Task=`996b4e12-313f-4ec2-95a4-3887c8a67116`::moving from state init -> state preparing >Thread-1034::INFO::2012-12-03 00:23:28,275::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1034::INFO::2012-12-03 00:23:28,275::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00823998451233', 'lastCheck': 1354487001.911128, 'code': 0, 'valid': True}} >Thread-1034::DEBUG::2012-12-03 00:23:28,275::task::1151::TaskManager.Task::(prepare) Task=`996b4e12-313f-4ec2-95a4-3887c8a67116`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00823998451233', 'lastCheck': 1354487001.911128, 'code': 0, 'valid': True}} >Thread-1034::DEBUG::2012-12-03 00:23:28,276::task::568::TaskManager.Task::(_updateState) Task=`996b4e12-313f-4ec2-95a4-3887c8a67116`::moving from state preparing -> state finished >Thread-1034::DEBUG::2012-12-03 00:23:28,276::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1034::DEBUG::2012-12-03 00:23:28,276::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1034::DEBUG::2012-12-03 00:23:28,276::task::957::TaskManager.Task::(_decref) Task=`996b4e12-313f-4ec2-95a4-3887c8a67116`::ref 0 aborting False >Thread-1040::DEBUG::2012-12-03 00:23:39,234::task::568::TaskManager.Task::(_updateState) Task=`462aef08-0835-4a19-99cf-067ff39adb53`::moving from state init -> state preparing >Thread-1040::INFO::2012-12-03 00:23:39,235::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1040::INFO::2012-12-03 00:23:39,235::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00866007804871', 'lastCheck': 1354487011.9304819, 'code': 0, 'valid': True}} >Thread-1040::DEBUG::2012-12-03 00:23:39,235::task::1151::TaskManager.Task::(prepare) Task=`462aef08-0835-4a19-99cf-067ff39adb53`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00866007804871', 'lastCheck': 1354487011.9304819, 'code': 0, 'valid': True}} >Thread-1040::DEBUG::2012-12-03 00:23:39,235::task::568::TaskManager.Task::(_updateState) Task=`462aef08-0835-4a19-99cf-067ff39adb53`::moving from state preparing -> state finished >Thread-1040::DEBUG::2012-12-03 00:23:39,235::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1040::DEBUG::2012-12-03 00:23:39,235::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1040::DEBUG::2012-12-03 00:23:39,235::task::957::TaskManager.Task::(_decref) Task=`462aef08-0835-4a19-99cf-067ff39adb53`::ref 0 aborting False >Thread-1046::DEBUG::2012-12-03 00:23:50,186::task::568::TaskManager.Task::(_updateState) Task=`00d4d990-b873-4ac6-85b3-80157fd98304`::moving from state init -> state preparing >Thread-1046::INFO::2012-12-03 00:23:50,186::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1046::INFO::2012-12-03 00:23:50,186::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00818800926208', 'lastCheck': 1354487021.948529, 'code': 0, 'valid': True}} >Thread-1046::DEBUG::2012-12-03 00:23:50,186::task::1151::TaskManager.Task::(prepare) Task=`00d4d990-b873-4ac6-85b3-80157fd98304`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00818800926208', 'lastCheck': 1354487021.948529, 'code': 0, 'valid': True}} >Thread-1046::DEBUG::2012-12-03 00:23:50,186::task::568::TaskManager.Task::(_updateState) Task=`00d4d990-b873-4ac6-85b3-80157fd98304`::moving from state preparing -> state finished >Thread-1046::DEBUG::2012-12-03 00:23:50,187::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1046::DEBUG::2012-12-03 00:23:50,187::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1046::DEBUG::2012-12-03 00:23:50,187::task::957::TaskManager.Task::(_decref) Task=`00d4d990-b873-4ac6-85b3-80157fd98304`::ref 0 aborting False >Thread-1052::DEBUG::2012-12-03 00:24:01,142::task::568::TaskManager.Task::(_updateState) Task=`6c0b3c64-5ef9-4b2b-9655-f91c9af82c28`::moving from state init -> state preparing >Thread-1052::INFO::2012-12-03 00:24:01,142::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1052::INFO::2012-12-03 00:24:01,142::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0143239498138', 'lastCheck': 1354487031.9751661, 'code': 0, 'valid': True}} >Thread-1052::DEBUG::2012-12-03 00:24:01,142::task::1151::TaskManager.Task::(prepare) Task=`6c0b3c64-5ef9-4b2b-9655-f91c9af82c28`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0143239498138', 'lastCheck': 1354487031.9751661, 'code': 0, 'valid': True}} >Thread-1052::DEBUG::2012-12-03 00:24:01,142::task::568::TaskManager.Task::(_updateState) Task=`6c0b3c64-5ef9-4b2b-9655-f91c9af82c28`::moving from state preparing -> state finished >Thread-1052::DEBUG::2012-12-03 00:24:01,143::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1052::DEBUG::2012-12-03 00:24:01,143::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1052::DEBUG::2012-12-03 00:24:01,143::task::957::TaskManager.Task::(_decref) Task=`6c0b3c64-5ef9-4b2b-9655-f91c9af82c28`::ref 0 aborting False >Thread-1058::DEBUG::2012-12-03 00:24:12,097::task::568::TaskManager.Task::(_updateState) Task=`19de1bcc-a632-4981-beb2-de0c72ffae9e`::moving from state init -> state preparing >Thread-1058::INFO::2012-12-03 00:24:12,097::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1058::INFO::2012-12-03 00:24:12,098::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00830292701721', 'lastCheck': 1354487052.011174, 'code': 0, 'valid': True}} >Thread-1058::DEBUG::2012-12-03 00:24:12,098::task::1151::TaskManager.Task::(prepare) Task=`19de1bcc-a632-4981-beb2-de0c72ffae9e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00830292701721', 'lastCheck': 1354487052.011174, 'code': 0, 'valid': True}} >Thread-1058::DEBUG::2012-12-03 00:24:12,098::task::568::TaskManager.Task::(_updateState) Task=`19de1bcc-a632-4981-beb2-de0c72ffae9e`::moving from state preparing -> state finished >Thread-1058::DEBUG::2012-12-03 00:24:12,098::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1058::DEBUG::2012-12-03 00:24:12,098::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1058::DEBUG::2012-12-03 00:24:12,098::task::957::TaskManager.Task::(_decref) Task=`19de1bcc-a632-4981-beb2-de0c72ffae9e`::ref 0 aborting False >Thread-1064::DEBUG::2012-12-03 00:24:23,061::task::568::TaskManager.Task::(_updateState) Task=`68a2c483-d542-4785-99bc-dd5ea2fbe1f1`::moving from state init -> state preparing >Thread-1064::INFO::2012-12-03 00:24:23,061::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1064::INFO::2012-12-03 00:24:23,061::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00849485397339', 'lastCheck': 1354487062.0355589, 'code': 0, 'valid': True}} >Thread-1064::DEBUG::2012-12-03 00:24:23,061::task::1151::TaskManager.Task::(prepare) Task=`68a2c483-d542-4785-99bc-dd5ea2fbe1f1`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00849485397339', 'lastCheck': 1354487062.0355589, 'code': 0, 'valid': True}} >Thread-1064::DEBUG::2012-12-03 00:24:23,061::task::568::TaskManager.Task::(_updateState) Task=`68a2c483-d542-4785-99bc-dd5ea2fbe1f1`::moving from state preparing -> state finished >Thread-1064::DEBUG::2012-12-03 00:24:23,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1064::DEBUG::2012-12-03 00:24:23,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1064::DEBUG::2012-12-03 00:24:23,062::task::957::TaskManager.Task::(_decref) Task=`68a2c483-d542-4785-99bc-dd5ea2fbe1f1`::ref 0 aborting False >Thread-1070::DEBUG::2012-12-03 00:24:34,014::task::568::TaskManager.Task::(_updateState) Task=`65a4fc0c-99b3-4182-af29-a36b2769d542`::moving from state init -> state preparing >Thread-1070::INFO::2012-12-03 00:24:34,015::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1070::INFO::2012-12-03 00:24:34,015::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00589489936829', 'lastCheck': 1354487072.051348, 'code': 0, 'valid': True}} >Thread-1070::DEBUG::2012-12-03 00:24:34,015::task::1151::TaskManager.Task::(prepare) Task=`65a4fc0c-99b3-4182-af29-a36b2769d542`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00589489936829', 'lastCheck': 1354487072.051348, 'code': 0, 'valid': True}} >Thread-1070::DEBUG::2012-12-03 00:24:34,015::task::568::TaskManager.Task::(_updateState) Task=`65a4fc0c-99b3-4182-af29-a36b2769d542`::moving from state preparing -> state finished >Thread-1070::DEBUG::2012-12-03 00:24:34,015::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1070::DEBUG::2012-12-03 00:24:34,015::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1070::DEBUG::2012-12-03 00:24:34,015::task::957::TaskManager.Task::(_decref) Task=`65a4fc0c-99b3-4182-af29-a36b2769d542`::ref 0 aborting False >Thread-1076::DEBUG::2012-12-03 00:24:45,024::task::568::TaskManager.Task::(_updateState) Task=`65a1b2ec-1ccb-4a84-8358-58d17fbf391e`::moving from state init -> state preparing >Thread-1076::INFO::2012-12-03 00:24:45,024::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1076::INFO::2012-12-03 00:24:45,025::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00770998001099', 'lastCheck': 1354487082.068686, 'code': 0, 'valid': True}} >Thread-1076::DEBUG::2012-12-03 00:24:45,025::task::1151::TaskManager.Task::(prepare) Task=`65a1b2ec-1ccb-4a84-8358-58d17fbf391e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00770998001099', 'lastCheck': 1354487082.068686, 'code': 0, 'valid': True}} >Thread-1076::DEBUG::2012-12-03 00:24:45,025::task::568::TaskManager.Task::(_updateState) Task=`65a1b2ec-1ccb-4a84-8358-58d17fbf391e`::moving from state preparing -> state finished >Thread-1076::DEBUG::2012-12-03 00:24:45,025::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1076::DEBUG::2012-12-03 00:24:45,025::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1076::DEBUG::2012-12-03 00:24:45,025::task::957::TaskManager.Task::(_decref) Task=`65a1b2ec-1ccb-4a84-8358-58d17fbf391e`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 00:24:52,068::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:24:52,069::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:24:52,069::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:24:52,276::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:24:52,277::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:24:52,277::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:24:52,313::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:24:52,313::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:24:52,322::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f88715a0e40bd7b8575a75e585467485b3ce9fc4'] >Thread-276::DEBUG::2012-12-03 00:24:52,325::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:24:52,325::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:24:52,325::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-1144::DEBUG::2012-12-03 00:24:55,966::task::568::TaskManager.Task::(_updateState) Task=`51eb1647-3d79-4e32-8e4c-8cb44de2a494`::moving from state init -> state preparing >Thread-1144::INFO::2012-12-03 00:24:55,966::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1144::INFO::2012-12-03 00:24:55,967::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00709104537964', 'lastCheck': 1354487092.3425789, 'code': 0, 'valid': True}} >Thread-1144::DEBUG::2012-12-03 00:24:55,967::task::1151::TaskManager.Task::(prepare) Task=`51eb1647-3d79-4e32-8e4c-8cb44de2a494`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00709104537964', 'lastCheck': 1354487092.3425789, 'code': 0, 'valid': True}} >Thread-1144::DEBUG::2012-12-03 00:24:55,967::task::568::TaskManager.Task::(_updateState) Task=`51eb1647-3d79-4e32-8e4c-8cb44de2a494`::moving from state preparing -> state finished >Thread-1144::DEBUG::2012-12-03 00:24:55,967::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1144::DEBUG::2012-12-03 00:24:55,967::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1144::DEBUG::2012-12-03 00:24:55,967::task::957::TaskManager.Task::(_decref) Task=`51eb1647-3d79-4e32-8e4c-8cb44de2a494`::ref 0 aborting False >Thread-1150::DEBUG::2012-12-03 00:25:06,898::task::568::TaskManager.Task::(_updateState) Task=`0f775934-905f-4776-a14b-3a1d7ff9bb9b`::moving from state init -> state preparing >Thread-1150::INFO::2012-12-03 00:25:06,898::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1150::INFO::2012-12-03 00:25:06,898::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00548720359802', 'lastCheck': 1354487102.3580599, 'code': 0, 'valid': True}} >Thread-1150::DEBUG::2012-12-03 00:25:06,898::task::1151::TaskManager.Task::(prepare) Task=`0f775934-905f-4776-a14b-3a1d7ff9bb9b`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00548720359802', 'lastCheck': 1354487102.3580599, 'code': 0, 'valid': True}} >Thread-1150::DEBUG::2012-12-03 00:25:06,898::task::568::TaskManager.Task::(_updateState) Task=`0f775934-905f-4776-a14b-3a1d7ff9bb9b`::moving from state preparing -> state finished >Thread-1150::DEBUG::2012-12-03 00:25:06,899::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1150::DEBUG::2012-12-03 00:25:06,899::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1150::DEBUG::2012-12-03 00:25:06,899::task::957::TaskManager.Task::(_decref) Task=`0f775934-905f-4776-a14b-3a1d7ff9bb9b`::ref 0 aborting False >Thread-1156::DEBUG::2012-12-03 00:25:17,865::task::568::TaskManager.Task::(_updateState) Task=`fe96acc5-b868-4b8f-a8cf-3cfdb89dc9c0`::moving from state init -> state preparing >Thread-1156::INFO::2012-12-03 00:25:17,865::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1156::INFO::2012-12-03 00:25:17,866::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00707912445068', 'lastCheck': 1354487112.375001, 'code': 0, 'valid': True}} >Thread-1156::DEBUG::2012-12-03 00:25:17,866::task::1151::TaskManager.Task::(prepare) Task=`fe96acc5-b868-4b8f-a8cf-3cfdb89dc9c0`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00707912445068', 'lastCheck': 1354487112.375001, 'code': 0, 'valid': True}} >Thread-1156::DEBUG::2012-12-03 00:25:17,866::task::568::TaskManager.Task::(_updateState) Task=`fe96acc5-b868-4b8f-a8cf-3cfdb89dc9c0`::moving from state preparing -> state finished >Thread-1156::DEBUG::2012-12-03 00:25:17,866::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1156::DEBUG::2012-12-03 00:25:17,866::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1156::DEBUG::2012-12-03 00:25:17,866::task::957::TaskManager.Task::(_decref) Task=`fe96acc5-b868-4b8f-a8cf-3cfdb89dc9c0`::ref 0 aborting False >Thread-1162::DEBUG::2012-12-03 00:25:28,865::task::568::TaskManager.Task::(_updateState) Task=`3205f28a-be79-4e97-ad58-8f782caa51d3`::moving from state init -> state preparing >Thread-1162::INFO::2012-12-03 00:25:28,865::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1162::INFO::2012-12-03 00:25:28,865::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00941705703735', 'lastCheck': 1354487122.4014671, 'code': 0, 'valid': True}} >Thread-1162::DEBUG::2012-12-03 00:25:28,866::task::1151::TaskManager.Task::(prepare) Task=`3205f28a-be79-4e97-ad58-8f782caa51d3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00941705703735', 'lastCheck': 1354487122.4014671, 'code': 0, 'valid': True}} >Thread-1162::DEBUG::2012-12-03 00:25:28,866::task::568::TaskManager.Task::(_updateState) Task=`3205f28a-be79-4e97-ad58-8f782caa51d3`::moving from state preparing -> state finished >Thread-1162::DEBUG::2012-12-03 00:25:28,866::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1162::DEBUG::2012-12-03 00:25:28,866::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1162::DEBUG::2012-12-03 00:25:28,866::task::957::TaskManager.Task::(_decref) Task=`3205f28a-be79-4e97-ad58-8f782caa51d3`::ref 0 aborting False >Thread-1168::DEBUG::2012-12-03 00:25:39,868::task::568::TaskManager.Task::(_updateState) Task=`70bf6764-201d-4633-8434-e5fc299740e5`::moving from state init -> state preparing >Thread-1168::INFO::2012-12-03 00:25:39,868::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1168::INFO::2012-12-03 00:25:39,868::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00822520256042', 'lastCheck': 1354487132.420119, 'code': 0, 'valid': True}} >Thread-1168::DEBUG::2012-12-03 00:25:39,868::task::1151::TaskManager.Task::(prepare) Task=`70bf6764-201d-4633-8434-e5fc299740e5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00822520256042', 'lastCheck': 1354487132.420119, 'code': 0, 'valid': True}} >Thread-1168::DEBUG::2012-12-03 00:25:39,868::task::568::TaskManager.Task::(_updateState) Task=`70bf6764-201d-4633-8434-e5fc299740e5`::moving from state preparing -> state finished >Thread-1168::DEBUG::2012-12-03 00:25:39,869::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1168::DEBUG::2012-12-03 00:25:39,869::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1168::DEBUG::2012-12-03 00:25:39,869::task::957::TaskManager.Task::(_decref) Task=`70bf6764-201d-4633-8434-e5fc299740e5`::ref 0 aborting False >Thread-1174::DEBUG::2012-12-03 00:25:51,088::task::568::TaskManager.Task::(_updateState) Task=`d723c8f7-56d7-4395-a368-151d4687bc3e`::moving from state init -> state preparing >Thread-1174::INFO::2012-12-03 00:25:51,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1174::INFO::2012-12-03 00:25:51,089::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00817704200745', 'lastCheck': 1354487142.4389901, 'code': 0, 'valid': True}} >Thread-1174::DEBUG::2012-12-03 00:25:51,089::task::1151::TaskManager.Task::(prepare) Task=`d723c8f7-56d7-4395-a368-151d4687bc3e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00817704200745', 'lastCheck': 1354487142.4389901, 'code': 0, 'valid': True}} >Thread-1174::DEBUG::2012-12-03 00:25:51,089::task::568::TaskManager.Task::(_updateState) Task=`d723c8f7-56d7-4395-a368-151d4687bc3e`::moving from state preparing -> state finished >Thread-1174::DEBUG::2012-12-03 00:25:51,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1174::DEBUG::2012-12-03 00:25:51,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1174::DEBUG::2012-12-03 00:25:51,089::task::957::TaskManager.Task::(_decref) Task=`d723c8f7-56d7-4395-a368-151d4687bc3e`::ref 0 aborting False >Thread-1180::DEBUG::2012-12-03 00:26:02,039::task::568::TaskManager.Task::(_updateState) Task=`81f9d9f0-fd24-4a9b-b880-e33f6f974489`::moving from state init -> state preparing >Thread-1180::INFO::2012-12-03 00:26:02,039::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1180::INFO::2012-12-03 00:26:02,040::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0122799873352', 'lastCheck': 1354487152.4639399, 'code': 0, 'valid': True}} >Thread-1180::DEBUG::2012-12-03 00:26:02,040::task::1151::TaskManager.Task::(prepare) Task=`81f9d9f0-fd24-4a9b-b880-e33f6f974489`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0122799873352', 'lastCheck': 1354487152.4639399, 'code': 0, 'valid': True}} >Thread-1180::DEBUG::2012-12-03 00:26:02,040::task::568::TaskManager.Task::(_updateState) Task=`81f9d9f0-fd24-4a9b-b880-e33f6f974489`::moving from state preparing -> state finished >Thread-1180::DEBUG::2012-12-03 00:26:02,040::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1180::DEBUG::2012-12-03 00:26:02,040::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1180::DEBUG::2012-12-03 00:26:02,040::task::957::TaskManager.Task::(_decref) Task=`81f9d9f0-fd24-4a9b-b880-e33f6f974489`::ref 0 aborting False >Thread-1186::DEBUG::2012-12-03 00:26:12,999::task::568::TaskManager.Task::(_updateState) Task=`97f65f0b-7f33-4b52-b1a8-5b23ae1ddcc7`::moving from state init -> state preparing >Thread-1186::INFO::2012-12-03 00:26:12,999::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1186::INFO::2012-12-03 00:26:12,999::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00795078277588', 'lastCheck': 1354487172.5023201, 'code': 0, 'valid': True}} >Thread-1186::DEBUG::2012-12-03 00:26:12,999::task::1151::TaskManager.Task::(prepare) Task=`97f65f0b-7f33-4b52-b1a8-5b23ae1ddcc7`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00795078277588', 'lastCheck': 1354487172.5023201, 'code': 0, 'valid': True}} >Thread-1186::DEBUG::2012-12-03 00:26:13,000::task::568::TaskManager.Task::(_updateState) Task=`97f65f0b-7f33-4b52-b1a8-5b23ae1ddcc7`::moving from state preparing -> state finished >Thread-1186::DEBUG::2012-12-03 00:26:13,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1186::DEBUG::2012-12-03 00:26:13,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1186::DEBUG::2012-12-03 00:26:13,000::task::957::TaskManager.Task::(_decref) Task=`97f65f0b-7f33-4b52-b1a8-5b23ae1ddcc7`::ref 0 aborting False >Thread-1192::DEBUG::2012-12-03 00:26:23,980::task::568::TaskManager.Task::(_updateState) Task=`8263daa8-61c7-4baf-bdb3-0bbc1aafa8ac`::moving from state init -> state preparing >Thread-1192::INFO::2012-12-03 00:26:23,981::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1192::INFO::2012-12-03 00:26:23,981::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0277750492096', 'lastCheck': 1354487182.544348, 'code': 0, 'valid': True}} >Thread-1192::DEBUG::2012-12-03 00:26:23,981::task::1151::TaskManager.Task::(prepare) Task=`8263daa8-61c7-4baf-bdb3-0bbc1aafa8ac`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0277750492096', 'lastCheck': 1354487182.544348, 'code': 0, 'valid': True}} >Thread-1192::DEBUG::2012-12-03 00:26:23,981::task::568::TaskManager.Task::(_updateState) Task=`8263daa8-61c7-4baf-bdb3-0bbc1aafa8ac`::moving from state preparing -> state finished >Thread-1192::DEBUG::2012-12-03 00:26:23,981::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1192::DEBUG::2012-12-03 00:26:23,981::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1192::DEBUG::2012-12-03 00:26:23,981::task::957::TaskManager.Task::(_decref) Task=`8263daa8-61c7-4baf-bdb3-0bbc1aafa8ac`::ref 0 aborting False >Thread-1198::DEBUG::2012-12-03 00:26:34,920::task::568::TaskManager.Task::(_updateState) Task=`44fdba89-f0da-4077-8b70-ad0fea0f2115`::moving from state init -> state preparing >Thread-1198::INFO::2012-12-03 00:26:34,920::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1198::INFO::2012-12-03 00:26:34,921::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.007728099823', 'lastCheck': 1354487192.5622921, 'code': 0, 'valid': True}} >Thread-1198::DEBUG::2012-12-03 00:26:34,921::task::1151::TaskManager.Task::(prepare) Task=`44fdba89-f0da-4077-8b70-ad0fea0f2115`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.007728099823', 'lastCheck': 1354487192.5622921, 'code': 0, 'valid': True}} >Thread-1198::DEBUG::2012-12-03 00:26:34,921::task::568::TaskManager.Task::(_updateState) Task=`44fdba89-f0da-4077-8b70-ad0fea0f2115`::moving from state preparing -> state finished >Thread-1198::DEBUG::2012-12-03 00:26:34,921::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1198::DEBUG::2012-12-03 00:26:34,921::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1198::DEBUG::2012-12-03 00:26:34,921::task::957::TaskManager.Task::(_decref) Task=`44fdba89-f0da-4077-8b70-ad0fea0f2115`::ref 0 aborting False >Thread-1204::DEBUG::2012-12-03 00:26:45,888::task::568::TaskManager.Task::(_updateState) Task=`eee1e939-36db-478c-a78c-ea1b21d6e127`::moving from state init -> state preparing >Thread-1204::INFO::2012-12-03 00:26:45,888::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1204::INFO::2012-12-03 00:26:45,888::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00713205337524', 'lastCheck': 1354487202.578856, 'code': 0, 'valid': True}} >Thread-1204::DEBUG::2012-12-03 00:26:45,888::task::1151::TaskManager.Task::(prepare) Task=`eee1e939-36db-478c-a78c-ea1b21d6e127`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00713205337524', 'lastCheck': 1354487202.578856, 'code': 0, 'valid': True}} >Thread-1204::DEBUG::2012-12-03 00:26:45,888::task::568::TaskManager.Task::(_updateState) Task=`eee1e939-36db-478c-a78c-ea1b21d6e127`::moving from state preparing -> state finished >Thread-1204::DEBUG::2012-12-03 00:26:45,888::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1204::DEBUG::2012-12-03 00:26:45,889::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1204::DEBUG::2012-12-03 00:26:45,889::task::957::TaskManager.Task::(_decref) Task=`eee1e939-36db-478c-a78c-ea1b21d6e127`::ref 0 aborting False >Thread-1210::DEBUG::2012-12-03 00:26:56,837::task::568::TaskManager.Task::(_updateState) Task=`1e8e2cce-6904-4ef8-a9d8-d942a88aa701`::moving from state init -> state preparing >Thread-1210::INFO::2012-12-03 00:26:56,837::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1210::INFO::2012-12-03 00:26:56,838::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00673818588257', 'lastCheck': 1354487212.598078, 'code': 0, 'valid': True}} >Thread-1210::DEBUG::2012-12-03 00:26:56,838::task::1151::TaskManager.Task::(prepare) Task=`1e8e2cce-6904-4ef8-a9d8-d942a88aa701`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00673818588257', 'lastCheck': 1354487212.598078, 'code': 0, 'valid': True}} >Thread-1210::DEBUG::2012-12-03 00:26:56,838::task::568::TaskManager.Task::(_updateState) Task=`1e8e2cce-6904-4ef8-a9d8-d942a88aa701`::moving from state preparing -> state finished >Thread-1210::DEBUG::2012-12-03 00:26:56,838::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1210::DEBUG::2012-12-03 00:26:56,838::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1210::DEBUG::2012-12-03 00:26:56,838::task::957::TaskManager.Task::(_decref) Task=`1e8e2cce-6904-4ef8-a9d8-d942a88aa701`::ref 0 aborting False >Thread-1216::DEBUG::2012-12-03 00:27:07,773::task::568::TaskManager.Task::(_updateState) Task=`b0c4b73b-3867-4d9e-b3c8-66605190dfad`::moving from state init -> state preparing >Thread-1216::INFO::2012-12-03 00:27:07,773::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1216::INFO::2012-12-03 00:27:07,773::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0077760219574', 'lastCheck': 1354487222.61622, 'code': 0, 'valid': True}} >Thread-1216::DEBUG::2012-12-03 00:27:07,773::task::1151::TaskManager.Task::(prepare) Task=`b0c4b73b-3867-4d9e-b3c8-66605190dfad`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0077760219574', 'lastCheck': 1354487222.61622, 'code': 0, 'valid': True}} >Thread-1216::DEBUG::2012-12-03 00:27:07,774::task::568::TaskManager.Task::(_updateState) Task=`b0c4b73b-3867-4d9e-b3c8-66605190dfad`::moving from state preparing -> state finished >Thread-1216::DEBUG::2012-12-03 00:27:07,774::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1216::DEBUG::2012-12-03 00:27:07,774::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1216::DEBUG::2012-12-03 00:27:07,774::task::957::TaskManager.Task::(_decref) Task=`b0c4b73b-3867-4d9e-b3c8-66605190dfad`::ref 0 aborting False >Thread-1222::DEBUG::2012-12-03 00:27:18,790::task::568::TaskManager.Task::(_updateState) Task=`9288b46e-0eeb-467e-99df-34fe3eab341e`::moving from state init -> state preparing >Thread-1222::INFO::2012-12-03 00:27:18,791::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1222::INFO::2012-12-03 00:27:18,791::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0094838142395', 'lastCheck': 1354487232.6363699, 'code': 0, 'valid': True}} >Thread-1222::DEBUG::2012-12-03 00:27:18,791::task::1151::TaskManager.Task::(prepare) Task=`9288b46e-0eeb-467e-99df-34fe3eab341e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0094838142395', 'lastCheck': 1354487232.6363699, 'code': 0, 'valid': True}} >Thread-1222::DEBUG::2012-12-03 00:27:18,791::task::568::TaskManager.Task::(_updateState) Task=`9288b46e-0eeb-467e-99df-34fe3eab341e`::moving from state preparing -> state finished >Thread-1222::DEBUG::2012-12-03 00:27:18,791::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1222::DEBUG::2012-12-03 00:27:18,791::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1222::DEBUG::2012-12-03 00:27:18,791::task::957::TaskManager.Task::(_decref) Task=`9288b46e-0eeb-467e-99df-34fe3eab341e`::ref 0 aborting False >Thread-1228::DEBUG::2012-12-03 00:27:29,734::task::568::TaskManager.Task::(_updateState) Task=`fa6a6f1b-e424-4228-8ce0-cfdc29d9144a`::moving from state init -> state preparing >Thread-1228::INFO::2012-12-03 00:27:29,734::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1228::INFO::2012-12-03 00:27:29,735::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00681090354919', 'lastCheck': 1354487242.6583869, 'code': 0, 'valid': True}} >Thread-1228::DEBUG::2012-12-03 00:27:29,735::task::1151::TaskManager.Task::(prepare) Task=`fa6a6f1b-e424-4228-8ce0-cfdc29d9144a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00681090354919', 'lastCheck': 1354487242.6583869, 'code': 0, 'valid': True}} >Thread-1228::DEBUG::2012-12-03 00:27:29,735::task::568::TaskManager.Task::(_updateState) Task=`fa6a6f1b-e424-4228-8ce0-cfdc29d9144a`::moving from state preparing -> state finished >Thread-1228::DEBUG::2012-12-03 00:27:29,735::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1228::DEBUG::2012-12-03 00:27:29,735::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1228::DEBUG::2012-12-03 00:27:29,735::task::957::TaskManager.Task::(_decref) Task=`fa6a6f1b-e424-4228-8ce0-cfdc29d9144a`::ref 0 aborting False >Thread-1234::DEBUG::2012-12-03 00:27:40,694::task::568::TaskManager.Task::(_updateState) Task=`fb23bc7a-242d-451f-975f-314b1be03146`::moving from state init -> state preparing >Thread-1234::INFO::2012-12-03 00:27:40,694::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1234::INFO::2012-12-03 00:27:40,694::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00805687904358', 'lastCheck': 1354487252.6771369, 'code': 0, 'valid': True}} >Thread-1234::DEBUG::2012-12-03 00:27:40,694::task::1151::TaskManager.Task::(prepare) Task=`fb23bc7a-242d-451f-975f-314b1be03146`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00805687904358', 'lastCheck': 1354487252.6771369, 'code': 0, 'valid': True}} >Thread-1234::DEBUG::2012-12-03 00:27:40,694::task::568::TaskManager.Task::(_updateState) Task=`fb23bc7a-242d-451f-975f-314b1be03146`::moving from state preparing -> state finished >Thread-1234::DEBUG::2012-12-03 00:27:40,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1234::DEBUG::2012-12-03 00:27:40,695::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1234::DEBUG::2012-12-03 00:27:40,695::task::957::TaskManager.Task::(_decref) Task=`fb23bc7a-242d-451f-975f-314b1be03146`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 00:29:52,932::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:29:52,932::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:29:52,933::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:29:53,141::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:29:53,143::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:29:53,143::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:29:53,180::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:29:53,180::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:29:53,187::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=f88715a0e40bd7b8575a75e585467485b3ce9fc4'] >Thread-276::DEBUG::2012-12-03 00:29:53,190::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:29:53,190::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:29:53,190::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-1302::DEBUG::2012-12-03 00:30:11,137::task::568::TaskManager.Task::(_updateState) Task=`7633053b-c165-438b-a6f7-33ad31feb2b6`::moving from state init -> state preparing >Thread-1302::INFO::2012-12-03 00:30:11,138::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1302::INFO::2012-12-03 00:30:11,138::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00775408744812', 'lastCheck': 1354487403.221518, 'code': 0, 'valid': True}} >Thread-1302::DEBUG::2012-12-03 00:30:11,138::task::1151::TaskManager.Task::(prepare) Task=`7633053b-c165-438b-a6f7-33ad31feb2b6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00775408744812', 'lastCheck': 1354487403.221518, 'code': 0, 'valid': True}} >Thread-1302::DEBUG::2012-12-03 00:30:11,138::task::568::TaskManager.Task::(_updateState) Task=`7633053b-c165-438b-a6f7-33ad31feb2b6`::moving from state preparing -> state finished >Thread-1302::DEBUG::2012-12-03 00:30:11,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1302::DEBUG::2012-12-03 00:30:11,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1302::DEBUG::2012-12-03 00:30:11,139::task::957::TaskManager.Task::(_decref) Task=`7633053b-c165-438b-a6f7-33ad31feb2b6`::ref 0 aborting False >Thread-1308::DEBUG::2012-12-03 00:30:22,086::task::568::TaskManager.Task::(_updateState) Task=`bb56c661-a23b-4ff8-823d-6f5950fa20e3`::moving from state init -> state preparing >Thread-1308::INFO::2012-12-03 00:30:22,086::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1308::INFO::2012-12-03 00:30:22,086::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0123860836029', 'lastCheck': 1354487413.2443459, 'code': 0, 'valid': True}} >Thread-1308::DEBUG::2012-12-03 00:30:22,086::task::1151::TaskManager.Task::(prepare) Task=`bb56c661-a23b-4ff8-823d-6f5950fa20e3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0123860836029', 'lastCheck': 1354487413.2443459, 'code': 0, 'valid': True}} >Thread-1308::DEBUG::2012-12-03 00:30:22,086::task::568::TaskManager.Task::(_updateState) Task=`bb56c661-a23b-4ff8-823d-6f5950fa20e3`::moving from state preparing -> state finished >Thread-1308::DEBUG::2012-12-03 00:30:22,086::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1308::DEBUG::2012-12-03 00:30:22,086::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1308::DEBUG::2012-12-03 00:30:22,087::task::957::TaskManager.Task::(_decref) Task=`bb56c661-a23b-4ff8-823d-6f5950fa20e3`::ref 0 aborting False >Thread-1314::DEBUG::2012-12-03 00:30:33,050::task::568::TaskManager.Task::(_updateState) Task=`5502c3b7-d4d2-496f-92b1-459d3258f77e`::moving from state init -> state preparing >Thread-1314::INFO::2012-12-03 00:30:33,051::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1314::INFO::2012-12-03 00:30:33,051::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0093469619751', 'lastCheck': 1354487423.2706079, 'code': 0, 'valid': True}} >Thread-1314::DEBUG::2012-12-03 00:30:33,051::task::1151::TaskManager.Task::(prepare) Task=`5502c3b7-d4d2-496f-92b1-459d3258f77e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0093469619751', 'lastCheck': 1354487423.2706079, 'code': 0, 'valid': True}} >Thread-1314::DEBUG::2012-12-03 00:30:33,051::task::568::TaskManager.Task::(_updateState) Task=`5502c3b7-d4d2-496f-92b1-459d3258f77e`::moving from state preparing -> state finished >Thread-1314::DEBUG::2012-12-03 00:30:33,051::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1314::DEBUG::2012-12-03 00:30:33,051::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1314::DEBUG::2012-12-03 00:30:33,051::task::957::TaskManager.Task::(_decref) Task=`5502c3b7-d4d2-496f-92b1-459d3258f77e`::ref 0 aborting False >Thread-1320::DEBUG::2012-12-03 00:30:44,010::task::568::TaskManager.Task::(_updateState) Task=`0118d9d9-b805-4fc0-b56b-c451403117cb`::moving from state init -> state preparing >Thread-1320::INFO::2012-12-03 00:30:44,010::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1320::INFO::2012-12-03 00:30:44,010::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00811505317688', 'lastCheck': 1354487443.3081281, 'code': 0, 'valid': True}} >Thread-1320::DEBUG::2012-12-03 00:30:44,010::task::1151::TaskManager.Task::(prepare) Task=`0118d9d9-b805-4fc0-b56b-c451403117cb`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00811505317688', 'lastCheck': 1354487443.3081281, 'code': 0, 'valid': True}} >Thread-1320::DEBUG::2012-12-03 00:30:44,011::task::568::TaskManager.Task::(_updateState) Task=`0118d9d9-b805-4fc0-b56b-c451403117cb`::moving from state preparing -> state finished >Thread-1320::DEBUG::2012-12-03 00:30:44,011::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1320::DEBUG::2012-12-03 00:30:44,011::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1320::DEBUG::2012-12-03 00:30:44,011::task::957::TaskManager.Task::(_decref) Task=`0118d9d9-b805-4fc0-b56b-c451403117cb`::ref 0 aborting False >Thread-1326::DEBUG::2012-12-03 00:30:55,259::task::568::TaskManager.Task::(_updateState) Task=`3a102cfe-99a3-4a59-9fa7-32e946ecb03f`::moving from state init -> state preparing >Thread-1326::INFO::2012-12-03 00:30:55,259::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1326::INFO::2012-12-03 00:30:55,259::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0107400417328', 'lastCheck': 1354487453.331214, 'code': 0, 'valid': True}} >Thread-1326::DEBUG::2012-12-03 00:30:55,260::task::1151::TaskManager.Task::(prepare) Task=`3a102cfe-99a3-4a59-9fa7-32e946ecb03f`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0107400417328', 'lastCheck': 1354487453.331214, 'code': 0, 'valid': True}} >Thread-1326::DEBUG::2012-12-03 00:30:55,260::task::568::TaskManager.Task::(_updateState) Task=`3a102cfe-99a3-4a59-9fa7-32e946ecb03f`::moving from state preparing -> state finished >Thread-1326::DEBUG::2012-12-03 00:30:55,260::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1326::DEBUG::2012-12-03 00:30:55,260::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1326::DEBUG::2012-12-03 00:30:55,260::task::957::TaskManager.Task::(_decref) Task=`3a102cfe-99a3-4a59-9fa7-32e946ecb03f`::ref 0 aborting False >Thread-1332::DEBUG::2012-12-03 00:31:05,353::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1332::DEBUG::2012-12-03 00:31:05,353::task::568::TaskManager.Task::(_updateState) Task=`068daaeb-f91b-45ba-9e48-5f9f51b53b16`::moving from state init -> state preparing >Thread-1332::INFO::2012-12-03 00:31:05,354::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1332::INFO::2012-12-03 00:31:05,354::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1332::DEBUG::2012-12-03 00:31:05,354::task::1151::TaskManager.Task::(prepare) Task=`068daaeb-f91b-45ba-9e48-5f9f51b53b16`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}} >Thread-1332::DEBUG::2012-12-03 00:31:05,354::task::568::TaskManager.Task::(_updateState) Task=`068daaeb-f91b-45ba-9e48-5f9f51b53b16`::moving from state preparing -> state finished >Thread-1332::DEBUG::2012-12-03 00:31:05,354::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1332::DEBUG::2012-12-03 00:31:05,354::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1332::DEBUG::2012-12-03 00:31:05,354::task::957::TaskManager.Task::(_decref) Task=`068daaeb-f91b-45ba-9e48-5f9f51b53b16`::ref 0 aborting False >Thread-1333::DEBUG::2012-12-03 00:31:05,871::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1333::DEBUG::2012-12-03 00:31:05,871::task::568::TaskManager.Task::(_updateState) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::moving from state init -> state preparing >Thread-1333::INFO::2012-12-03 00:31:05,871::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', prevID=1, prevLVER='0', recoveryMode=None, scsiFencing='true', maxHostID=250, domVersion='3', options=None) >Thread-1333::DEBUG::2012-12-03 00:31:05,871::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`23fe2408-0184-4876-902a-714477986dd2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1333::DEBUG::2012-12-03 00:31:05,872::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-1333::DEBUG::2012-12-03 00:31:05,872::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-1333::DEBUG::2012-12-03 00:31:05,872::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`23fe2408-0184-4876-902a-714477986dd2`::Granted request >Thread-1333::DEBUG::2012-12-03 00:31:05,872::task::794::TaskManager.Task::(resourceAcquired) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-1333::DEBUG::2012-12-03 00:31:05,872::task::957::TaskManager.Task::(_decref) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::ref 1 aborting False >Thread-1333::INFO::2012-12-03 00:31:05,873::logUtils::39::dispatcher::(wrapper) Run and protect: spmStart, Return response: None >Thread-1333::DEBUG::2012-12-03 00:31:05,873::task::1146::TaskManager.Task::(prepare) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::Prepare: 1 jobs exist, move to acquiring >Thread-1333::DEBUG::2012-12-03 00:31:05,873::task::568::TaskManager.Task::(_updateState) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::moving from state preparing -> state acquiring >Thread-1333::DEBUG::2012-12-03 00:31:05,873::task::568::TaskManager.Task::(_updateState) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::moving from state acquiring -> state queued >Thread-1333::DEBUG::2012-12-03 00:31:05,873::taskManager::50::TaskManager::(_queueTask) queuing task: f768dad4-76b2-435a-963b-91b1579c59d9 >Thread-1333::DEBUG::2012-12-03 00:31:05,873::taskManager::56::TaskManager::(_queueTask) task queued: f768dad4-76b2-435a-963b-91b1579c59d9 >Thread-1333::DEBUG::2012-12-03 00:31:05,873::task::1148::TaskManager.Task::(prepare) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::returning >Thread-1333::DEBUG::2012-12-03 00:31:05,873::task::957::TaskManager.Task::(_decref) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::ref 0 aborting False >Thread-10::DEBUG::2012-12-03 00:31:05,874::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,874::threadPool::212::Misc.ThreadPool.WorkerThread::(run) Task: f768dad4-76b2-435a-963b-91b1579c59d9 running: <bound method Task.commit of <storage.task.Task instance at 0x7effa046ef38>> with: None >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,874::task::1159::TaskManager.Task::(commit) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::committing task: f768dad4-76b2-435a-963b-91b1579c59d9 >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,875::task::568::TaskManager.Task::(_updateState) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::moving from state queued -> state running >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,875::task::872::TaskManager.Task::(_runJobs) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x16e8c90>> (args: (1, '0', 'true', 250, 3) kwargs: {}) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,875::task::304::TaskManager.Task::(run) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x16e8c90>> (args: (1, '0', 'true', 250, 3) kwargs: {}) callback None >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,875::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,875::misc::1063::SamplingMethod::(__call__) Got in to sampling method >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=271d1ab19005a7e5495db6db529735062f04db7d'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,886::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:05,886::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,094::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:06,096::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,096::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1392::WARNING::2012-12-03 00:31:06,123::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >f768dad4-76b2-435a-963b-91b1579c59d9::ERROR::2012-12-03 00:31:06,134::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `225bca28-f0a0-4899-b137-32acd04f57d4` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,134::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,135::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-1396::DEBUG::2012-12-03 00:31:06,184::task::568::TaskManager.Task::(_updateState) Task=`92f56a6d-a972-48b2-a6ec-d0f0114d3a00`::moving from state init -> state preparing >Thread-1396::INFO::2012-12-03 00:31:06,184::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-1396::INFO::2012-12-03 00:31:06,184::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00999808311462', 'lastCheck': 1354487463.3515511, 'code': 0, 'valid': True}} >Thread-1396::DEBUG::2012-12-03 00:31:06,184::task::1151::TaskManager.Task::(prepare) Task=`92f56a6d-a972-48b2-a6ec-d0f0114d3a00`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00999808311462', 'lastCheck': 1354487463.3515511, 'code': 0, 'valid': True}} >Thread-1396::DEBUG::2012-12-03 00:31:06,184::task::568::TaskManager.Task::(_updateState) Task=`92f56a6d-a972-48b2-a6ec-d0f0114d3a00`::moving from state preparing -> state finished >Thread-1396::DEBUG::2012-12-03 00:31:06,184::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1396::DEBUG::2012-12-03 00:31:06,184::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1396::DEBUG::2012-12-03 00:31:06,185::task::957::TaskManager.Task::(_decref) Task=`92f56a6d-a972-48b2-a6ec-d0f0114d3a00`::ref 0 aborting False >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,339::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:06,340::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,341::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::ERROR::2012-12-03 00:31:06,369::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `5971512c-ee51-475e-b956-3872c38ce193` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('5971512c-ee51-475e-b956-3872c38ce193',) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,369::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,370::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,574::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:06,576::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,576::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::ERROR::2012-12-03 00:31:06,605::sp::1557::Storage.StoragePool::(updateMonitoringThreads) Unexpected error while trying to monitor domain `2ee486e4-abfd-4f11-9b29-42bfc74d7064` >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1552, in updateMonitoringThreads > .startMonitoring(sdCache.produce(sdUUID), self.id) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,605::misc::1071::SamplingMethod::(__call__) Returning last result >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,614::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=271d1ab19005a7e5495db6db529735062f04db7d'] >f768dad4-76b2-435a-963b-91b1579c59d9::INFO::2012-12-03 00:31:06,614::sp::244::Storage.StoragePool::(startSpm) expected previd:-1 lver:0 got request for previd:1 lver:0 >f768dad4-76b2-435a-963b-91b1579c59d9::INFO::2012-12-03 00:31:06,614::safelease::160::SANLock::(acquireHostId) Acquiring host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 (id: 2) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,615::safelease::178::SANLock::(acquireHostId) Host id for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully acquired (id: 2) >f768dad4-76b2-435a-963b-91b1579c59d9::INFO::2012-12-03 00:31:06,615::safelease::211::SANLock::(acquire) Acquiring cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 (id: 2) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,936::safelease::235::SANLock::(acquire) Cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully acquired (id: 2) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,936::sp::256::Storage.StoragePool::(startSpm) spm lock acquired successfully >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,940::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=0', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=271d1ab19005a7e5495db6db529735062f04db7d'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,940::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,940::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,941::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,975::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,976::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`779e819f-a938-4c50-b73b-09cdf7b5b3d6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,976::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,976::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,976::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`779e819f-a938-4c50-b73b-09cdf7b5b3d6`::Granted request >f768dad4-76b2-435a-963b-91b1579c59d9::INFO::2012-12-03 00:31:06,977::sp::403::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,977::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76`ReqID=`5cb90a84-252d-41bf-bfd5-dacbd147060c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,977::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' for lock type 'exclusive' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,977::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' is free. Now locking as 'exclusive' (1 active user) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,977::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76`ReqID=`5cb90a84-252d-41bf-bfd5-dacbd147060c`::Granted request >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,978::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,978::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' (0 active users) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,978::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76' is free, finding out if anyone is waiting for it. >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,978::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a00efcc1-5685-4dbe-ae70-440e9f403b76', Clearing records. >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,978::sp::407::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,978::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:06,979::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-1522::DEBUG::2012-12-03 00:31:07,027::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1522::DEBUG::2012-12-03 00:31:07,028::task::568::TaskManager.Task::(_updateState) Task=`673b9337-b6c8-4435-b10a-e5019610a3e5`::moving from state init -> state preparing >Thread-1522::INFO::2012-12-03 00:31:07,028::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='f768dad4-76b2-435a-963b-91b1579c59d9', spUUID=None, options=None) >Thread-1522::DEBUG::2012-12-03 00:31:07,029::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: f768dad4-76b2-435a-963b-91b1579c59d9 >Thread-1522::DEBUG::2012-12-03 00:31:07,029::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'} >Thread-1522::INFO::2012-12-03 00:31:07,029::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'}} >Thread-1522::DEBUG::2012-12-03 00:31:07,029::task::1151::TaskManager.Task::(prepare) Task=`673b9337-b6c8-4435-b10a-e5019610a3e5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'}} >Thread-1522::DEBUG::2012-12-03 00:31:07,029::task::568::TaskManager.Task::(_updateState) Task=`673b9337-b6c8-4435-b10a-e5019610a3e5`::moving from state preparing -> state finished >Thread-1522::DEBUG::2012-12-03 00:31:07,030::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1522::DEBUG::2012-12-03 00:31:07,030::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1522::DEBUG::2012-12-03 00:31:07,030::task::957::TaskManager.Task::(_decref) Task=`673b9337-b6c8-4435-b10a-e5019610a3e5`::ref 0 aborting False >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,185::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:07,187::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,187::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,221::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,221::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,239::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:07,239::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:07,240::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,240::sp::414::Storage.StoragePool::(_upgradePool) Registering with state change event >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,240::sp::416::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads >Thread-1585::DEBUG::2012-12-03 00:31:07,240::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1585::DEBUG::2012-12-03 00:31:07,241::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,245::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,245::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,245::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:07,246::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1585::DEBUG::2012-12-03 00:31:07,462::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1585::WARNING::2012-12-03 00:31:07,463::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1585::DEBUG::2012-12-03 00:31:07,463::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1586::DEBUG::2012-12-03 00:31:07,492::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1586::DEBUG::2012-12-03 00:31:07,493::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-1586::DEBUG::2012-12-03 00:31:07,699::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-1586::WARNING::2012-12-03 00:31:07,701::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-1586::DEBUG::2012-12-03 00:31:07,701::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1587::DEBUG::2012-12-03 00:31:07,731::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1587::DEBUG::2012-12-03 00:31:07,732::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1587::DEBUG::2012-12-03 00:31:07,937::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1587::WARNING::2012-12-03 00:31:07,939::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1587::DEBUG::2012-12-03 00:31:07,939::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,004::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,004::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1774::DEBUG::2012-12-03 00:31:08,174::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1774::DEBUG::2012-12-03 00:31:08,175::task::568::TaskManager.Task::(_updateState) Task=`a97a5b47-8ede-44da-bc6e-058f7088b8e4`::moving from state init -> state preparing >Thread-1774::INFO::2012-12-03 00:31:08,175::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='f768dad4-76b2-435a-963b-91b1579c59d9', spUUID=None, options=None) >Thread-1774::DEBUG::2012-12-03 00:31:08,175::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: f768dad4-76b2-435a-963b-91b1579c59d9 >Thread-1774::DEBUG::2012-12-03 00:31:08,175::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'} >Thread-1774::INFO::2012-12-03 00:31:08,175::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'}} >Thread-1774::DEBUG::2012-12-03 00:31:08,175::task::1151::TaskManager.Task::(prepare) Task=`a97a5b47-8ede-44da-bc6e-058f7088b8e4`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'}} >Thread-1774::DEBUG::2012-12-03 00:31:08,175::task::568::TaskManager.Task::(_updateState) Task=`a97a5b47-8ede-44da-bc6e-058f7088b8e4`::moving from state preparing -> state finished >Thread-1774::DEBUG::2012-12-03 00:31:08,175::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1774::DEBUG::2012-12-03 00:31:08,176::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1774::DEBUG::2012-12-03 00:31:08,176::task::957::TaskManager.Task::(_decref) Task=`a97a5b47-8ede-44da-bc6e-058f7088b8e4`::ref 0 aborting False >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,213::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >f768dad4-76b2-435a-963b-91b1579c59d9::WARNING::2012-12-03 00:31:08,214::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,215::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >f768dad4-76b2-435a-963b-91b1579c59d9::ERROR::2012-12-03 00:31:08,243::sp::277::Storage.StoragePool::(startSpm) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 275, in startSpm > self.checkBackupDomain(__securityOverride=True) > File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper > return f(self, *args, **kwargs) > File "/usr/share/vdsm/storage/sp.py", line 1568, in checkBackupDomain > dom = sdCache.produce(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,247::sp::305::Storage.StoragePool::(startSpm) ended. >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,247::task::568::TaskManager.Task::(_updateState) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::moving from state running -> state finished >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,247::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,247::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,247::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,248::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,248::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,248::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,248::task::880::TaskManager.Task::(_runJobs) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::Task.run: exit - success: result >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,248::task::957::TaskManager.Task::(_decref) Task=`f768dad4-76b2-435a-963b-91b1579c59d9`::ref 0 aborting False >f768dad4-76b2-435a-963b-91b1579c59d9::DEBUG::2012-12-03 00:31:08,248::threadPool::67::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-1838::DEBUG::2012-12-03 00:31:09,323::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1838::DEBUG::2012-12-03 00:31:09,324::task::568::TaskManager.Task::(_updateState) Task=`db63bb7b-7d85-4331-bbf1-ff609235dcb2`::moving from state init -> state preparing >Thread-1838::INFO::2012-12-03 00:31:09,324::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='f768dad4-76b2-435a-963b-91b1579c59d9', spUUID=None, options=None) >Thread-1838::DEBUG::2012-12-03 00:31:09,324::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: f768dad4-76b2-435a-963b-91b1579c59d9 >Thread-1838::DEBUG::2012-12-03 00:31:09,324::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'} >Thread-1838::INFO::2012-12-03 00:31:09,324::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'}} >Thread-1838::DEBUG::2012-12-03 00:31:09,324::task::1151::TaskManager.Task::(prepare) Task=`db63bb7b-7d85-4331-bbf1-ff609235dcb2`::finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'f768dad4-76b2-435a-963b-91b1579c59d9'}} >Thread-1838::DEBUG::2012-12-03 00:31:09,324::task::568::TaskManager.Task::(_updateState) Task=`db63bb7b-7d85-4331-bbf1-ff609235dcb2`::moving from state preparing -> state finished >Thread-1838::DEBUG::2012-12-03 00:31:09,325::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1838::DEBUG::2012-12-03 00:31:09,325::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1838::DEBUG::2012-12-03 00:31:09,325::task::957::TaskManager.Task::(_decref) Task=`db63bb7b-7d85-4331-bbf1-ff609235dcb2`::ref 0 aborting False >Thread-1839::DEBUG::2012-12-03 00:31:09,469::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1839::DEBUG::2012-12-03 00:31:09,470::task::568::TaskManager.Task::(_updateState) Task=`7fa802a0-0413-496f-b2a1-9799d33cbc65`::moving from state init -> state preparing >Thread-1839::INFO::2012-12-03 00:31:09,470::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1839::INFO::2012-12-03 00:31:09,470::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-1839::DEBUG::2012-12-03 00:31:09,470::task::1151::TaskManager.Task::(prepare) Task=`7fa802a0-0413-496f-b2a1-9799d33cbc65`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-1839::DEBUG::2012-12-03 00:31:09,470::task::568::TaskManager.Task::(_updateState) Task=`7fa802a0-0413-496f-b2a1-9799d33cbc65`::moving from state preparing -> state finished >Thread-1839::DEBUG::2012-12-03 00:31:09,470::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1839::DEBUG::2012-12-03 00:31:09,471::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1839::DEBUG::2012-12-03 00:31:09,471::task::957::TaskManager.Task::(_decref) Task=`7fa802a0-0413-496f-b2a1-9799d33cbc65`::ref 0 aborting False >Thread-1840::DEBUG::2012-12-03 00:31:09,621::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1840::DEBUG::2012-12-03 00:31:09,622::task::568::TaskManager.Task::(_updateState) Task=`097e7e4c-5cb9-4fe4-9687-d692a9544aec`::moving from state init -> state preparing >Thread-1840::INFO::2012-12-03 00:31:09,622::logUtils::37::dispatcher::(wrapper) Run and protect: clearTask(taskID='f768dad4-76b2-435a-963b-91b1579c59d9', spUUID=None, options=None) >Thread-1840::DEBUG::2012-12-03 00:31:09,622::taskManager::157::TaskManager::(clearTask) Entry. taskID: f768dad4-76b2-435a-963b-91b1579c59d9 >Thread-1840::DEBUG::2012-12-03 00:31:09,622::taskManager::162::TaskManager::(clearTask) Return. >Thread-1840::INFO::2012-12-03 00:31:09,622::logUtils::39::dispatcher::(wrapper) Run and protect: clearTask, Return response: None >Thread-1840::DEBUG::2012-12-03 00:31:09,622::task::1151::TaskManager.Task::(prepare) Task=`097e7e4c-5cb9-4fe4-9687-d692a9544aec`::finished: None >Thread-1840::DEBUG::2012-12-03 00:31:09,623::task::568::TaskManager.Task::(_updateState) Task=`097e7e4c-5cb9-4fe4-9687-d692a9544aec`::moving from state preparing -> state finished >Thread-1840::DEBUG::2012-12-03 00:31:09,623::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-1840::DEBUG::2012-12-03 00:31:09,623::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1840::DEBUG::2012-12-03 00:31:09,623::task::957::TaskManager.Task::(_decref) Task=`097e7e4c-5cb9-4fe4-9687-d692a9544aec`::ref 0 aborting False >Thread-1841::DEBUG::2012-12-03 00:31:09,789::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-1841::DEBUG::2012-12-03 00:31:09,790::task::568::TaskManager.Task::(_updateState) Task=`cf515934-33d5-4068-8fb5-1846f32c615f`::moving from state init -> state preparing >Thread-1841::INFO::2012-12-03 00:31:09,790::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-1841::DEBUG::2012-12-03 00:31:09,790::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`dcf575d0-4bf3-46f4-a6a3-a26869fbfb47`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-1841::DEBUG::2012-12-03 00:31:09,790::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-1841::DEBUG::2012-12-03 00:31:09,791::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-1841::DEBUG::2012-12-03 00:31:09,791::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`dcf575d0-4bf3-46f4-a6a3-a26869fbfb47`::Granted request >Thread-1841::DEBUG::2012-12-03 00:31:09,791::task::794::TaskManager.Task::(resourceAcquired) Task=`cf515934-33d5-4068-8fb5-1846f32c615f`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-1841::DEBUG::2012-12-03 00:31:09,791::task::957::TaskManager.Task::(_decref) Task=`cf515934-33d5-4068-8fb5-1846f32c615f`::ref 1 aborting False >Thread-1841::DEBUG::2012-12-03 00:31:09,797::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1841::DEBUG::2012-12-03 00:31:09,797::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-1841::DEBUG::2012-12-03 00:31:10,003::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-1841::WARNING::2012-12-03 00:31:10,004::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-1841::DEBUG::2012-12-03 00:31:10,004::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1841::WARNING::2012-12-03 00:31:10,033::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-1841::DEBUG::2012-12-03 00:31:10,033::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1841::DEBUG::2012-12-03 00:31:10,033::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-1841::DEBUG::2012-12-03 00:31:10,239::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-1841::WARNING::2012-12-03 00:31:10,240::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-1841::DEBUG::2012-12-03 00:31:10,240::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1841::WARNING::2012-12-03 00:31:10,270::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('5971512c-ee51-475e-b956-3872c38ce193',) >Thread-1841::DEBUG::2012-12-03 00:31:10,270::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-1841::DEBUG::2012-12-03 00:31:10,271::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-1841::DEBUG::2012-12-03 00:31:10,475::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-1841::WARNING::2012-12-03 00:31:10,477::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-1841::DEBUG::2012-12-03 00:31:10,477::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1841::WARNING::2012-12-03 00:31:10,506::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-1841::INFO::2012-12-03 00:31:10,506::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 4, 'lver': 1}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-1841::DEBUG::2012-12-03 00:31:10,506::task::1151::TaskManager.Task::(prepare) Task=`cf515934-33d5-4068-8fb5-1846f32c615f`::finished: {'info': {'spm_id': 2, 'master_uuid': 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 4, 'lver': 1}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-1841::DEBUG::2012-12-03 00:31:10,507::task::568::TaskManager.Task::(_updateState) Task=`cf515934-33d5-4068-8fb5-1846f32c615f`::moving from state preparing -> state finished >Thread-1841::DEBUG::2012-12-03 00:31:10,507::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-1841::DEBUG::2012-12-03 00:31:10,507::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-1841::DEBUG::2012-12-03 00:31:10,507::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-1841::DEBUG::2012-12-03 00:31:10,507::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-1841::DEBUG::2012-12-03 00:31:10,507::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-1841::DEBUG::2012-12-03 00:31:10,507::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-1841::DEBUG::2012-12-03 00:31:10,508::task::957::TaskManager.Task::(_decref) Task=`cf515934-33d5-4068-8fb5-1846f32c615f`::ref 0 aborting False >Thread-2029::DEBUG::2012-12-03 00:31:10,766::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-2029::DEBUG::2012-12-03 00:31:10,767::task::568::TaskManager.Task::(_updateState) Task=`2139eeaa-77b8-40ed-a19e-660a60e3bc0f`::moving from state init -> state preparing >Thread-2029::INFO::2012-12-03 00:31:10,767::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-2029::DEBUG::2012-12-03 00:31:10,767::taskManager::177::TaskManager::(getAllTasksInfo) Entry. >Thread-2029::DEBUG::2012-12-03 00:31:10,767::taskManager::188::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-2029::INFO::2012-12-03 00:31:10,767::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-2029::DEBUG::2012-12-03 00:31:10,767::task::1151::TaskManager.Task::(prepare) Task=`2139eeaa-77b8-40ed-a19e-660a60e3bc0f`::finished: {'allTasksInfo': {}} >Thread-2029::DEBUG::2012-12-03 00:31:10,767::task::568::TaskManager.Task::(_updateState) Task=`2139eeaa-77b8-40ed-a19e-660a60e3bc0f`::moving from state preparing -> state finished >Thread-2029::DEBUG::2012-12-03 00:31:10,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2029::DEBUG::2012-12-03 00:31:10,768::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2029::DEBUG::2012-12-03 00:31:10,768::task::957::TaskManager.Task::(_decref) Task=`2139eeaa-77b8-40ed-a19e-660a60e3bc0f`::ref 0 aborting False >Thread-2032::DEBUG::2012-12-03 00:31:17,198::task::568::TaskManager.Task::(_updateState) Task=`647cc8f8-00e1-4752-872f-e063ea24406d`::moving from state init -> state preparing >Thread-2032::INFO::2012-12-03 00:31:17,198::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2032::INFO::2012-12-03 00:31:17,198::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00909113883972', 'lastCheck': 1354487473.3733909, 'code': 0, 'valid': True}} >Thread-2032::DEBUG::2012-12-03 00:31:17,198::task::1151::TaskManager.Task::(prepare) Task=`647cc8f8-00e1-4752-872f-e063ea24406d`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00909113883972', 'lastCheck': 1354487473.3733909, 'code': 0, 'valid': True}} >Thread-2032::DEBUG::2012-12-03 00:31:17,199::task::568::TaskManager.Task::(_updateState) Task=`647cc8f8-00e1-4752-872f-e063ea24406d`::moving from state preparing -> state finished >Thread-2032::DEBUG::2012-12-03 00:31:17,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2032::DEBUG::2012-12-03 00:31:17,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2032::DEBUG::2012-12-03 00:31:17,199::task::957::TaskManager.Task::(_decref) Task=`647cc8f8-00e1-4752-872f-e063ea24406d`::ref 0 aborting False >Thread-2035::DEBUG::2012-12-03 00:31:20,763::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-2035::DEBUG::2012-12-03 00:31:20,763::task::568::TaskManager.Task::(_updateState) Task=`9328cdcb-f25b-44e4-9ed7-6d58bd189cbc`::moving from state init -> state preparing >Thread-2035::INFO::2012-12-03 00:31:20,763::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2035::INFO::2012-12-03 00:31:20,764::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-2035::DEBUG::2012-12-03 00:31:20,764::task::1151::TaskManager.Task::(prepare) Task=`9328cdcb-f25b-44e4-9ed7-6d58bd189cbc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-2035::DEBUG::2012-12-03 00:31:20,764::task::568::TaskManager.Task::(_updateState) Task=`9328cdcb-f25b-44e4-9ed7-6d58bd189cbc`::moving from state preparing -> state finished >Thread-2035::DEBUG::2012-12-03 00:31:20,764::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2035::DEBUG::2012-12-03 00:31:20,764::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2035::DEBUG::2012-12-03 00:31:20,764::task::957::TaskManager.Task::(_decref) Task=`9328cdcb-f25b-44e4-9ed7-6d58bd189cbc`::ref 0 aborting False >Thread-2036::DEBUG::2012-12-03 00:31:20,921::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-2036::DEBUG::2012-12-03 00:31:20,921::task::568::TaskManager.Task::(_updateState) Task=`80b21580-1fbd-4686-8b2e-b69099efbf7e`::moving from state init -> state preparing >Thread-2036::INFO::2012-12-03 00:31:20,921::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2036::DEBUG::2012-12-03 00:31:20,921::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`70bc4d6a-2bee-41ab-854d-7b16af2f06ff`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2036::DEBUG::2012-12-03 00:31:20,922::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-2036::DEBUG::2012-12-03 00:31:20,922::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-2036::DEBUG::2012-12-03 00:31:20,922::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`70bc4d6a-2bee-41ab-854d-7b16af2f06ff`::Granted request >Thread-2036::DEBUG::2012-12-03 00:31:20,922::task::794::TaskManager.Task::(resourceAcquired) Task=`80b21580-1fbd-4686-8b2e-b69099efbf7e`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-2036::DEBUG::2012-12-03 00:31:20,922::task::957::TaskManager.Task::(_decref) Task=`80b21580-1fbd-4686-8b2e-b69099efbf7e`::ref 1 aborting False >Thread-2036::DEBUG::2012-12-03 00:31:20,928::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2036::DEBUG::2012-12-03 00:31:20,928::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-2036::DEBUG::2012-12-03 00:31:21,133::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-2036::WARNING::2012-12-03 00:31:21,135::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-2036::DEBUG::2012-12-03 00:31:21,135::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2036::WARNING::2012-12-03 00:31:21,164::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-2036::DEBUG::2012-12-03 00:31:21,165::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2036::DEBUG::2012-12-03 00:31:21,165::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-2036::DEBUG::2012-12-03 00:31:21,372::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-2036::WARNING::2012-12-03 00:31:21,373::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-2036::DEBUG::2012-12-03 00:31:21,373::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2036::WARNING::2012-12-03 00:31:21,401::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('5971512c-ee51-475e-b956-3872c38ce193',) >Thread-2036::DEBUG::2012-12-03 00:31:21,401::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2036::DEBUG::2012-12-03 00:31:21,402::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2036::DEBUG::2012-12-03 00:31:21,606::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2036::WARNING::2012-12-03 00:31:21,607::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2036::DEBUG::2012-12-03 00:31:21,607::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2036::WARNING::2012-12-03 00:31:21,636::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2036::INFO::2012-12-03 00:31:21,636::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 4, 'lver': 1}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-2036::DEBUG::2012-12-03 00:31:21,636::task::1151::TaskManager.Task::(prepare) Task=`80b21580-1fbd-4686-8b2e-b69099efbf7e`::finished: {'info': {'spm_id': 2, 'master_uuid': 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 4, 'lver': 1}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-2036::DEBUG::2012-12-03 00:31:21,636::task::568::TaskManager.Task::(_updateState) Task=`80b21580-1fbd-4686-8b2e-b69099efbf7e`::moving from state preparing -> state finished >Thread-2036::DEBUG::2012-12-03 00:31:21,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-2036::DEBUG::2012-12-03 00:31:21,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2036::DEBUG::2012-12-03 00:31:21,637::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-2036::DEBUG::2012-12-03 00:31:21,637::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-2036::DEBUG::2012-12-03 00:31:21,637::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-2036::DEBUG::2012-12-03 00:31:21,637::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-2036::DEBUG::2012-12-03 00:31:21,638::task::957::TaskManager.Task::(_decref) Task=`80b21580-1fbd-4686-8b2e-b69099efbf7e`::ref 0 aborting False >Thread-2226::DEBUG::2012-12-03 00:31:28,200::task::568::TaskManager.Task::(_updateState) Task=`7b7fb012-a742-4e72-b398-9221f55880fd`::moving from state init -> state preparing >Thread-2226::INFO::2012-12-03 00:31:28,200::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2226::INFO::2012-12-03 00:31:28,200::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00788688659668', 'lastCheck': 1354487483.396508, 'code': 0, 'valid': True}} >Thread-2226::DEBUG::2012-12-03 00:31:28,200::task::1151::TaskManager.Task::(prepare) Task=`7b7fb012-a742-4e72-b398-9221f55880fd`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00788688659668', 'lastCheck': 1354487483.396508, 'code': 0, 'valid': True}} >Thread-2226::DEBUG::2012-12-03 00:31:28,200::task::568::TaskManager.Task::(_updateState) Task=`7b7fb012-a742-4e72-b398-9221f55880fd`::moving from state preparing -> state finished >Thread-2226::DEBUG::2012-12-03 00:31:28,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2226::DEBUG::2012-12-03 00:31:28,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2226::DEBUG::2012-12-03 00:31:28,201::task::957::TaskManager.Task::(_decref) Task=`7b7fb012-a742-4e72-b398-9221f55880fd`::ref 0 aborting False >Thread-2229::DEBUG::2012-12-03 00:31:31,883::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-2229::DEBUG::2012-12-03 00:31:31,884::task::568::TaskManager.Task::(_updateState) Task=`1c65bbdd-e468-4027-b7ea-3dfbba85c59d`::moving from state init -> state preparing >Thread-2229::INFO::2012-12-03 00:31:31,884::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2229::INFO::2012-12-03 00:31:31,884::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-2229::DEBUG::2012-12-03 00:31:31,884::task::1151::TaskManager.Task::(prepare) Task=`1c65bbdd-e468-4027-b7ea-3dfbba85c59d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}} >Thread-2229::DEBUG::2012-12-03 00:31:31,884::task::568::TaskManager.Task::(_updateState) Task=`1c65bbdd-e468-4027-b7ea-3dfbba85c59d`::moving from state preparing -> state finished >Thread-2229::DEBUG::2012-12-03 00:31:31,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2229::DEBUG::2012-12-03 00:31:31,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2229::DEBUG::2012-12-03 00:31:31,885::task::957::TaskManager.Task::(_decref) Task=`1c65bbdd-e468-4027-b7ea-3dfbba85c59d`::ref 0 aborting False >Thread-2230::DEBUG::2012-12-03 00:31:32,026::BindingXMLRPC::160::vds::(wrapper) [10.35.202.53] >Thread-2230::DEBUG::2012-12-03 00:31:32,027::task::568::TaskManager.Task::(_updateState) Task=`7be20427-738c-4351-859c-a3118e890e24`::moving from state init -> state preparing >Thread-2230::INFO::2012-12-03 00:31:32,027::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-2230::DEBUG::2012-12-03 00:31:32,027::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b4213232-e678-4340-8d23-086974c20ff1`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-2230::DEBUG::2012-12-03 00:31:32,027::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-2230::DEBUG::2012-12-03 00:31:32,027::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-2230::DEBUG::2012-12-03 00:31:32,028::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`b4213232-e678-4340-8d23-086974c20ff1`::Granted request >Thread-2230::DEBUG::2012-12-03 00:31:32,028::task::794::TaskManager.Task::(resourceAcquired) Task=`7be20427-738c-4351-859c-a3118e890e24`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (shared) >Thread-2230::DEBUG::2012-12-03 00:31:32,028::task::957::TaskManager.Task::(_decref) Task=`7be20427-738c-4351-859c-a3118e890e24`::ref 1 aborting False >Thread-2230::DEBUG::2012-12-03 00:31:32,035::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2230::DEBUG::2012-12-03 00:31:32,035::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 225bca28-f0a0-4899-b137-32acd04f57d4' (cwd None) >Thread-2230::DEBUG::2012-12-03 00:31:32,241::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found\n'; <rc> = 5 >Thread-2230::WARNING::2012-12-03 00:31:32,243::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "225bca28-f0a0-4899-b137-32acd04f57d4" not found'] >Thread-2230::DEBUG::2012-12-03 00:31:32,243::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2230::WARNING::2012-12-03 00:31:32,278::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('225bca28-f0a0-4899-b137-32acd04f57d4',) >Thread-2230::DEBUG::2012-12-03 00:31:32,278::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2230::DEBUG::2012-12-03 00:31:32,279::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 5971512c-ee51-475e-b956-3872c38ce193' (cwd None) >Thread-2230::DEBUG::2012-12-03 00:31:32,484::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found\n'; <rc> = 5 >Thread-2230::WARNING::2012-12-03 00:31:32,486::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "5971512c-ee51-475e-b956-3872c38ce193" not found'] >Thread-2230::DEBUG::2012-12-03 00:31:32,486::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2230::WARNING::2012-12-03 00:31:32,517::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('5971512c-ee51-475e-b956-3872c38ce193',) >Thread-2230::DEBUG::2012-12-03 00:31:32,517::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-2230::DEBUG::2012-12-03 00:31:32,517::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 2ee486e4-abfd-4f11-9b29-42bfc74d7064' (cwd None) >Thread-2230::DEBUG::2012-12-03 00:31:32,725::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found\n'; <rc> = 5 >Thread-2230::WARNING::2012-12-03 00:31:32,726::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "2ee486e4-abfd-4f11-9b29-42bfc74d7064" not found'] >Thread-2230::DEBUG::2012-12-03 00:31:32,726::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2230::WARNING::2012-12-03 00:31:32,754::sp::1427::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1422, in getInfo > dom = sdCache.produce(item) > File "/usr/share/vdsm/storage/sdc.py", line 110, in produce > dom.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('2ee486e4-abfd-4f11-9b29-42bfc74d7064',) >Thread-2230::INFO::2012-12-03 00:31:32,754::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 4, 'lver': 1}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-2230::DEBUG::2012-12-03 00:31:32,754::task::1151::TaskManager.Task::(prepare) Task=`7be20427-738c-4351-859c-a3118e890e24`::finished: {'info': {'spm_id': 2, 'master_uuid': 'a00efcc1-5685-4dbe-ae70-440e9f403b76', 'name': 'dc', 'version': '3', 'domains': '225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'pool_status': 'connected', 'isoprefix': '', 'type': 'NFS', 'master_ver': 4, 'lver': 1}, 'dominfo': {'225bca28-f0a0-4899-b137-32acd04f57d4': {'status': 'Active'}, 'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'status': 'Active', 'diskfree': '45385515008', 'alerts': [], 'disktotal': '53687091200'}, '5971512c-ee51-475e-b956-3872c38ce193': {'status': 'Active'}, '2ee486e4-abfd-4f11-9b29-42bfc74d7064': {'status': 'Active'}}} >Thread-2230::DEBUG::2012-12-03 00:31:32,754::task::568::TaskManager.Task::(_updateState) Task=`7be20427-738c-4351-859c-a3118e890e24`::moving from state preparing -> state finished >Thread-2230::DEBUG::2012-12-03 00:31:32,755::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-2230::DEBUG::2012-12-03 00:31:32,755::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2230::DEBUG::2012-12-03 00:31:32,755::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-2230::DEBUG::2012-12-03 00:31:32,755::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-2230::DEBUG::2012-12-03 00:31:32,755::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-2230::DEBUG::2012-12-03 00:31:32,755::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-2230::DEBUG::2012-12-03 00:31:32,756::task::957::TaskManager.Task::(_decref) Task=`7be20427-738c-4351-859c-a3118e890e24`::ref 0 aborting False >Thread-2420::DEBUG::2012-12-03 00:31:39,150::task::568::TaskManager.Task::(_updateState) Task=`7ab4878f-ae67-4cc2-a2d7-d3469778815a`::moving from state init -> state preparing >Thread-2420::INFO::2012-12-03 00:31:39,150::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-2420::INFO::2012-12-03 00:31:39,150::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00791192054749', 'lastCheck': 1354487493.4152679, 'code': 0, 'valid': True}} >Thread-2420::DEBUG::2012-12-03 00:31:39,150::task::1151::TaskManager.Task::(prepare) Task=`7ab4878f-ae67-4cc2-a2d7-d3469778815a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00791192054749', 'lastCheck': 1354487493.4152679, 'code': 0, 'valid': True}} >Thread-2420::DEBUG::2012-12-03 00:31:39,150::task::568::TaskManager.Task::(_updateState) Task=`7ab4878f-ae67-4cc2-a2d7-d3469778815a`::moving from state preparing -> state finished >Thread-2420::DEBUG::2012-12-03 00:31:39,151::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-2420::DEBUG::2012-12-03 00:31:39,151::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-2420::DEBUG::2012-12-03 00:31:39,151::task::957::TaskManager.Task::(_decref) Task=`7ab4878f-ae67-4cc2-a2d7-d3469778815a`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 00:34:53,791::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:34:53,791::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:34:53,792::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:34:53,997::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:34:53,998::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:34:53,998::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:34:54,034::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:34:54,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:34:54,039::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 00:34:54,041::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:34:54,041::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:34:54,041::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 00:39:54,688::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:39:54,688::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:39:54,689::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:39:54,896::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:39:54,897::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:39:54,897::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:39:54,934::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:39:54,934::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:39:54,939::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 00:39:54,941::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:39:54,941::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:39:54,941::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 00:44:55,525::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:44:55,526::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:44:55,526::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:44:55,731::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:44:55,733::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:44:55,733::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:44:55,770::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:44:55,770::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:44:55,778::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 00:44:55,781::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:44:55,781::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:44:55,781::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 00:49:56,460::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:49:56,460::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:49:56,461::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:49:56,668::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:49:56,669::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:49:56,669::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:49:56,705::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:49:56,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:49:56,714::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 00:49:56,716::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:49:56,716::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:49:56,717::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 00:54:57,347::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:54:57,348::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:54:57,351::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:54:57,559::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:54:57,560::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:54:57,561::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-2726::WARNING::2012-12-03 00:54:57,586::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 00:54:57,597::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:54:57,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:54:57,607::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 00:54:57,609::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:54:57,610::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:54:57,610::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 00:59:58,224::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:59:58,224::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 00:59:58,224::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 00:59:58,437::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 00:59:58,439::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 00:59:58,439::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 00:59:58,474::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 00:59:58,475::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 00:59:58,569::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 00:59:58,572::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 00:59:58,572::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 00:59:58,572::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:04:59,214::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:04:59,215::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:04:59,215::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:04:59,439::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:04:59,442::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:04:59,442::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:04:59,478::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:04:59,478::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:04:59,483::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:04:59,485::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:04:59,485::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:04:59,485::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:10:00,089::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:10:00,089::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:10:00,089::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:10:00,301::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:10:00,302::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:10:00,303::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:10:00,342::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:10:00,342::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:10:00,351::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:10:00,353::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:10:00,354::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:10:00,354::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:15:00,930::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:15:00,930::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:15:00,930::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:15:01,146::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:15:01,147::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:15:01,148::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:15:01,184::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:15:01,185::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:15:01,193::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:15:01,196::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:15:01,196::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:15:01,197::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:20:01,748::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:20:01,748::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:20:01,748::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:20:01,960::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:20:01,961::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:20:01,962::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:20:01,998::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:20:01,998::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:20:02,003::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:20:02,005::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:20:02,005::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:20:02,005::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:25:02,599::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:25:02,599::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:25:02,600::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:25:02,807::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:25:02,808::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:25:02,809::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3083::WARNING::2012-12-03 01:25:02,827::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3085::WARNING::2012-12-03 01:25:02,829::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3086::WARNING::2012-12-03 01:25:02,829::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3088::WARNING::2012-12-03 01:25:02,830::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3089::WARNING::2012-12-03 01:25:02,831::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3091::WARNING::2012-12-03 01:25:02,832::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3092::WARNING::2012-12-03 01:25:02,833::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3094::WARNING::2012-12-03 01:25:02,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3095::WARNING::2012-12-03 01:25:02,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3097::WARNING::2012-12-03 01:25:02,835::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3098::WARNING::2012-12-03 01:25:02,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3099::WARNING::2012-12-03 01:25:02,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3100::WARNING::2012-12-03 01:25:02,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3101::WARNING::2012-12-03 01:25:02,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3102::WARNING::2012-12-03 01:25:02,838::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3103::WARNING::2012-12-03 01:25:02,838::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3104::WARNING::2012-12-03 01:25:02,839::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 01:25:02,860::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:25:02,861::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:25:03,490::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:25:03,491::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:25:03,491::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:25:03,527::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:25:03,527::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:25:03,532::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:25:03,534::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:25:03,535::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:25:03,535::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-03 01:25:03,535::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-03 01:25:03,535::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-3167::DEBUG::2012-12-03 01:25:03,536::misc::1183::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Emitting event >Thread-3167::DEBUG::2012-12-03 01:25:03,536::misc::1194::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Calling registered method `<functools.partial object at 0x7effa0411c58>` >Thread-3167::DEBUG::2012-12-03 01:25:03,536::misc::1204::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Event emitted >Thread-276::DEBUG::2012-12-03 01:25:13,556::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-3169::DEBUG::2012-12-03 01:25:13,556::misc::1183::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Emitting event >Thread-3169::DEBUG::2012-12-03 01:25:13,557::misc::1194::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Calling registered method `<functools.partial object at 0x7effa0411c58>` >Thread-3169::DEBUG::2012-12-03 01:25:13,557::misc::1204::Event.Storage.DomainMonitor.onDomainConnectivityStateChange::(_emit) Event emitted >Thread-276::DEBUG::2012-12-03 01:30:04,199::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:30:04,199::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:30:04,200::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:30:04,407::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:30:04,408::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:30:04,409::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3209::WARNING::2012-12-03 01:30:04,426::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3211::WARNING::2012-12-03 01:30:04,428::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3212::WARNING::2012-12-03 01:30:04,428::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3213::WARNING::2012-12-03 01:30:04,428::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 01:30:04,446::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:30:04,446::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:30:04,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:30:04,453::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:30:04,454::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:30:04,454::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:35:05,047::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:35:05,047::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:35:05,048::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:35:05,252::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:35:05,254::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:35:05,254::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:35:05,290::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:35:05,290::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:35:05,295::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:35:05,297::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:35:05,298::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:35:05,298::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:40:05,964::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:40:05,964::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:40:05,965::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:40:06,170::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:40:06,172::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:40:06,172::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3347::WARNING::2012-12-03 01:40:06,196::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3350::WARNING::2012-12-03 01:40:06,198::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3352::WARNING::2012-12-03 01:40:06,199::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 01:40:06,211::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:40:06,212::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:40:06,220::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:40:06,223::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:40:06,223::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:40:06,224::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:45:06,926::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:45:06,926::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:45:06,930::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:45:07,142::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:45:07,143::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:45:07,144::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:45:07,178::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:45:07,179::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:45:07,187::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:45:07,190::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:45:07,190::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:45:07,191::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:50:07,774::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:50:07,774::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:50:07,774::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:50:07,984::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:50:07,985::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:50:07,986::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:50:08,024::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:50:08,024::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:50:08,033::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:50:08,036::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:50:08,036::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:50:08,036::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 01:55:08,671::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:55:08,672::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 01:55:08,672::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 01:55:08,884::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 01:55:08,887::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 01:55:08,887::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 01:55:08,923::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 01:55:08,923::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 01:55:08,977::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 01:55:08,979::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 01:55:08,979::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 01:55:08,979::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:00:09,581::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:00:09,581::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:00:09,582::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:00:09,791::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:00:09,793::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:00:09,793::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:00:09,830::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:00:09,830::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:00:09,835::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:00:09,837::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:00:09,837::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:00:09,837::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:05:10,429::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:05:10,429::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:05:10,433::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:05:10,659::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:05:10,661::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:05:10,662::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3657::WARNING::2012-12-03 02:05:10,687::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3659::WARNING::2012-12-03 02:05:10,688::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3661::WARNING::2012-12-03 02:05:10,689::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3662::WARNING::2012-12-03 02:05:10,690::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3663::WARNING::2012-12-03 02:05:10,690::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3664::WARNING::2012-12-03 02:05:10,691::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-3665::WARNING::2012-12-03 02:05:10,692::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 02:05:10,702::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:05:10,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:05:10,709::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:05:10,712::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:05:10,712::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:05:10,712::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:10:11,332::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:10:11,333::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:10:11,333::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:10:11,550::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:10:11,552::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:10:11,552::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-3725::WARNING::2012-12-03 02:10:11,578::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 02:10:11,588::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:10:11,589::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:10:11,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:10:11,600::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:10:11,600::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:10:11,600::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:15:12,233::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:15:12,233::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:15:12,234::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:15:12,443::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:15:12,445::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:15:12,445::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:15:12,482::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:15:12,482::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:15:12,491::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:15:12,493::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:15:12,494::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:15:12,494::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:20:13,092::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:20:13,092::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:20:13,093::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:20:13,298::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:20:13,299::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:20:13,299::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:20:13,337::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:20:13,337::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:20:13,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:20:13,349::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:20:13,349::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:20:13,349::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:25:13,942::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:25:13,942::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:25:13,942::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:25:14,148::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:25:14,149::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:25:14,149::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:25:14,186::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:25:14,186::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:25:14,191::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:25:14,193::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:25:14,193::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:25:14,193::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:30:14,769::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:30:14,769::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:30:14,770::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:30:14,974::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:30:14,976::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:30:14,976::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:30:15,013::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:30:15,013::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:30:15,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:30:15,020::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:30:15,020::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:30:15,020::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:35:15,648::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:35:15,648::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:35:15,649::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:35:15,852::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:35:15,853::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:35:15,853::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4027::WARNING::2012-12-03 02:35:15,876::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4035::WARNING::2012-12-03 02:35:15,880::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4036::WARNING::2012-12-03 02:35:15,881::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 02:35:15,889::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:35:15,889::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:35:15,899::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:35:15,901::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:35:15,901::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:35:15,901::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:40:16,497::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:40:16,497::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:40:16,497::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:40:16,708::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:40:16,709::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:40:16,710::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:40:16,746::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:40:16,746::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:40:16,751::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:40:16,753::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:40:16,753::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:40:16,753::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:45:17,319::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:45:17,319::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:45:17,319::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:45:17,525::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:45:17,527::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:45:17,527::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:45:17,563::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:45:17,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:45:17,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:45:17,570::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:45:17,570::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:45:17,571::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:50:18,297::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:50:18,297::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:50:18,298::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:50:18,508::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:50:18,509::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:50:18,510::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 02:50:18,546::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:50:18,546::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:50:18,551::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:50:18,553::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:50:18,553::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:50:18,554::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 02:55:19,106::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:55:19,106::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 02:55:19,107::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 02:55:19,314::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 02:55:19,316::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 02:55:19,316::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4282::WARNING::2012-12-03 02:55:19,343::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 02:55:19,353::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 02:55:19,353::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 02:55:19,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 02:55:19,366::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 02:55:19,366::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 02:55:19,366::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:00:19,897::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:00:19,898::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:00:19,898::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:00:20,107::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:00:20,108::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:00:20,108::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:00:20,145::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:00:20,145::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:00:20,154::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:00:20,157::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:00:20,157::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:00:20,157::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:05:20,770::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:05:20,771::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:05:20,773::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:05:20,999::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:05:21,001::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:05:21,002::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4405::WARNING::2012-12-03 03:05:21,027::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 03:05:21,039::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:05:21,039::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:05:21,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:05:21,046::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:05:21,046::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:05:21,046::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:10:21,699::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:10:21,699::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:10:21,700::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:10:21,913::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:10:21,915::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:10:21,915::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4451::WARNING::2012-12-03 03:10:21,934::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4452::WARNING::2012-12-03 03:10:21,935::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4453::WARNING::2012-12-03 03:10:21,935::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4455::WARNING::2012-12-03 03:10:21,936::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4456::WARNING::2012-12-03 03:10:21,937::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4458::WARNING::2012-12-03 03:10:21,938::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4459::WARNING::2012-12-03 03:10:21,939::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4460::WARNING::2012-12-03 03:10:21,939::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4461::WARNING::2012-12-03 03:10:21,940::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4462::WARNING::2012-12-03 03:10:21,940::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4463::WARNING::2012-12-03 03:10:21,941::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4464::WARNING::2012-12-03 03:10:21,942::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 03:10:21,954::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:10:21,954::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:10:21,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:10:21,961::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:10:21,961::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:10:21,961::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:15:22,593::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:15:22,593::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:15:22,594::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:15:22,801::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:15:22,802::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:15:22,803::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:15:22,838::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:15:22,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:15:22,847::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:15:22,849::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:15:22,850::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:15:22,850::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:20:23,408::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:20:23,408::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:20:23,409::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:20:23,691::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:20:23,693::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:20:23,693::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:20:23,729::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:20:23,729::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:20:23,740::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:20:23,742::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:20:23,742::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:20:23,743::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:25:24,331::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:25:24,331::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:25:24,331::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:25:24,538::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:25:24,539::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:25:24,539::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:25:24,576::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:25:24,576::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:25:24,585::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:25:24,587::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:25:24,587::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:25:24,587::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:30:25,176::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:30:25,176::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:30:25,177::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:30:25,378::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:30:25,379::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:30:25,380::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:30:25,416::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:30:25,416::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:30:25,421::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:30:25,423::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:30:25,423::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:30:25,424::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:35:26,038::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:35:26,038::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:35:26,038::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:35:26,244::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:35:26,245::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:35:26,246::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:35:26,282::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:35:26,283::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:35:26,292::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:35:26,295::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:35:26,295::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:35:26,295::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:40:26,899::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:40:26,899::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:40:26,899::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:40:27,103::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:40:27,105::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:40:27,105::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:40:27,142::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:40:27,142::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:40:27,150::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:40:27,152::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:40:27,153::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:40:27,153::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:45:27,756::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:45:27,756::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:45:27,760::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:45:27,991::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:45:27,993::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:45:27,994::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 03:45:28,030::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:45:28,031::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:45:28,041::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:45:28,044::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:45:28,044::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:45:28,044::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:50:28,693::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:50:28,693::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:50:28,693::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:50:28,907::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:50:28,909::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:50:28,909::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-4955::WARNING::2012-12-03 03:50:28,932::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4956::WARNING::2012-12-03 03:50:28,932::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4957::WARNING::2012-12-03 03:50:28,933::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4959::WARNING::2012-12-03 03:50:28,934::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4960::WARNING::2012-12-03 03:50:28,935::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4961::WARNING::2012-12-03 03:50:28,935::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4962::WARNING::2012-12-03 03:50:28,936::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4963::WARNING::2012-12-03 03:50:28,936::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4964::WARNING::2012-12-03 03:50:28,937::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4965::WARNING::2012-12-03 03:50:28,938::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4966::WARNING::2012-12-03 03:50:28,938::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4967::WARNING::2012-12-03 03:50:28,939::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-4968::WARNING::2012-12-03 03:50:28,939::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 03:50:28,977::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:50:28,978::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:50:28,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:50:28,985::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:50:28,985::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:50:28,985::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 03:55:29,594::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:55:29,594::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 03:55:29,595::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 03:55:29,799::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 03:55:29,801::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 03:55:29,801::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5029::WARNING::2012-12-03 03:55:29,830::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 03:55:29,839::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 03:55:29,839::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 03:55:29,848::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 03:55:29,851::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 03:55:29,851::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 03:55:29,851::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:00:30,593::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:00:30,593::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:00:30,593::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:00:30,797::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:00:30,799::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:00:30,799::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:00:30,835::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:00:30,836::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:00:30,844::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:00:30,847::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:00:30,847::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:00:30,847::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:05:31,453::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:05:31,454::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:05:31,456::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:05:31,676::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:05:31,678::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:05:31,679::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:05:31,715::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:05:31,716::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:05:31,724::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:05:31,727::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:05:31,727::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:05:31,727::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:10:32,319::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:10:32,319::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:10:32,320::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:10:32,533::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:10:32,535::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:10:32,535::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:10:32,573::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:10:32,574::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:10:32,582::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:10:32,585::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:10:32,585::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:10:32,585::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:15:33,195::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:15:33,196::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:15:33,196::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:15:33,402::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:15:33,404::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:15:33,404::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:15:33,441::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:15:33,441::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:15:33,446::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:15:33,448::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:15:33,448::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:15:33,448::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:20:34,058::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:20:34,059::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:20:34,059::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:20:34,265::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:20:34,266::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:20:34,267::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:20:34,303::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:20:34,303::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:20:34,311::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:20:34,313::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:20:34,313::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:20:34,313::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:25:34,895::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:25:34,895::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:25:34,896::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:25:35,100::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:25:35,102::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:25:35,102::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:25:35,137::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:25:35,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:25:35,146::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:25:35,149::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:25:35,149::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:25:35,149::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:30:35,722::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:30:35,723::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:30:35,723::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:30:35,931::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:30:35,932::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:30:35,933::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:30:35,970::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:30:35,970::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:30:35,979::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:30:35,981::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:30:35,982::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:30:35,982::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:35:36,596::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:35:36,597::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:35:36,597::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:35:36,803::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:35:36,805::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:35:36,805::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:35:36,841::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:35:36,842::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:35:36,850::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:35:36,853::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:35:36,853::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:35:36,853::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:40:37,524::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:40:37,525::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:40:37,525::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:40:37,733::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:40:37,735::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:40:37,735::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:40:37,772::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:40:37,773::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:40:37,782::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:40:37,784::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:40:37,784::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:40:37,785::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:45:38,388::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:45:38,389::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:45:38,389::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:45:38,595::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:45:38,597::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:45:38,597::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:45:38,634::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:45:38,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:45:38,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:45:38,643::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:45:38,643::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:45:38,643::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:50:39,294::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:50:39,294::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:50:39,295::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:50:39,499::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:50:39,500::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:50:39,501::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:50:39,537::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:50:39,537::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:50:39,546::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:50:39,548::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:50:39,548::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:50:39,548::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 04:55:40,158::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:55:40,158::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 04:55:40,158::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 04:55:40,364::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 04:55:40,365::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 04:55:40,366::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 04:55:40,404::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 04:55:40,404::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 04:55:40,412::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 04:55:40,414::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 04:55:40,414::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 04:55:40,415::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:00:41,002::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:00:41,002::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:00:41,003::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:00:41,210::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:00:41,211::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:00:41,212::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-5832::WARNING::2012-12-03 05:00:41,237::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 05:00:41,250::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:00:41,250::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:00:41,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:00:41,262::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:00:41,262::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:00:41,262::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:05:41,849::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:05:41,849::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:05:41,853::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:05:42,076::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:05:42,078::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:05:42,079::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:05:42,115::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:05:42,116::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:05:42,120::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:05:42,123::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:05:42,123::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:05:42,123::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:10:42,756::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:10:42,757::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:10:42,757::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:10:42,971::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:10:42,972::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:10:42,973::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:10:43,010::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:10:43,011::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:10:43,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:10:43,022::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:10:43,022::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:10:43,022::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:15:43,706::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:15:43,706::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:15:43,707::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:15:43,912::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:15:43,913::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:15:43,914::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:15:43,947::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:15:43,948::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:15:43,952::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:15:43,954::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:15:43,954::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:15:43,955::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:20:44,580::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:20:44,580::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:20:44,581::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:20:44,784::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:20:44,786::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:20:44,786::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:20:44,823::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:20:44,824::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:20:44,832::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:20:44,835::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:20:44,835::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:20:44,835::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:25:45,419::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:25:45,420::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:25:45,420::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:25:45,629::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:25:45,630::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:25:45,631::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:25:45,666::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:25:45,667::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:25:45,671::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:25:45,674::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:25:45,674::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:25:45,674::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:30:46,305::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:30:46,306::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:30:46,306::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:30:46,511::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:30:46,512::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:30:46,513::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:30:46,549::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:30:46,550::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:30:46,558::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:30:46,561::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:30:46,561::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:30:46,561::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:35:47,235::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:35:47,236::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:35:47,236::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:35:47,439::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:35:47,440::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:35:47,441::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6261::WARNING::2012-12-03 05:35:47,464::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6267::WARNING::2012-12-03 05:35:47,467::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6268::WARNING::2012-12-03 05:35:47,467::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 05:35:47,478::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:35:47,478::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:35:47,485::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:35:47,487::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:35:47,487::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:35:47,487::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:40:48,101::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:40:48,102::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:40:48,102::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:40:48,307::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:40:48,308::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:40:48,309::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6331::WARNING::2012-12-03 05:40:48,337::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 05:40:48,346::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:40:48,346::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:40:48,355::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:40:48,357::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:40:48,358::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:40:48,358::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:45:48,982::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:45:48,983::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:45:48,983::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:45:49,195::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:45:49,197::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:45:49,197::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:45:49,232::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:45:49,232::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:45:49,237::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:45:49,239::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:45:49,239::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:45:49,240::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:50:49,812::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:50:49,812::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:50:49,813::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:50:50,021::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:50:50,022::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:50:50,023::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 05:50:50,058::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:50:50,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:50:50,063::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:50:50,066::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:50:50,066::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:50:50,066::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 05:55:50,639::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:55:50,640::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 05:55:50,643::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 05:55:50,857::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 05:55:50,859::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 05:55:50,860::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6516::WARNING::2012-12-03 05:55:50,886::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6517::WARNING::2012-12-03 05:55:50,886::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6518::WARNING::2012-12-03 05:55:50,887::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 05:55:50,899::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 05:55:50,899::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 05:55:50,914::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 05:55:50,918::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 05:55:50,918::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 05:55:50,918::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:00:51,530::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:00:51,530::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:00:51,531::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:00:51,736::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:00:51,737::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:00:51,738::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:00:51,774::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:00:51,775::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:00:51,782::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:00:51,784::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:00:51,785::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:00:51,785::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:05:52,370::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:05:52,370::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:05:52,371::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:05:52,595::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:05:52,596::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:05:52,597::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:05:52,633::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:05:52,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:05:52,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:05:52,653::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:05:52,653::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:05:52,653::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:10:53,248::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:10:53,249::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:10:53,252::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:10:53,467::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:10:53,469::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:10:53,469::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-6702::WARNING::2012-12-03 06:10:53,495::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6703::WARNING::2012-12-03 06:10:53,496::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-6704::WARNING::2012-12-03 06:10:53,496::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 06:10:53,505::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:10:53,505::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:10:53,514::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:10:53,517::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:10:53,517::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:10:53,517::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:15:54,084::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:15:54,084::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:15:54,084::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:15:54,292::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:15:54,293::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:15:54,294::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:15:54,329::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:15:54,329::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:15:54,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:15:54,340::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:15:54,340::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:15:54,340::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:20:54,936::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:20:54,937::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:20:54,937::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:20:55,149::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:20:55,151::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:20:55,151::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:20:55,186::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:20:55,187::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:20:55,191::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:20:55,193::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:20:55,194::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:20:55,194::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:25:55,776::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:25:55,776::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:25:55,777::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:25:55,984::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:25:55,986::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:25:55,987::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:25:56,024::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:25:56,024::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:25:56,032::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:25:56,035::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:25:56,035::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:25:56,035::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:30:56,614::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:30:56,614::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:30:56,614::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:30:56,820::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:30:56,822::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:30:56,822::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:30:56,857::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:30:56,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:30:56,867::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:30:56,869::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:30:56,869::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:30:56,869::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:35:57,491::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:35:57,491::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:35:57,491::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:35:57,697::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:35:57,699::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:35:57,699::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7011::WARNING::2012-12-03 06:35:57,725::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 06:35:57,735::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:35:57,736::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:35:57,740::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:35:57,743::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:35:57,743::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:35:57,743::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:40:58,371::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:40:58,371::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:40:58,371::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:40:58,576::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:40:58,577::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:40:58,578::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:40:58,615::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:40:58,615::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:40:58,624::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:40:58,626::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:40:58,627::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:40:58,627::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:45:59,239::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:45:59,239::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:45:59,240::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:45:59,446::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:45:59,448::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:45:59,448::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:45:59,484::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:45:59,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:45:59,489::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:45:59,491::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:45:59,492::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:45:59,492::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:51:00,129::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:51:00,130::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:51:00,130::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:51:00,335::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:51:00,337::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:51:00,337::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 06:51:00,374::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:51:00,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:51:00,379::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:51:00,382::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:51:00,382::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:51:00,382::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 06:56:01,041::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:56:01,041::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 06:56:01,042::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 06:56:01,248::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 06:56:01,250::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 06:56:01,250::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7259::WARNING::2012-12-03 06:56:01,276::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7260::WARNING::2012-12-03 06:56:01,276::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7262::WARNING::2012-12-03 06:56:01,277::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7261::WARNING::2012-12-03 06:56:01,277::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 06:56:01,287::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 06:56:01,287::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 06:56:01,292::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 06:56:01,294::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 06:56:01,294::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 06:56:01,295::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:01:01,966::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:01:01,967::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:01:01,967::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:01:02,189::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:01:02,190::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:01:02,191::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:01:02,227::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:01:02,227::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:01:02,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:01:02,239::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:01:02,239::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:01:02,239::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:06:02,818::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:06:02,819::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:06:02,822::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:06:03,037::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:06:03,038::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:06:03,039::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:06:03,074::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:06:03,074::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:06:03,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:06:03,086::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:06:03,086::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:06:03,086::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:11:03,704::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:11:03,704::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:11:03,705::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:11:03,919::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:11:03,920::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:11:03,921::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:11:03,957::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:11:03,957::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:11:03,965::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:11:03,968::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:11:03,968::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:11:03,968::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:16:04,589::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:16:04,589::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:16:04,589::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:16:04,794::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:16:04,796::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:16:04,796::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:16:04,833::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:16:04,834::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:16:04,842::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:16:04,844::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:16:04,844::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:16:04,844::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:21:05,460::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:21:05,460::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:21:05,460::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:21:05,665::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:21:05,667::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:21:05,667::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:21:05,704::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:21:05,704::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:21:05,713::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:21:05,716::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:21:05,716::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:21:05,716::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:26:06,329::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:26:06,329::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:26:06,330::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:26:06,533::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:26:06,535::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:26:06,535::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:26:06,572::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:26:06,572::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:26:06,578::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:26:06,580::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:26:06,580::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:26:06,580::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:31:07,162::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:31:07,162::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:31:07,162::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:31:07,369::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:31:07,370::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:31:07,371::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:31:07,407::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:31:07,407::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:31:07,412::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:31:07,414::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:31:07,414::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:31:07,414::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:36:08,014::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:36:08,014::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:36:08,015::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:36:08,218::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:36:08,220::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:36:08,220::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7756::WARNING::2012-12-03 07:36:08,248::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 07:36:08,259::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:36:08,259::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:36:08,267::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:36:08,270::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:36:08,270::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:36:08,270::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:41:08,874::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:41:08,874::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:41:08,875::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:41:09,081::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:41:09,082::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:41:09,082::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:41:09,119::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:41:09,120::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:41:09,128::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:41:09,131::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:41:09,131::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:41:09,131::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:46:09,796::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:46:09,797::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:46:09,797::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:46:10,001::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:46:10,002::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:46:10,002::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:46:10,039::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:46:10,040::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:46:10,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:46:10,051::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:46:10,051::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:46:10,051::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:51:10,686::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:51:10,686::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:51:10,687::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:51:10,891::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:51:10,893::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:51:10,893::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-7934::WARNING::2012-12-03 07:51:10,915::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7940::WARNING::2012-12-03 07:51:10,918::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-7944::WARNING::2012-12-03 07:51:10,920::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 07:51:10,929::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:51:10,930::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:51:10,938::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:51:10,941::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:51:10,941::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:51:10,941::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 07:56:11,520::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:56:11,520::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 07:56:11,521::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 07:56:11,728::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 07:56:11,730::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 07:56:11,730::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 07:56:11,766::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 07:56:11,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 07:56:11,775::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 07:56:11,778::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 07:56:11,778::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 07:56:11,778::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:01:12,386::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:01:12,386::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:01:12,386::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:01:12,603::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:01:12,605::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:01:12,605::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:01:12,641::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:01:12,642::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:01:12,651::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:01:12,654::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:01:12,654::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:01:12,654::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:06:13,371::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:06:13,372::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:06:13,375::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:06:13,587::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:06:13,589::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:06:13,589::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:06:13,624::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:06:13,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:06:13,634::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:06:13,637::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:06:13,637::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:06:13,637::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:11:14,270::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:11:14,270::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:11:14,270::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:11:14,480::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:11:14,482::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:11:14,482::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8186::WARNING::2012-12-03 08:11:14,508::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 08:11:14,517::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:11:14,518::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:11:14,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:11:14,524::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:11:14,525::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:11:14,525::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:16:15,188::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:16:15,188::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:16:15,188::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:16:15,394::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:16:15,396::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:16:15,396::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:16:15,432::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:16:15,433::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:16:15,437::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:16:15,440::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:16:15,440::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:16:15,440::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:21:16,030::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:21:16,030::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:21:16,030::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:21:16,237::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:21:16,240::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:21:16,240::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:21:16,277::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:21:16,277::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:21:16,286::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:21:16,289::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:21:16,289::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:21:16,289::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:26:16,904::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:26:16,905::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:26:16,905::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:26:17,109::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:26:17,111::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:26:17,111::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8374::WARNING::2012-12-03 08:26:17,137::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 08:26:17,147::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:26:17,147::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:26:17,158::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:26:17,160::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:26:17,160::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:26:17,161::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:31:17,730::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:31:17,731::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:31:17,731::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:31:17,935::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:31:17,936::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:31:17,937::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:31:17,973::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:31:17,973::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:31:17,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:31:17,984::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:31:17,985::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:31:17,985::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:36:18,567::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:36:18,568::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:36:18,568::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:36:18,782::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:36:18,784::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:36:18,784::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:36:18,820::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:36:18,820::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:36:18,825::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:36:18,827::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:36:18,827::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:36:18,827::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:41:19,429::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:41:19,430::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:41:19,430::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:41:19,637::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:41:19,639::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:41:19,639::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-8542::WARNING::2012-12-03 08:41:19,657::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8544::WARNING::2012-12-03 08:41:19,658::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8548::WARNING::2012-12-03 08:41:19,661::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8551::WARNING::2012-12-03 08:41:19,662::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8556::WARNING::2012-12-03 08:41:19,664::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8557::WARNING::2012-12-03 08:41:19,665::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8559::WARNING::2012-12-03 08:41:19,666::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8561::WARNING::2012-12-03 08:41:19,667::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8560::WARNING::2012-12-03 08:41:19,667::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8563::WARNING::2012-12-03 08:41:19,668::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-8564::WARNING::2012-12-03 08:41:19,669::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 08:41:19,680::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:41:19,680::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:41:19,688::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:41:19,690::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:41:19,691::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:41:19,691::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:46:20,254::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:46:20,254::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:46:20,255::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:46:20,461::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:46:20,463::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:46:20,463::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:46:20,498::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:46:20,499::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:46:20,509::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:46:20,512::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:46:20,512::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:46:20,512::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:51:21,096::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:51:21,096::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:51:21,096::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:51:21,302::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:51:21,303::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:51:21,304::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:51:21,340::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:51:21,341::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:51:21,349::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:51:21,352::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:51:21,352::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:51:21,352::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 08:56:21,944::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:56:21,944::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 08:56:21,944::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 08:56:22,151::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 08:56:22,152::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 08:56:22,152::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 08:56:22,187::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 08:56:22,188::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 08:56:22,196::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 08:56:22,199::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 08:56:22,199::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 08:56:22,199::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:01:22,764::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:01:22,764::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:01:22,767::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:01:22,987::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:01:22,989::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:01:22,990::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:01:23,026::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:01:23,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:01:23,031::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:01:23,033::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:01:23,034::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:01:23,034::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:06:23,618::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:06:23,618::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:06:23,619::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:06:23,824::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:06:23,825::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:06:23,826::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:06:23,863::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:06:23,863::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:06:23,868::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:06:23,870::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:06:23,870::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:06:23,870::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:11:24,478::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:11:24,478::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:11:24,479::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:11:24,690::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:11:24,692::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:11:24,692::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:11:24,732::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:11:24,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:11:24,741::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:11:24,744::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:11:24,744::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:11:24,744::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:16:25,351::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:16:25,351::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:16:25,352::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:16:25,555::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:16:25,557::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:16:25,557::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:16:25,594::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:16:25,594::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:16:25,598::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:16:25,600::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:16:25,601::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:16:25,601::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:21:26,290::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:21:26,290::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:21:26,291::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:21:26,495::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:21:26,496::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:21:26,497::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:21:26,533::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:21:26,533::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:21:26,542::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:21:26,544::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:21:26,545::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:21:26,545::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:26:27,148::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:26:27,148::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:26:27,149::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:26:27,352::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:26:27,354::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:26:27,354::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:26:27,391::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:26:27,391::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:26:27,400::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:26:27,402::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:26:27,403::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:26:27,403::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:31:27,975::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:31:27,976::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:31:27,976::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:31:28,179::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:31:28,180::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:31:28,180::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:31:28,216::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:31:28,216::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:31:28,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:31:28,227::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:31:28,227::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:31:28,228::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:36:28,791::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:36:28,791::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:36:28,792::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:36:28,996::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:36:28,998::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:36:28,998::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:36:29,034::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:36:29,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:36:29,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:36:29,047::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:36:29,047::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:36:29,047::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:41:29,637::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:41:29,637::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:41:29,638::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:41:29,843::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:41:29,844::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:41:29,845::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9288::WARNING::2012-12-03 09:41:29,864::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-9296::WARNING::2012-12-03 09:41:29,868::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-9297::WARNING::2012-12-03 09:41:29,868::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-9301::WARNING::2012-12-03 09:41:29,871::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-9303::WARNING::2012-12-03 09:41:29,872::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-9304::WARNING::2012-12-03 09:41:29,872::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-9305::WARNING::2012-12-03 09:41:29,873::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 09:41:29,885::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:41:29,886::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:41:29,891::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:41:29,893::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:41:29,893::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:41:29,893::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:46:30,479::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:46:30,479::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:46:30,479::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:46:30,684::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:46:30,685::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:46:30,686::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:46:30,723::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:46:30,724::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:46:30,728::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:46:30,730::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:46:30,730::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:46:30,731::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:51:31,382::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:51:31,383::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:51:31,383::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:51:31,587::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:51:31,589::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:51:31,589::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:51:31,625::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:51:31,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:51:31,630::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:51:31,632::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:51:31,632::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:51:31,633::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 09:56:32,270::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:56:32,271::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 09:56:32,274::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 09:56:32,491::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 09:56:32,493::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 09:56:32,493::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 09:56:32,530::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 09:56:32,530::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 09:56:32,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 09:56:32,538::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 09:56:32,539::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 09:56:32,539::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:01:33,142::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:01:33,142::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:01:33,143::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:01:33,363::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:01:33,365::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:01:33,366::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:01:33,401::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:01:33,402::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:01:33,410::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 10:01:33,413::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:01:33,413::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:01:33,413::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:06:34,017::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:06:34,017::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:06:34,018::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:06:34,231::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:06:34,232::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:06:34,233::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:06:34,270::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:06:34,270::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:06:34,275::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 10:06:34,277::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:06:34,277::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:06:34,277::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:11:34,918::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:11:34,918::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:11:34,921::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:11:35,133::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:11:35,135::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:11:35,137::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:11:35,173::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:11:35,173::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:11:35,182::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 10:11:35,184::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:11:35,184::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:11:35,184::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:16:35,762::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:16:35,762::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:16:35,763::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:16:35,974::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:16:35,976::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:16:35,976::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:16:36,013::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:16:36,013::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:16:36,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=6ad2b99221b32ea58302abda418654dda7fee9f0'] >Thread-276::DEBUG::2012-12-03 10:16:36,020::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:16:36,020::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:16:36,020::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-9743::DEBUG::2012-12-03 10:17:45,225::BindingXMLRPC::872::vds::(wrapper) client [10.35.202.79]::call getCapabilities with () {} >Thread-9743::DEBUG::2012-12-03 10:17:45,237::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None) >Thread-9743::DEBUG::2012-12-03 10:17:45,250::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-9743::DEBUG::2012-12-03 10:17:45,250::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None) >Thread-9743::DEBUG::2012-12-03 10:17:45,261::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ''; <rc> = 1 >Thread-9743::DEBUG::2012-12-03 10:17:45,261::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-9743::DEBUG::2012-12-03 10:17:45,273::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9743::DEBUG::2012-12-03 10:17:45,274::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-9743::DEBUG::2012-12-03 10:17:45,286::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9743::DEBUG::2012-12-03 10:17:45,286::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-9743::DEBUG::2012-12-03 10:17:45,297::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9743::DEBUG::2012-12-03 10:17:45,298::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.2.1.el6.x86_64', 'buildtime': 1341511738.0, 'version': '2.6.32'}, 'vdsm': {'release': '0.239.gitf693264.el6', 'buildtime': '1343813065', 'version': '4.10.0'}, 'spice-server': {'release': '5.el6', 'buildtime': '1333631639', 'version': '0.10.1'}, 'libvirt': {'release': '21.el6', 'buildtime': '1337779106', 'version': '0.9.10'}}, 'cpuModel': 'Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz', 'hooks': {}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'interface': 'engine', 'gateway': '0.0.0.0', 'ports': []}, 'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'bridged': True, 'interface': 'ovirtmgmt', 'gateway': '10.35.1.254', 'ports': ['em1']}}, 'bridges': {'ovirtmgmt': {'addr': '10.35.1.62', 'cfg': {'DELAY': '0', 'BOOTPROTO': 'dhcp', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.254.0', 'stp': 'off', 'ports': ['em1']}, 'engine': {'addr': '', 'cfg': {'DEVICE': 'engine', 'DELAY': '0', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'STP': 'no'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': []}}, 'uuid': '4C4C4544-0043-5010-8038-CAC04F37354A_18:03:73:cb:c1:ce', 'lastClientIface': 'ovirtmgmt', 'nics': {'em1': {'addr': '', 'cfg': {'DEVICE': 'em1', 'HWADDR': '18:03:73:CB:C1:CE', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BRIDGE': 'ovirtmgmt'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '18:03:73:cb:c1:ce', 'speed': 100}}, 'software_revision': '0.239', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_Penryn,model_Westmere,model_SandyBridge', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:27c1ec9f7eff', 'netConfigDirty': 'False', 'memSize': '7750', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '1600.000', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '6.3.0.2.el6', 'version': '6Server', 'name': 'RHEL'}, 'lastClient': '10.35.202.53'}} >Thread-9745::DEBUG::2012-12-03 10:17:45,920::BindingXMLRPC::160::vds::(wrapper) [10.35.202.79] >Thread-9745::DEBUG::2012-12-03 10:17:45,920::task::568::TaskManager.Task::(_updateState) Task=`4608bc62-00be-4809-b999-0f6c9016928b`::moving from state init -> state preparing >Thread-9745::INFO::2012-12-03 10:17:45,921::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-9745::INFO::2012-12-03 10:17:45,921::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-9745::DEBUG::2012-12-03 10:17:45,921::task::1151::TaskManager.Task::(prepare) Task=`4608bc62-00be-4809-b999-0f6c9016928b`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-9745::DEBUG::2012-12-03 10:17:45,921::task::568::TaskManager.Task::(_updateState) Task=`4608bc62-00be-4809-b999-0f6c9016928b`::moving from state preparing -> state finished >Thread-9745::DEBUG::2012-12-03 10:17:45,921::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9745::DEBUG::2012-12-03 10:17:45,921::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9745::DEBUG::2012-12-03 10:17:45,921::task::957::TaskManager.Task::(_decref) Task=`4608bc62-00be-4809-b999-0f6c9016928b`::ref 0 aborting False >Thread-9746::DEBUG::2012-12-03 10:17:46,084::BindingXMLRPC::160::vds::(wrapper) [10.35.202.79] >Thread-9746::DEBUG::2012-12-03 10:17:46,085::task::568::TaskManager.Task::(_updateState) Task=`87bf9dd3-c599-4286-ae1f-524b228663e5`::moving from state init -> state preparing >Thread-9746::INFO::2012-12-03 10:17:46,085::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='479cf195-bb99-45c5-b400-433d810eaa48', conList=[{'connection': 'multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b', 'port': ''}], options=None) >Thread-9746::DEBUG::2012-12-03 10:17:46,091::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9746::DEBUG::2012-12-03 10:17:46,091::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9746::DEBUG::2012-12-03 10:17:46,092::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9746::DEBUG::2012-12-03 10:17:46,092::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9746::DEBUG::2012-12-03 10:17:46,092::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9746::DEBUG::2012-12-03 10:17:46,092::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9746::INFO::2012-12-03 10:17:46,092::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-9746::DEBUG::2012-12-03 10:17:46,092::task::1151::TaskManager.Task::(prepare) Task=`87bf9dd3-c599-4286-ae1f-524b228663e5`::finished: {'statuslist': [{'status': 0, 'id': 'e6a956d2-a4a8-4613-a163-e52a1d262e5b'}]} >Thread-9746::DEBUG::2012-12-03 10:17:46,093::task::568::TaskManager.Task::(_updateState) Task=`87bf9dd3-c599-4286-ae1f-524b228663e5`::moving from state preparing -> state finished >Thread-9746::DEBUG::2012-12-03 10:17:46,093::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9746::DEBUG::2012-12-03 10:17:46,093::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9746::DEBUG::2012-12-03 10:17:46,093::task::957::TaskManager.Task::(_decref) Task=`87bf9dd3-c599-4286-ae1f-524b228663e5`::ref 0 aborting False >Thread-9747::DEBUG::2012-12-03 10:17:46,324::BindingXMLRPC::160::vds::(wrapper) [10.35.202.79] >Thread-9747::DEBUG::2012-12-03 10:17:46,325::task::568::TaskManager.Task::(_updateState) Task=`e360d43b-c963-48d0-870c-c37b65ff0aa2`::moving from state init -> state preparing >Thread-9747::INFO::2012-12-03 10:17:46,325::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', hostID=2, scsiKey='479cf195-bb99-45c5-b400-433d810eaa48', msdUUID='a00efcc1-5685-4dbe-ae70-440e9f403b76', masterVersion=4, options=None) >Thread-9747::DEBUG::2012-12-03 10:17:46,325::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e7f07f0e-c943-4bec-aaf3-deacd3d878a7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9747::DEBUG::2012-12-03 10:17:46,325::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'shared' >Thread-9747::DEBUG::2012-12-03 10:17:46,325::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'shared' (1 active user) >Thread-9747::DEBUG::2012-12-03 10:17:46,325::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`e7f07f0e-c943-4bec-aaf3-deacd3d878a7`::Granted request >Thread-9747::DEBUG::2012-12-03 10:17:46,326::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-9747::DEBUG::2012-12-03 10:17:46,326::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9747::DEBUG::2012-12-03 10:17:46,326::misc::1061::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-9747::DEBUG::2012-12-03 10:17:46,326::misc::1063::SamplingMethod::(__call__) Got in to sampling method >Thread-9747::DEBUG::2012-12-03 10:17:46,326::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) >Thread-9747::DEBUG::2012-12-03 10:17:46,351::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9747::DEBUG::2012-12-03 10:17:46,351::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9747::DEBUG::2012-12-03 10:17:46,984::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/multipath' (cwd None) >Thread-9747::DEBUG::2012-12-03 10:17:47,072::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0 >Thread-9747::DEBUG::2012-12-03 10:17:47,073::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9747::DEBUG::2012-12-03 10:17:47,073::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9747::DEBUG::2012-12-03 10:17:47,073::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9747::DEBUG::2012-12-03 10:17:47,073::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9747::DEBUG::2012-12-03 10:17:47,073::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-9747::DEBUG::2012-12-03 10:17:47,073::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-9747::DEBUG::2012-12-03 10:17:47,074::misc::1071::SamplingMethod::(__call__) Returning last result >Thread-9747::DEBUG::2012-12-03 10:17:47,074::sp::1519::Storage.StoragePool::(getMasterDomain) Master domain a00efcc1-5685-4dbe-ae70-440e9f403b76 verified, version 4 >Thread-9747::DEBUG::2012-12-03 10:17:47,074::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9747::DEBUG::2012-12-03 10:17:47,074::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9747::DEBUG::2012-12-03 10:17:47,074::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9747::DEBUG::2012-12-03 10:17:47,075::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9747::INFO::2012-12-03 10:17:47,075::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: None >Thread-9747::DEBUG::2012-12-03 10:17:47,075::task::1151::TaskManager.Task::(prepare) Task=`e360d43b-c963-48d0-870c-c37b65ff0aa2`::finished: None >Thread-9747::DEBUG::2012-12-03 10:17:47,075::task::568::TaskManager.Task::(_updateState) Task=`e360d43b-c963-48d0-870c-c37b65ff0aa2`::moving from state preparing -> state finished >Thread-9747::DEBUG::2012-12-03 10:17:47,075::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9747::DEBUG::2012-12-03 10:17:47,075::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9747::DEBUG::2012-12-03 10:17:47,075::task::957::TaskManager.Task::(_decref) Task=`e360d43b-c963-48d0-870c-c37b65ff0aa2`::ref 0 aborting False >Thread-9748::DEBUG::2012-12-03 10:17:47,242::task::568::TaskManager.Task::(_updateState) Task=`47674710-c772-4b1c-9d82-1a80398a7f75`::moving from state init -> state preparing >Thread-9748::INFO::2012-12-03 10:17:47,242::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9748::INFO::2012-12-03 10:17:47,243::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00752305984497', 'lastCheck': 1354522666.179229, 'code': 0, 'valid': True}} >Thread-9748::DEBUG::2012-12-03 10:17:47,243::task::1151::TaskManager.Task::(prepare) Task=`47674710-c772-4b1c-9d82-1a80398a7f75`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00752305984497', 'lastCheck': 1354522666.179229, 'code': 0, 'valid': True}} >Thread-9748::DEBUG::2012-12-03 10:17:47,243::task::568::TaskManager.Task::(_updateState) Task=`47674710-c772-4b1c-9d82-1a80398a7f75`::moving from state preparing -> state finished >Thread-9748::DEBUG::2012-12-03 10:17:47,243::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9748::DEBUG::2012-12-03 10:17:47,243::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9748::DEBUG::2012-12-03 10:17:47,243::task::957::TaskManager.Task::(_decref) Task=`47674710-c772-4b1c-9d82-1a80398a7f75`::ref 0 aborting False >Thread-9751::DEBUG::2012-12-03 10:17:53,667::BindingXMLRPC::160::vds::(wrapper) [10.35.202.79] >Thread-9751::DEBUG::2012-12-03 10:17:53,668::task::568::TaskManager.Task::(_updateState) Task=`513d9386-09ee-4906-a68b-5ec60b0c5f95`::moving from state init -> state preparing >Thread-9751::INFO::2012-12-03 10:17:53,668::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-9751::DEBUG::2012-12-03 10:17:53,668::taskManager::102::TaskManager::(getAllTasksStatuses) Entry. >Thread-9751::DEBUG::2012-12-03 10:17:53,668::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {} >Thread-9751::INFO::2012-12-03 10:17:53,668::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-9751::DEBUG::2012-12-03 10:17:53,668::task::1151::TaskManager.Task::(prepare) Task=`513d9386-09ee-4906-a68b-5ec60b0c5f95`::finished: {'allTasksStatus': {}} >Thread-9751::DEBUG::2012-12-03 10:17:53,668::task::568::TaskManager.Task::(_updateState) Task=`513d9386-09ee-4906-a68b-5ec60b0c5f95`::moving from state preparing -> state finished >Thread-9751::DEBUG::2012-12-03 10:17:53,669::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9751::DEBUG::2012-12-03 10:17:53,669::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9751::DEBUG::2012-12-03 10:17:53,669::task::957::TaskManager.Task::(_decref) Task=`513d9386-09ee-4906-a68b-5ec60b0c5f95`::ref 0 aborting False >Thread-9752::DEBUG::2012-12-03 10:17:53,756::task::568::TaskManager.Task::(_updateState) Task=`e1eb6b21-947b-4c67-a170-69978d652366`::moving from state init -> state preparing >Thread-9752::INFO::2012-12-03 10:17:53,756::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9752::INFO::2012-12-03 10:17:53,756::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00752305984497', 'lastCheck': 1354522666.179229, 'code': 0, 'valid': True}} >Thread-9752::DEBUG::2012-12-03 10:17:53,756::task::1151::TaskManager.Task::(prepare) Task=`e1eb6b21-947b-4c67-a170-69978d652366`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00752305984497', 'lastCheck': 1354522666.179229, 'code': 0, 'valid': True}} >Thread-9752::DEBUG::2012-12-03 10:17:53,756::task::568::TaskManager.Task::(_updateState) Task=`e1eb6b21-947b-4c67-a170-69978d652366`::moving from state preparing -> state finished >Thread-9752::DEBUG::2012-12-03 10:17:53,756::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9752::DEBUG::2012-12-03 10:17:53,757::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9752::DEBUG::2012-12-03 10:17:53,757::task::957::TaskManager.Task::(_decref) Task=`e1eb6b21-947b-4c67-a170-69978d652366`::ref 0 aborting False >Thread-9753::DEBUG::2012-12-03 10:17:53,823::BindingXMLRPC::160::vds::(wrapper) [10.35.202.79] >Thread-9753::DEBUG::2012-12-03 10:17:53,823::task::568::TaskManager.Task::(_updateState) Task=`b8f84a27-9a96-48a8-badb-4b1000edc3a8`::moving from state init -> state preparing >Thread-9753::INFO::2012-12-03 10:17:53,823::logUtils::37::dispatcher::(wrapper) Run and protect: spmStop(spUUID='479cf195-bb99-45c5-b400-433d810eaa48', options=None) >Thread-9753::DEBUG::2012-12-03 10:17:53,823::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`290511a6-7c3a-48d0-b8b2-85c196191ebc`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9753::DEBUG::2012-12-03 10:17:53,824::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-9753::DEBUG::2012-12-03 10:17:53,824::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-9753::DEBUG::2012-12-03 10:17:53,824::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`290511a6-7c3a-48d0-b8b2-85c196191ebc`::Granted request >Thread-9753::DEBUG::2012-12-03 10:17:53,824::task::794::TaskManager.Task::(resourceAcquired) Task=`b8f84a27-9a96-48a8-badb-4b1000edc3a8`::_resourcesAcquired: Storage.479cf195-bb99-45c5-b400-433d810eaa48 (exclusive) >Thread-9753::DEBUG::2012-12-03 10:17:53,824::task::957::TaskManager.Task::(_decref) Task=`b8f84a27-9a96-48a8-badb-4b1000edc3a8`::ref 1 aborting False >Thread-9753::DEBUG::2012-12-03 10:17:53,824::sp::315::Storage.StoragePool::(_shutDownUpgrade) Shutting down upgrade process >Thread-9753::DEBUG::2012-12-03 10:17:53,825::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`01eb1262-c1c6-451d-8657-1639d32f9ae7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9753::DEBUG::2012-12-03 10:17:53,825::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' for lock type 'exclusive' >Thread-9753::DEBUG::2012-12-03 10:17:53,825::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free. Now locking as 'exclusive' (1 active user) >Thread-9753::DEBUG::2012-12-03 10:17:53,825::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48`ReqID=`01eb1262-c1c6-451d-8657-1639d32f9ae7`::Granted request >Thread-9753::DEBUG::2012-12-03 10:17:53,825::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4`ReqID=`c4415af7-bb63-40b4-adec-71e2691ba5ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9753::DEBUG::2012-12-03 10:17:53,825::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4' for lock type 'exclusive' >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4' is free. Now locking as 'exclusive' (1 active user) >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4`ReqID=`c4415af7-bb63-40b4-adec-71e2691ba5ba`::Granted request >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4' >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4' (0 active users) >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4' is free, finding out if anyone is waiting for it. >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_225bca28-f0a0-4899-b137-32acd04f57d4', Clearing records. >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193`ReqID=`5f3e17c1-6f3d-43a4-b5b0-0ef336cf39ad`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193' for lock type 'exclusive' >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193' is free. Now locking as 'exclusive' (1 active user) >Thread-9753::DEBUG::2012-12-03 10:17:53,826::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193`ReqID=`5f3e17c1-6f3d-43a4-b5b0-0ef336cf39ad`::Granted request >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193' >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193' (0 active users) >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193' is free, finding out if anyone is waiting for it. >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5971512c-ee51-475e-b956-3872c38ce193', Clearing records. >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064`ReqID=`ee2310a5-08e1-459c-aba5-aff3466be17a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '485' at 'registerResource' >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064' for lock type 'exclusive' >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064' is free. Now locking as 'exclusive' (1 active user) >Thread-9753::DEBUG::2012-12-03 10:17:53,827::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064`ReqID=`ee2310a5-08e1-459c-aba5-aff3466be17a`::Granted request >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064' >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064' (0 active users) >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064' is free, finding out if anyone is waiting for it. >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_2ee486e4-abfd-4f11-9b29-42bfc74d7064', Clearing records. >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9753::DEBUG::2012-12-03 10:17:53,828::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9753::DEBUG::2012-12-03 10:17:53,832::sp::355::Storage.StoragePool::(cleanupMasterMount) master `/rhev/data-center/mnt/blockSD/d74f7d88-4896-443a-b319-b0c53ff9618b/master` is not mounted, skipping >Thread-9753::DEBUG::2012-12-03 10:17:53,832::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction >Thread-9753::DEBUG::2012-12-03 10:17:53,833::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes >Thread-9753::DEBUG::2012-12-03 10:17:53,833::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=-1', 'POOL_SPM_LVER=1', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=d1c00ae898046087371176db56b18be1895cf267'] >Thread-9753::DEBUG::2012-12-03 10:17:53,881::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction >Thread-9753::INFO::2012-12-03 10:17:53,881::safelease::239::SANLock::(release) Releasing cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-9753::DEBUG::2012-12-03 10:17:53,895::safelease::250::SANLock::(release) Cluster lock for domain a00efcc1-5685-4dbe-ae70-440e9f403b76 successfully released >Thread-9753::INFO::2012-12-03 10:17:53,895::logUtils::39::dispatcher::(wrapper) Run and protect: spmStop, Return response: None >Thread-9753::DEBUG::2012-12-03 10:17:53,895::task::1151::TaskManager.Task::(prepare) Task=`b8f84a27-9a96-48a8-badb-4b1000edc3a8`::finished: None >Thread-9753::DEBUG::2012-12-03 10:17:53,895::task::568::TaskManager.Task::(_updateState) Task=`b8f84a27-9a96-48a8-badb-4b1000edc3a8`::moving from state preparing -> state finished >Thread-9753::DEBUG::2012-12-03 10:17:53,895::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.479cf195-bb99-45c5-b400-433d810eaa48': < ResourceRef 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', isValid: 'True' obj: 'None'>} >Thread-9753::DEBUG::2012-12-03 10:17:53,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9753::DEBUG::2012-12-03 10:17:53,896::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' >Thread-9753::DEBUG::2012-12-03 10:17:53,896::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' (0 active users) >Thread-9753::DEBUG::2012-12-03 10:17:53,896::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48' is free, finding out if anyone is waiting for it. >Thread-9753::DEBUG::2012-12-03 10:17:53,896::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.479cf195-bb99-45c5-b400-433d810eaa48', Clearing records. >Thread-9753::DEBUG::2012-12-03 10:17:53,896::task::957::TaskManager.Task::(_decref) Task=`b8f84a27-9a96-48a8-badb-4b1000edc3a8`::ref 0 aborting False >Thread-9759::DEBUG::2012-12-03 10:18:04,796::task::568::TaskManager.Task::(_updateState) Task=`c85a77df-8834-4627-8e7a-892acc65def8`::moving from state init -> state preparing >Thread-9759::INFO::2012-12-03 10:18:04,796::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9759::INFO::2012-12-03 10:18:04,797::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00581407546997', 'lastCheck': 1354522676.196806, 'code': 0, 'valid': True}} >Thread-9759::DEBUG::2012-12-03 10:18:04,797::task::1151::TaskManager.Task::(prepare) Task=`c85a77df-8834-4627-8e7a-892acc65def8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00581407546997', 'lastCheck': 1354522676.196806, 'code': 0, 'valid': True}} >Thread-9759::DEBUG::2012-12-03 10:18:04,797::task::568::TaskManager.Task::(_updateState) Task=`c85a77df-8834-4627-8e7a-892acc65def8`::moving from state preparing -> state finished >Thread-9759::DEBUG::2012-12-03 10:18:04,797::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9759::DEBUG::2012-12-03 10:18:04,797::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9759::DEBUG::2012-12-03 10:18:04,797::task::957::TaskManager.Task::(_decref) Task=`c85a77df-8834-4627-8e7a-892acc65def8`::ref 0 aborting False >Thread-9765::DEBUG::2012-12-03 10:18:15,814::task::568::TaskManager.Task::(_updateState) Task=`f480e1fe-9c2c-455c-b73e-97db4623cfed`::moving from state init -> state preparing >Thread-9765::INFO::2012-12-03 10:18:15,814::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9765::INFO::2012-12-03 10:18:15,815::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00547099113464', 'lastCheck': 1354522686.2177429, 'code': 0, 'valid': True}} >Thread-9765::DEBUG::2012-12-03 10:18:15,815::task::1151::TaskManager.Task::(prepare) Task=`f480e1fe-9c2c-455c-b73e-97db4623cfed`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00547099113464', 'lastCheck': 1354522686.2177429, 'code': 0, 'valid': True}} >Thread-9765::DEBUG::2012-12-03 10:18:15,815::task::568::TaskManager.Task::(_updateState) Task=`f480e1fe-9c2c-455c-b73e-97db4623cfed`::moving from state preparing -> state finished >Thread-9765::DEBUG::2012-12-03 10:18:15,815::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9765::DEBUG::2012-12-03 10:18:15,815::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9765::DEBUG::2012-12-03 10:18:15,815::task::957::TaskManager.Task::(_decref) Task=`f480e1fe-9c2c-455c-b73e-97db4623cfed`::ref 0 aborting False >Thread-9771::DEBUG::2012-12-03 10:18:26,768::task::568::TaskManager.Task::(_updateState) Task=`f42d075f-be9a-4de6-aed5-7987f32a40c6`::moving from state init -> state preparing >Thread-9771::INFO::2012-12-03 10:18:26,768::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9771::INFO::2012-12-03 10:18:26,768::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00225591659546', 'lastCheck': 1354522706.2433469, 'code': 0, 'valid': True}} >Thread-9771::DEBUG::2012-12-03 10:18:26,768::task::1151::TaskManager.Task::(prepare) Task=`f42d075f-be9a-4de6-aed5-7987f32a40c6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00225591659546', 'lastCheck': 1354522706.2433469, 'code': 0, 'valid': True}} >Thread-9771::DEBUG::2012-12-03 10:18:26,768::task::568::TaskManager.Task::(_updateState) Task=`f42d075f-be9a-4de6-aed5-7987f32a40c6`::moving from state preparing -> state finished >Thread-9771::DEBUG::2012-12-03 10:18:26,768::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9771::DEBUG::2012-12-03 10:18:26,769::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9771::DEBUG::2012-12-03 10:18:26,769::task::957::TaskManager.Task::(_decref) Task=`f42d075f-be9a-4de6-aed5-7987f32a40c6`::ref 0 aborting False >Thread-9777::DEBUG::2012-12-03 10:18:37,731::task::568::TaskManager.Task::(_updateState) Task=`11533b39-05ac-4ae3-8680-de89f8640ff3`::moving from state init -> state preparing >Thread-9777::INFO::2012-12-03 10:18:37,731::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9777::INFO::2012-12-03 10:18:37,731::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00520205497742', 'lastCheck': 1354522716.257304, 'code': 0, 'valid': True}} >Thread-9777::DEBUG::2012-12-03 10:18:37,731::task::1151::TaskManager.Task::(prepare) Task=`11533b39-05ac-4ae3-8680-de89f8640ff3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00520205497742', 'lastCheck': 1354522716.257304, 'code': 0, 'valid': True}} >Thread-9777::DEBUG::2012-12-03 10:18:37,732::task::568::TaskManager.Task::(_updateState) Task=`11533b39-05ac-4ae3-8680-de89f8640ff3`::moving from state preparing -> state finished >Thread-9777::DEBUG::2012-12-03 10:18:37,732::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9777::DEBUG::2012-12-03 10:18:37,732::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9777::DEBUG::2012-12-03 10:18:37,732::task::957::TaskManager.Task::(_decref) Task=`11533b39-05ac-4ae3-8680-de89f8640ff3`::ref 0 aborting False >Thread-9783::DEBUG::2012-12-03 10:18:48,691::task::568::TaskManager.Task::(_updateState) Task=`25cd96c1-d7d9-4e50-b62e-5669f0949bf2`::moving from state init -> state preparing >Thread-9783::INFO::2012-12-03 10:18:48,691::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9783::INFO::2012-12-03 10:18:48,691::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00235509872437', 'lastCheck': 1354522726.2710891, 'code': 0, 'valid': True}} >Thread-9783::DEBUG::2012-12-03 10:18:48,691::task::1151::TaskManager.Task::(prepare) Task=`25cd96c1-d7d9-4e50-b62e-5669f0949bf2`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00235509872437', 'lastCheck': 1354522726.2710891, 'code': 0, 'valid': True}} >Thread-9783::DEBUG::2012-12-03 10:18:48,691::task::568::TaskManager.Task::(_updateState) Task=`25cd96c1-d7d9-4e50-b62e-5669f0949bf2`::moving from state preparing -> state finished >Thread-9783::DEBUG::2012-12-03 10:18:48,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9783::DEBUG::2012-12-03 10:18:48,692::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9783::DEBUG::2012-12-03 10:18:48,692::task::957::TaskManager.Task::(_decref) Task=`25cd96c1-d7d9-4e50-b62e-5669f0949bf2`::ref 0 aborting False >Thread-9789::DEBUG::2012-12-03 10:18:59,903::task::568::TaskManager.Task::(_updateState) Task=`8480e22a-853f-46fd-b952-dafa72b6a824`::moving from state init -> state preparing >Thread-9789::INFO::2012-12-03 10:18:59,903::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9789::INFO::2012-12-03 10:18:59,903::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00215601921082', 'lastCheck': 1354522736.2830551, 'code': 0, 'valid': True}} >Thread-9789::DEBUG::2012-12-03 10:18:59,903::task::1151::TaskManager.Task::(prepare) Task=`8480e22a-853f-46fd-b952-dafa72b6a824`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00215601921082', 'lastCheck': 1354522736.2830551, 'code': 0, 'valid': True}} >Thread-9789::DEBUG::2012-12-03 10:18:59,903::task::568::TaskManager.Task::(_updateState) Task=`8480e22a-853f-46fd-b952-dafa72b6a824`::moving from state preparing -> state finished >Thread-9789::DEBUG::2012-12-03 10:18:59,903::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9789::DEBUG::2012-12-03 10:18:59,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9789::DEBUG::2012-12-03 10:18:59,904::task::957::TaskManager.Task::(_decref) Task=`8480e22a-853f-46fd-b952-dafa72b6a824`::ref 0 aborting False >Thread-9795::DEBUG::2012-12-03 10:19:11,103::task::568::TaskManager.Task::(_updateState) Task=`4e8c7c2d-ba71-4fb9-a825-6d78d268e3cc`::moving from state init -> state preparing >Thread-9795::INFO::2012-12-03 10:19:11,103::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9795::INFO::2012-12-03 10:19:11,103::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00233006477356', 'lastCheck': 1354522746.3004251, 'code': 0, 'valid': True}} >Thread-9795::DEBUG::2012-12-03 10:19:11,103::task::1151::TaskManager.Task::(prepare) Task=`4e8c7c2d-ba71-4fb9-a825-6d78d268e3cc`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00233006477356', 'lastCheck': 1354522746.3004251, 'code': 0, 'valid': True}} >Thread-9795::DEBUG::2012-12-03 10:19:11,104::task::568::TaskManager.Task::(_updateState) Task=`4e8c7c2d-ba71-4fb9-a825-6d78d268e3cc`::moving from state preparing -> state finished >Thread-9795::DEBUG::2012-12-03 10:19:11,104::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9795::DEBUG::2012-12-03 10:19:11,104::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9795::DEBUG::2012-12-03 10:19:11,104::task::957::TaskManager.Task::(_decref) Task=`4e8c7c2d-ba71-4fb9-a825-6d78d268e3cc`::ref 0 aborting False >Thread-9801::DEBUG::2012-12-03 10:19:22,116::task::568::TaskManager.Task::(_updateState) Task=`4e716f6f-21d5-4a38-ab89-fac0aeaee25e`::moving from state init -> state preparing >Thread-9801::INFO::2012-12-03 10:19:22,116::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9801::INFO::2012-12-03 10:19:22,116::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00230383872986', 'lastCheck': 1354522756.3124559, 'code': 0, 'valid': True}} >Thread-9801::DEBUG::2012-12-03 10:19:22,116::task::1151::TaskManager.Task::(prepare) Task=`4e716f6f-21d5-4a38-ab89-fac0aeaee25e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00230383872986', 'lastCheck': 1354522756.3124559, 'code': 0, 'valid': True}} >Thread-9801::DEBUG::2012-12-03 10:19:22,116::task::568::TaskManager.Task::(_updateState) Task=`4e716f6f-21d5-4a38-ab89-fac0aeaee25e`::moving from state preparing -> state finished >Thread-9801::DEBUG::2012-12-03 10:19:22,116::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9801::DEBUG::2012-12-03 10:19:22,117::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9801::DEBUG::2012-12-03 10:19:22,117::task::957::TaskManager.Task::(_decref) Task=`4e716f6f-21d5-4a38-ab89-fac0aeaee25e`::ref 0 aborting False >Thread-9807::DEBUG::2012-12-03 10:19:33,125::task::568::TaskManager.Task::(_updateState) Task=`4cc014d2-145e-4d75-b0fc-e81168fdf420`::moving from state init -> state preparing >Thread-9807::INFO::2012-12-03 10:19:33,125::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9807::INFO::2012-12-03 10:19:33,126::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00237393379211', 'lastCheck': 1354522766.3252671, 'code': 0, 'valid': True}} >Thread-9807::DEBUG::2012-12-03 10:19:33,126::task::1151::TaskManager.Task::(prepare) Task=`4cc014d2-145e-4d75-b0fc-e81168fdf420`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00237393379211', 'lastCheck': 1354522766.3252671, 'code': 0, 'valid': True}} >Thread-9807::DEBUG::2012-12-03 10:19:33,126::task::568::TaskManager.Task::(_updateState) Task=`4cc014d2-145e-4d75-b0fc-e81168fdf420`::moving from state preparing -> state finished >Thread-9807::DEBUG::2012-12-03 10:19:33,126::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9807::DEBUG::2012-12-03 10:19:33,126::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9807::DEBUG::2012-12-03 10:19:33,126::task::957::TaskManager.Task::(_decref) Task=`4cc014d2-145e-4d75-b0fc-e81168fdf420`::ref 0 aborting False >Thread-9813::DEBUG::2012-12-03 10:19:44,088::task::568::TaskManager.Task::(_updateState) Task=`80dc5481-5a54-4565-a650-6dfe1c8e1a6a`::moving from state init -> state preparing >Thread-9813::INFO::2012-12-03 10:19:44,088::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9813::INFO::2012-12-03 10:19:44,088::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00483918190002', 'lastCheck': 1354522776.341397, 'code': 0, 'valid': True}} >Thread-9813::DEBUG::2012-12-03 10:19:44,088::task::1151::TaskManager.Task::(prepare) Task=`80dc5481-5a54-4565-a650-6dfe1c8e1a6a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00483918190002', 'lastCheck': 1354522776.341397, 'code': 0, 'valid': True}} >Thread-9813::DEBUG::2012-12-03 10:19:44,088::task::568::TaskManager.Task::(_updateState) Task=`80dc5481-5a54-4565-a650-6dfe1c8e1a6a`::moving from state preparing -> state finished >Thread-9813::DEBUG::2012-12-03 10:19:44,089::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9813::DEBUG::2012-12-03 10:19:44,089::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9813::DEBUG::2012-12-03 10:19:44,089::task::957::TaskManager.Task::(_decref) Task=`80dc5481-5a54-4565-a650-6dfe1c8e1a6a`::ref 0 aborting False >Thread-9819::DEBUG::2012-12-03 10:19:55,092::task::568::TaskManager.Task::(_updateState) Task=`81a2e333-6965-47c4-9d2e-3bac3eec62c4`::moving from state init -> state preparing >Thread-9819::INFO::2012-12-03 10:19:55,092::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9819::INFO::2012-12-03 10:19:55,092::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00271892547607', 'lastCheck': 1354522786.3540039, 'code': 0, 'valid': True}} >Thread-9819::DEBUG::2012-12-03 10:19:55,092::task::1151::TaskManager.Task::(prepare) Task=`81a2e333-6965-47c4-9d2e-3bac3eec62c4`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00271892547607', 'lastCheck': 1354522786.3540039, 'code': 0, 'valid': True}} >Thread-9819::DEBUG::2012-12-03 10:19:55,092::task::568::TaskManager.Task::(_updateState) Task=`81a2e333-6965-47c4-9d2e-3bac3eec62c4`::moving from state preparing -> state finished >Thread-9819::DEBUG::2012-12-03 10:19:55,092::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9819::DEBUG::2012-12-03 10:19:55,092::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9819::DEBUG::2012-12-03 10:19:55,093::task::957::TaskManager.Task::(_decref) Task=`81a2e333-6965-47c4-9d2e-3bac3eec62c4`::ref 0 aborting False >Thread-9825::DEBUG::2012-12-03 10:20:06,130::task::568::TaskManager.Task::(_updateState) Task=`6739d9a4-4886-4dac-b8ba-b8dbb44f6dda`::moving from state init -> state preparing >Thread-9825::INFO::2012-12-03 10:20:06,131::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9825::INFO::2012-12-03 10:20:06,131::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00207781791687', 'lastCheck': 1354522796.3652561, 'code': 0, 'valid': True}} >Thread-9825::DEBUG::2012-12-03 10:20:06,131::task::1151::TaskManager.Task::(prepare) Task=`6739d9a4-4886-4dac-b8ba-b8dbb44f6dda`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00207781791687', 'lastCheck': 1354522796.3652561, 'code': 0, 'valid': True}} >Thread-9825::DEBUG::2012-12-03 10:20:06,131::task::568::TaskManager.Task::(_updateState) Task=`6739d9a4-4886-4dac-b8ba-b8dbb44f6dda`::moving from state preparing -> state finished >Thread-9825::DEBUG::2012-12-03 10:20:06,131::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9825::DEBUG::2012-12-03 10:20:06,131::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9825::DEBUG::2012-12-03 10:20:06,131::task::957::TaskManager.Task::(_decref) Task=`6739d9a4-4886-4dac-b8ba-b8dbb44f6dda`::ref 0 aborting False >Thread-9831::DEBUG::2012-12-03 10:20:17,084::task::568::TaskManager.Task::(_updateState) Task=`a6141001-4d30-4b7b-8d79-84a152a04e6a`::moving from state init -> state preparing >Thread-9831::INFO::2012-12-03 10:20:17,084::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9831::INFO::2012-12-03 10:20:17,084::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00193405151367', 'lastCheck': 1354522816.39678, 'code': 0, 'valid': True}} >Thread-9831::DEBUG::2012-12-03 10:20:17,084::task::1151::TaskManager.Task::(prepare) Task=`a6141001-4d30-4b7b-8d79-84a152a04e6a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00193405151367', 'lastCheck': 1354522816.39678, 'code': 0, 'valid': True}} >Thread-9831::DEBUG::2012-12-03 10:20:17,084::task::568::TaskManager.Task::(_updateState) Task=`a6141001-4d30-4b7b-8d79-84a152a04e6a`::moving from state preparing -> state finished >Thread-9831::DEBUG::2012-12-03 10:20:17,085::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9831::DEBUG::2012-12-03 10:20:17,085::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9831::DEBUG::2012-12-03 10:20:17,085::task::957::TaskManager.Task::(_decref) Task=`a6141001-4d30-4b7b-8d79-84a152a04e6a`::ref 0 aborting False >Thread-9837::DEBUG::2012-12-03 10:20:28,029::task::568::TaskManager.Task::(_updateState) Task=`9a40e7fc-64ef-4124-9bfd-a9d477357549`::moving from state init -> state preparing >Thread-9837::INFO::2012-12-03 10:20:28,030::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9837::INFO::2012-12-03 10:20:28,030::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00234508514404', 'lastCheck': 1354522826.4100571, 'code': 0, 'valid': True}} >Thread-9837::DEBUG::2012-12-03 10:20:28,030::task::1151::TaskManager.Task::(prepare) Task=`9a40e7fc-64ef-4124-9bfd-a9d477357549`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00234508514404', 'lastCheck': 1354522826.4100571, 'code': 0, 'valid': True}} >Thread-9837::DEBUG::2012-12-03 10:20:28,030::task::568::TaskManager.Task::(_updateState) Task=`9a40e7fc-64ef-4124-9bfd-a9d477357549`::moving from state preparing -> state finished >Thread-9837::DEBUG::2012-12-03 10:20:28,030::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9837::DEBUG::2012-12-03 10:20:28,030::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9837::DEBUG::2012-12-03 10:20:28,030::task::957::TaskManager.Task::(_decref) Task=`9a40e7fc-64ef-4124-9bfd-a9d477357549`::ref 0 aborting False >Thread-9843::DEBUG::2012-12-03 10:20:38,983::task::568::TaskManager.Task::(_updateState) Task=`f8bd225b-bcbb-4c49-8257-e3d50b875af1`::moving from state init -> state preparing >Thread-9843::INFO::2012-12-03 10:20:38,983::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9843::INFO::2012-12-03 10:20:38,983::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00445485115051', 'lastCheck': 1354522836.426847, 'code': 0, 'valid': True}} >Thread-9843::DEBUG::2012-12-03 10:20:38,984::task::1151::TaskManager.Task::(prepare) Task=`f8bd225b-bcbb-4c49-8257-e3d50b875af1`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00445485115051', 'lastCheck': 1354522836.426847, 'code': 0, 'valid': True}} >Thread-9843::DEBUG::2012-12-03 10:20:38,984::task::568::TaskManager.Task::(_updateState) Task=`f8bd225b-bcbb-4c49-8257-e3d50b875af1`::moving from state preparing -> state finished >Thread-9843::DEBUG::2012-12-03 10:20:38,984::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9843::DEBUG::2012-12-03 10:20:38,984::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9843::DEBUG::2012-12-03 10:20:38,984::task::957::TaskManager.Task::(_decref) Task=`f8bd225b-bcbb-4c49-8257-e3d50b875af1`::ref 0 aborting False >Thread-9849::DEBUG::2012-12-03 10:20:50,061::task::568::TaskManager.Task::(_updateState) Task=`6669e36d-e906-47c2-8c63-3af24646b549`::moving from state init -> state preparing >Thread-9849::INFO::2012-12-03 10:20:50,061::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9849::INFO::2012-12-03 10:20:50,061::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00218105316162', 'lastCheck': 1354522846.439271, 'code': 0, 'valid': True}} >Thread-9849::DEBUG::2012-12-03 10:20:50,061::task::1151::TaskManager.Task::(prepare) Task=`6669e36d-e906-47c2-8c63-3af24646b549`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00218105316162', 'lastCheck': 1354522846.439271, 'code': 0, 'valid': True}} >Thread-9849::DEBUG::2012-12-03 10:20:50,061::task::568::TaskManager.Task::(_updateState) Task=`6669e36d-e906-47c2-8c63-3af24646b549`::moving from state preparing -> state finished >Thread-9849::DEBUG::2012-12-03 10:20:50,062::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9849::DEBUG::2012-12-03 10:20:50,062::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9849::DEBUG::2012-12-03 10:20:50,062::task::957::TaskManager.Task::(_decref) Task=`6669e36d-e906-47c2-8c63-3af24646b549`::ref 0 aborting False >Thread-9855::DEBUG::2012-12-03 10:21:01,066::task::568::TaskManager.Task::(_updateState) Task=`7b8dbd82-c2ae-4d26-ada5-fddfa6c01ed1`::moving from state init -> state preparing >Thread-9855::INFO::2012-12-03 10:21:01,066::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9855::INFO::2012-12-03 10:21:01,066::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00246477127075', 'lastCheck': 1354522856.4522359, 'code': 0, 'valid': True}} >Thread-9855::DEBUG::2012-12-03 10:21:01,067::task::1151::TaskManager.Task::(prepare) Task=`7b8dbd82-c2ae-4d26-ada5-fddfa6c01ed1`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00246477127075', 'lastCheck': 1354522856.4522359, 'code': 0, 'valid': True}} >Thread-9855::DEBUG::2012-12-03 10:21:01,067::task::568::TaskManager.Task::(_updateState) Task=`7b8dbd82-c2ae-4d26-ada5-fddfa6c01ed1`::moving from state preparing -> state finished >Thread-9855::DEBUG::2012-12-03 10:21:01,067::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9855::DEBUG::2012-12-03 10:21:01,067::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9855::DEBUG::2012-12-03 10:21:01,067::task::957::TaskManager.Task::(_decref) Task=`7b8dbd82-c2ae-4d26-ada5-fddfa6c01ed1`::ref 0 aborting False >Thread-9861::DEBUG::2012-12-03 10:21:12,032::task::568::TaskManager.Task::(_updateState) Task=`d9f2812d-4f18-4061-983b-00e4bd06dab5`::moving from state init -> state preparing >Thread-9861::INFO::2012-12-03 10:21:12,032::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9861::INFO::2012-12-03 10:21:12,032::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00142002105713', 'lastCheck': 1354522866.5416241, 'code': 0, 'valid': True}} >Thread-9861::DEBUG::2012-12-03 10:21:12,033::task::1151::TaskManager.Task::(prepare) Task=`d9f2812d-4f18-4061-983b-00e4bd06dab5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00142002105713', 'lastCheck': 1354522866.5416241, 'code': 0, 'valid': True}} >Thread-9861::DEBUG::2012-12-03 10:21:12,033::task::568::TaskManager.Task::(_updateState) Task=`d9f2812d-4f18-4061-983b-00e4bd06dab5`::moving from state preparing -> state finished >Thread-9861::DEBUG::2012-12-03 10:21:12,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9861::DEBUG::2012-12-03 10:21:12,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9861::DEBUG::2012-12-03 10:21:12,033::task::957::TaskManager.Task::(_decref) Task=`d9f2812d-4f18-4061-983b-00e4bd06dab5`::ref 0 aborting False >Thread-9867::DEBUG::2012-12-03 10:21:22,999::task::568::TaskManager.Task::(_updateState) Task=`c294be2e-288b-4515-b244-da7d946737c4`::moving from state init -> state preparing >Thread-9867::INFO::2012-12-03 10:21:22,999::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9867::INFO::2012-12-03 10:21:22,999::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00266695022583', 'lastCheck': 1354522876.55439, 'code': 0, 'valid': True}} >Thread-9867::DEBUG::2012-12-03 10:21:22,999::task::1151::TaskManager.Task::(prepare) Task=`c294be2e-288b-4515-b244-da7d946737c4`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00266695022583', 'lastCheck': 1354522876.55439, 'code': 0, 'valid': True}} >Thread-9867::DEBUG::2012-12-03 10:21:22,999::task::568::TaskManager.Task::(_updateState) Task=`c294be2e-288b-4515-b244-da7d946737c4`::moving from state preparing -> state finished >Thread-9867::DEBUG::2012-12-03 10:21:23,000::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9867::DEBUG::2012-12-03 10:21:23,000::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9867::DEBUG::2012-12-03 10:21:23,000::task::957::TaskManager.Task::(_decref) Task=`c294be2e-288b-4515-b244-da7d946737c4`::ref 0 aborting False >Thread-9873::DEBUG::2012-12-03 10:21:33,987::task::568::TaskManager.Task::(_updateState) Task=`b07c9383-76f7-4377-bf86-7b1d31176396`::moving from state init -> state preparing >Thread-9873::INFO::2012-12-03 10:21:33,987::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9873::INFO::2012-12-03 10:21:33,988::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00273299217224', 'lastCheck': 1354522886.565753, 'code': 0, 'valid': True}} >Thread-9873::DEBUG::2012-12-03 10:21:33,988::task::1151::TaskManager.Task::(prepare) Task=`b07c9383-76f7-4377-bf86-7b1d31176396`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00273299217224', 'lastCheck': 1354522886.565753, 'code': 0, 'valid': True}} >Thread-9873::DEBUG::2012-12-03 10:21:33,988::task::568::TaskManager.Task::(_updateState) Task=`b07c9383-76f7-4377-bf86-7b1d31176396`::moving from state preparing -> state finished >Thread-9873::DEBUG::2012-12-03 10:21:33,988::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9873::DEBUG::2012-12-03 10:21:33,988::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9873::DEBUG::2012-12-03 10:21:33,988::task::957::TaskManager.Task::(_decref) Task=`b07c9383-76f7-4377-bf86-7b1d31176396`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 10:21:36,565::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:21:36,566::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:21:36,568::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:21:36,785::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:21:36,787::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:21:36,788::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-9935::WARNING::2012-12-03 10:21:36,815::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 10:21:36,825::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:21:36,825::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:21:36,829::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:21:36,831::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:21:36,832::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:21:36,832::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-9941::DEBUG::2012-12-03 10:21:44,943::task::568::TaskManager.Task::(_updateState) Task=`51a6e38a-3f2e-41da-b85f-7a2ea1d27320`::moving from state init -> state preparing >Thread-9941::INFO::2012-12-03 10:21:44,943::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9941::INFO::2012-12-03 10:21:44,943::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00131487846375', 'lastCheck': 1354522896.8442869, 'code': 0, 'valid': True}} >Thread-9941::DEBUG::2012-12-03 10:21:44,944::task::1151::TaskManager.Task::(prepare) Task=`51a6e38a-3f2e-41da-b85f-7a2ea1d27320`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00131487846375', 'lastCheck': 1354522896.8442869, 'code': 0, 'valid': True}} >Thread-9941::DEBUG::2012-12-03 10:21:44,944::task::568::TaskManager.Task::(_updateState) Task=`51a6e38a-3f2e-41da-b85f-7a2ea1d27320`::moving from state preparing -> state finished >Thread-9941::DEBUG::2012-12-03 10:21:44,944::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9941::DEBUG::2012-12-03 10:21:44,944::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9941::DEBUG::2012-12-03 10:21:44,944::task::957::TaskManager.Task::(_decref) Task=`51a6e38a-3f2e-41da-b85f-7a2ea1d27320`::ref 0 aborting False >Thread-9947::DEBUG::2012-12-03 10:21:56,067::task::568::TaskManager.Task::(_updateState) Task=`673435a1-dff4-46fa-b8e3-c6f63d5a9cf4`::moving from state init -> state preparing >Thread-9947::INFO::2012-12-03 10:21:56,068::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9947::INFO::2012-12-03 10:21:56,068::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00198006629944', 'lastCheck': 1354522906.8548319, 'code': 0, 'valid': True}} >Thread-9947::DEBUG::2012-12-03 10:21:56,068::task::1151::TaskManager.Task::(prepare) Task=`673435a1-dff4-46fa-b8e3-c6f63d5a9cf4`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00198006629944', 'lastCheck': 1354522906.8548319, 'code': 0, 'valid': True}} >Thread-9947::DEBUG::2012-12-03 10:21:56,068::task::568::TaskManager.Task::(_updateState) Task=`673435a1-dff4-46fa-b8e3-c6f63d5a9cf4`::moving from state preparing -> state finished >Thread-9947::DEBUG::2012-12-03 10:21:56,068::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9947::DEBUG::2012-12-03 10:21:56,068::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9947::DEBUG::2012-12-03 10:21:56,068::task::957::TaskManager.Task::(_decref) Task=`673435a1-dff4-46fa-b8e3-c6f63d5a9cf4`::ref 0 aborting False >Thread-9953::DEBUG::2012-12-03 10:22:07,419::task::568::TaskManager.Task::(_updateState) Task=`8f593340-92cf-4d91-a6c1-0a7dffae2296`::moving from state init -> state preparing >Thread-9953::INFO::2012-12-03 10:22:07,419::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9953::INFO::2012-12-03 10:22:07,419::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00149202346802', 'lastCheck': 1354522926.9013391, 'code': 0, 'valid': True}} >Thread-9953::DEBUG::2012-12-03 10:22:07,419::task::1151::TaskManager.Task::(prepare) Task=`8f593340-92cf-4d91-a6c1-0a7dffae2296`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00149202346802', 'lastCheck': 1354522926.9013391, 'code': 0, 'valid': True}} >Thread-9953::DEBUG::2012-12-03 10:22:07,419::task::568::TaskManager.Task::(_updateState) Task=`8f593340-92cf-4d91-a6c1-0a7dffae2296`::moving from state preparing -> state finished >Thread-9953::DEBUG::2012-12-03 10:22:07,419::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9953::DEBUG::2012-12-03 10:22:07,419::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9953::DEBUG::2012-12-03 10:22:07,420::task::957::TaskManager.Task::(_decref) Task=`8f593340-92cf-4d91-a6c1-0a7dffae2296`::ref 0 aborting False >Thread-9959::DEBUG::2012-12-03 10:22:18,578::task::568::TaskManager.Task::(_updateState) Task=`559e1854-221a-4d79-8f2b-d530f54f55f3`::moving from state init -> state preparing >Thread-9959::INFO::2012-12-03 10:22:18,578::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9959::INFO::2012-12-03 10:22:18,578::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00292992591858', 'lastCheck': 1354522936.913981, 'code': 0, 'valid': True}} >Thread-9959::DEBUG::2012-12-03 10:22:18,578::task::1151::TaskManager.Task::(prepare) Task=`559e1854-221a-4d79-8f2b-d530f54f55f3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00292992591858', 'lastCheck': 1354522936.913981, 'code': 0, 'valid': True}} >Thread-9959::DEBUG::2012-12-03 10:22:18,579::task::568::TaskManager.Task::(_updateState) Task=`559e1854-221a-4d79-8f2b-d530f54f55f3`::moving from state preparing -> state finished >Thread-9959::DEBUG::2012-12-03 10:22:18,579::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9959::DEBUG::2012-12-03 10:22:18,579::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9959::DEBUG::2012-12-03 10:22:18,579::task::957::TaskManager.Task::(_decref) Task=`559e1854-221a-4d79-8f2b-d530f54f55f3`::ref 0 aborting False >Thread-9965::DEBUG::2012-12-03 10:22:29,541::task::568::TaskManager.Task::(_updateState) Task=`6d19276c-0f03-416e-ac78-f3483c51a4d5`::moving from state init -> state preparing >Thread-9965::INFO::2012-12-03 10:22:29,542::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9965::INFO::2012-12-03 10:22:29,542::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00202703475952', 'lastCheck': 1354522946.927094, 'code': 0, 'valid': True}} >Thread-9965::DEBUG::2012-12-03 10:22:29,542::task::1151::TaskManager.Task::(prepare) Task=`6d19276c-0f03-416e-ac78-f3483c51a4d5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00202703475952', 'lastCheck': 1354522946.927094, 'code': 0, 'valid': True}} >Thread-9965::DEBUG::2012-12-03 10:22:29,542::task::568::TaskManager.Task::(_updateState) Task=`6d19276c-0f03-416e-ac78-f3483c51a4d5`::moving from state preparing -> state finished >Thread-9965::DEBUG::2012-12-03 10:22:29,542::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9965::DEBUG::2012-12-03 10:22:29,542::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9965::DEBUG::2012-12-03 10:22:29,542::task::957::TaskManager.Task::(_decref) Task=`6d19276c-0f03-416e-ac78-f3483c51a4d5`::ref 0 aborting False >Thread-9971::DEBUG::2012-12-03 10:22:40,509::task::568::TaskManager.Task::(_updateState) Task=`5117ed87-204e-4d13-8a2d-f386f2f700c8`::moving from state init -> state preparing >Thread-9971::INFO::2012-12-03 10:22:40,509::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9971::INFO::2012-12-03 10:22:40,509::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00425100326538', 'lastCheck': 1354522956.9450531, 'code': 0, 'valid': True}} >Thread-9971::DEBUG::2012-12-03 10:22:40,509::task::1151::TaskManager.Task::(prepare) Task=`5117ed87-204e-4d13-8a2d-f386f2f700c8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00425100326538', 'lastCheck': 1354522956.9450531, 'code': 0, 'valid': True}} >Thread-9971::DEBUG::2012-12-03 10:22:40,510::task::568::TaskManager.Task::(_updateState) Task=`5117ed87-204e-4d13-8a2d-f386f2f700c8`::moving from state preparing -> state finished >Thread-9971::DEBUG::2012-12-03 10:22:40,510::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9971::DEBUG::2012-12-03 10:22:40,510::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9971::DEBUG::2012-12-03 10:22:40,510::task::957::TaskManager.Task::(_decref) Task=`5117ed87-204e-4d13-8a2d-f386f2f700c8`::ref 0 aborting False >Thread-9977::DEBUG::2012-12-03 10:22:51,467::task::568::TaskManager.Task::(_updateState) Task=`96280154-239b-41e2-88a4-3edc39d8e568`::moving from state init -> state preparing >Thread-9977::INFO::2012-12-03 10:22:51,467::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9977::INFO::2012-12-03 10:22:51,467::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00184392929077', 'lastCheck': 1354522966.9570401, 'code': 0, 'valid': True}} >Thread-9977::DEBUG::2012-12-03 10:22:51,467::task::1151::TaskManager.Task::(prepare) Task=`96280154-239b-41e2-88a4-3edc39d8e568`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00184392929077', 'lastCheck': 1354522966.9570401, 'code': 0, 'valid': True}} >Thread-9977::DEBUG::2012-12-03 10:22:51,467::task::568::TaskManager.Task::(_updateState) Task=`96280154-239b-41e2-88a4-3edc39d8e568`::moving from state preparing -> state finished >Thread-9977::DEBUG::2012-12-03 10:22:51,467::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9977::DEBUG::2012-12-03 10:22:51,468::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9977::DEBUG::2012-12-03 10:22:51,468::task::957::TaskManager.Task::(_decref) Task=`96280154-239b-41e2-88a4-3edc39d8e568`::ref 0 aborting False >Thread-9983::DEBUG::2012-12-03 10:23:02,449::task::568::TaskManager.Task::(_updateState) Task=`347d0184-5305-4ac8-abd7-52157f9da020`::moving from state init -> state preparing >Thread-9983::INFO::2012-12-03 10:23:02,449::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9983::INFO::2012-12-03 10:23:02,449::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00158309936523', 'lastCheck': 1354522976.9687181, 'code': 0, 'valid': True}} >Thread-9983::DEBUG::2012-12-03 10:23:02,449::task::1151::TaskManager.Task::(prepare) Task=`347d0184-5305-4ac8-abd7-52157f9da020`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00158309936523', 'lastCheck': 1354522976.9687181, 'code': 0, 'valid': True}} >Thread-9983::DEBUG::2012-12-03 10:23:02,450::task::568::TaskManager.Task::(_updateState) Task=`347d0184-5305-4ac8-abd7-52157f9da020`::moving from state preparing -> state finished >Thread-9983::DEBUG::2012-12-03 10:23:02,450::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9983::DEBUG::2012-12-03 10:23:02,450::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9983::DEBUG::2012-12-03 10:23:02,450::task::957::TaskManager.Task::(_decref) Task=`347d0184-5305-4ac8-abd7-52157f9da020`::ref 0 aborting False >Thread-9989::DEBUG::2012-12-03 10:23:13,463::task::568::TaskManager.Task::(_updateState) Task=`218f5a8c-85bd-46db-8f17-08b4113d9e48`::moving from state init -> state preparing >Thread-9989::INFO::2012-12-03 10:23:13,463::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9989::INFO::2012-12-03 10:23:13,463::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00188779830933', 'lastCheck': 1354522986.987267, 'code': 0, 'valid': True}} >Thread-9989::DEBUG::2012-12-03 10:23:13,463::task::1151::TaskManager.Task::(prepare) Task=`218f5a8c-85bd-46db-8f17-08b4113d9e48`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00188779830933', 'lastCheck': 1354522986.987267, 'code': 0, 'valid': True}} >Thread-9989::DEBUG::2012-12-03 10:23:13,464::task::568::TaskManager.Task::(_updateState) Task=`218f5a8c-85bd-46db-8f17-08b4113d9e48`::moving from state preparing -> state finished >Thread-9989::DEBUG::2012-12-03 10:23:13,464::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9989::DEBUG::2012-12-03 10:23:13,464::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9989::DEBUG::2012-12-03 10:23:13,464::task::957::TaskManager.Task::(_decref) Task=`218f5a8c-85bd-46db-8f17-08b4113d9e48`::ref 0 aborting False >Thread-9995::DEBUG::2012-12-03 10:23:24,437::task::568::TaskManager.Task::(_updateState) Task=`0462a5d8-c4b7-4354-b81a-cc53200c0d9a`::moving from state init -> state preparing >Thread-9995::INFO::2012-12-03 10:23:24,438::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-9995::INFO::2012-12-03 10:23:24,438::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00297999382019', 'lastCheck': 1354522997.0009611, 'code': 0, 'valid': True}} >Thread-9995::DEBUG::2012-12-03 10:23:24,438::task::1151::TaskManager.Task::(prepare) Task=`0462a5d8-c4b7-4354-b81a-cc53200c0d9a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00297999382019', 'lastCheck': 1354522997.0009611, 'code': 0, 'valid': True}} >Thread-9995::DEBUG::2012-12-03 10:23:24,438::task::568::TaskManager.Task::(_updateState) Task=`0462a5d8-c4b7-4354-b81a-cc53200c0d9a`::moving from state preparing -> state finished >Thread-9995::DEBUG::2012-12-03 10:23:24,438::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-9995::DEBUG::2012-12-03 10:23:24,438::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-9995::DEBUG::2012-12-03 10:23:24,438::task::957::TaskManager.Task::(_decref) Task=`0462a5d8-c4b7-4354-b81a-cc53200c0d9a`::ref 0 aborting False >Thread-10001::DEBUG::2012-12-03 10:23:35,407::task::568::TaskManager.Task::(_updateState) Task=`a1948935-8966-4baa-b117-e8f62409f305`::moving from state init -> state preparing >Thread-10001::INFO::2012-12-03 10:23:35,407::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10001::INFO::2012-12-03 10:23:35,407::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00275802612305', 'lastCheck': 1354523007.0153699, 'code': 0, 'valid': True}} >Thread-10001::DEBUG::2012-12-03 10:23:35,407::task::1151::TaskManager.Task::(prepare) Task=`a1948935-8966-4baa-b117-e8f62409f305`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00275802612305', 'lastCheck': 1354523007.0153699, 'code': 0, 'valid': True}} >Thread-10001::DEBUG::2012-12-03 10:23:35,408::task::568::TaskManager.Task::(_updateState) Task=`a1948935-8966-4baa-b117-e8f62409f305`::moving from state preparing -> state finished >Thread-10001::DEBUG::2012-12-03 10:23:35,408::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10001::DEBUG::2012-12-03 10:23:35,408::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10001::DEBUG::2012-12-03 10:23:35,408::task::957::TaskManager.Task::(_decref) Task=`a1948935-8966-4baa-b117-e8f62409f305`::ref 0 aborting False >Thread-10007::DEBUG::2012-12-03 10:23:46,368::task::568::TaskManager.Task::(_updateState) Task=`788ae2c5-1cb1-42b5-af8e-1586d3e3cf16`::moving from state init -> state preparing >Thread-10007::INFO::2012-12-03 10:23:46,369::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10007::INFO::2012-12-03 10:23:46,369::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00598788261414', 'lastCheck': 1354523017.033282, 'code': 0, 'valid': True}} >Thread-10007::DEBUG::2012-12-03 10:23:46,369::task::1151::TaskManager.Task::(prepare) Task=`788ae2c5-1cb1-42b5-af8e-1586d3e3cf16`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00598788261414', 'lastCheck': 1354523017.033282, 'code': 0, 'valid': True}} >Thread-10007::DEBUG::2012-12-03 10:23:46,369::task::568::TaskManager.Task::(_updateState) Task=`788ae2c5-1cb1-42b5-af8e-1586d3e3cf16`::moving from state preparing -> state finished >Thread-10007::DEBUG::2012-12-03 10:23:46,369::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10007::DEBUG::2012-12-03 10:23:46,369::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10007::DEBUG::2012-12-03 10:23:46,370::task::957::TaskManager.Task::(_decref) Task=`788ae2c5-1cb1-42b5-af8e-1586d3e3cf16`::ref 0 aborting False >Thread-10013::DEBUG::2012-12-03 10:23:57,379::task::568::TaskManager.Task::(_updateState) Task=`e7cd1153-97ae-42ba-b236-f1eaa911dabe`::moving from state init -> state preparing >Thread-10013::INFO::2012-12-03 10:23:57,380::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10013::INFO::2012-12-03 10:23:57,380::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00213503837585', 'lastCheck': 1354523037.0610011, 'code': 0, 'valid': True}} >Thread-10013::DEBUG::2012-12-03 10:23:57,380::task::1151::TaskManager.Task::(prepare) Task=`e7cd1153-97ae-42ba-b236-f1eaa911dabe`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00213503837585', 'lastCheck': 1354523037.0610011, 'code': 0, 'valid': True}} >Thread-10013::DEBUG::2012-12-03 10:23:57,380::task::568::TaskManager.Task::(_updateState) Task=`e7cd1153-97ae-42ba-b236-f1eaa911dabe`::moving from state preparing -> state finished >Thread-10013::DEBUG::2012-12-03 10:23:57,380::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10013::DEBUG::2012-12-03 10:23:57,380::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10013::DEBUG::2012-12-03 10:23:57,380::task::957::TaskManager.Task::(_decref) Task=`e7cd1153-97ae-42ba-b236-f1eaa911dabe`::ref 0 aborting False >Thread-10019::DEBUG::2012-12-03 10:24:08,465::task::568::TaskManager.Task::(_updateState) Task=`ce2e3f74-a9a5-4249-8ad4-c1e432526085`::moving from state init -> state preparing >Thread-10019::INFO::2012-12-03 10:24:08,465::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10019::INFO::2012-12-03 10:24:08,465::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00189900398254', 'lastCheck': 1354523047.076839, 'code': 0, 'valid': True}} >Thread-10019::DEBUG::2012-12-03 10:24:08,466::task::1151::TaskManager.Task::(prepare) Task=`ce2e3f74-a9a5-4249-8ad4-c1e432526085`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00189900398254', 'lastCheck': 1354523047.076839, 'code': 0, 'valid': True}} >Thread-10019::DEBUG::2012-12-03 10:24:08,466::task::568::TaskManager.Task::(_updateState) Task=`ce2e3f74-a9a5-4249-8ad4-c1e432526085`::moving from state preparing -> state finished >Thread-10019::DEBUG::2012-12-03 10:24:08,466::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10019::DEBUG::2012-12-03 10:24:08,466::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10019::DEBUG::2012-12-03 10:24:08,466::task::957::TaskManager.Task::(_decref) Task=`ce2e3f74-a9a5-4249-8ad4-c1e432526085`::ref 0 aborting False >Thread-10025::DEBUG::2012-12-03 10:24:19,484::task::568::TaskManager.Task::(_updateState) Task=`c3fab13e-9d4b-438e-a04e-be121212241b`::moving from state init -> state preparing >Thread-10025::INFO::2012-12-03 10:24:19,484::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10025::INFO::2012-12-03 10:24:19,484::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00184392929077', 'lastCheck': 1354523057.088882, 'code': 0, 'valid': True}} >Thread-10025::DEBUG::2012-12-03 10:24:19,484::task::1151::TaskManager.Task::(prepare) Task=`c3fab13e-9d4b-438e-a04e-be121212241b`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00184392929077', 'lastCheck': 1354523057.088882, 'code': 0, 'valid': True}} >Thread-10025::DEBUG::2012-12-03 10:24:19,484::task::568::TaskManager.Task::(_updateState) Task=`c3fab13e-9d4b-438e-a04e-be121212241b`::moving from state preparing -> state finished >Thread-10025::DEBUG::2012-12-03 10:24:19,485::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10025::DEBUG::2012-12-03 10:24:19,485::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10025::DEBUG::2012-12-03 10:24:19,485::task::957::TaskManager.Task::(_decref) Task=`c3fab13e-9d4b-438e-a04e-be121212241b`::ref 0 aborting False >Thread-10031::DEBUG::2012-12-03 10:24:30,429::task::568::TaskManager.Task::(_updateState) Task=`1c19e02f-b7e5-4650-a115-7003a27413d6`::moving from state init -> state preparing >Thread-10031::INFO::2012-12-03 10:24:30,429::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10031::INFO::2012-12-03 10:24:30,429::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00230288505554', 'lastCheck': 1354523067.101573, 'code': 0, 'valid': True}} >Thread-10031::DEBUG::2012-12-03 10:24:30,430::task::1151::TaskManager.Task::(prepare) Task=`1c19e02f-b7e5-4650-a115-7003a27413d6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00230288505554', 'lastCheck': 1354523067.101573, 'code': 0, 'valid': True}} >Thread-10031::DEBUG::2012-12-03 10:24:30,430::task::568::TaskManager.Task::(_updateState) Task=`1c19e02f-b7e5-4650-a115-7003a27413d6`::moving from state preparing -> state finished >Thread-10031::DEBUG::2012-12-03 10:24:30,430::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10031::DEBUG::2012-12-03 10:24:30,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10031::DEBUG::2012-12-03 10:24:30,430::task::957::TaskManager.Task::(_decref) Task=`1c19e02f-b7e5-4650-a115-7003a27413d6`::ref 0 aborting False >Thread-10037::DEBUG::2012-12-03 10:24:41,363::task::568::TaskManager.Task::(_updateState) Task=`207e2b6f-190c-448a-8709-fc68f86e82ef`::moving from state init -> state preparing >Thread-10037::INFO::2012-12-03 10:24:41,363::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10037::INFO::2012-12-03 10:24:41,363::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00440907478333', 'lastCheck': 1354523077.115972, 'code': 0, 'valid': True}} >Thread-10037::DEBUG::2012-12-03 10:24:41,364::task::1151::TaskManager.Task::(prepare) Task=`207e2b6f-190c-448a-8709-fc68f86e82ef`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00440907478333', 'lastCheck': 1354523077.115972, 'code': 0, 'valid': True}} >Thread-10037::DEBUG::2012-12-03 10:24:41,364::task::568::TaskManager.Task::(_updateState) Task=`207e2b6f-190c-448a-8709-fc68f86e82ef`::moving from state preparing -> state finished >Thread-10037::DEBUG::2012-12-03 10:24:41,364::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10037::DEBUG::2012-12-03 10:24:41,364::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10037::DEBUG::2012-12-03 10:24:41,364::task::957::TaskManager.Task::(_decref) Task=`207e2b6f-190c-448a-8709-fc68f86e82ef`::ref 0 aborting False >Thread-10043::DEBUG::2012-12-03 10:24:52,294::task::568::TaskManager.Task::(_updateState) Task=`4b874fa6-abbb-4f0f-a3c1-0637e3d1ee77`::moving from state init -> state preparing >Thread-10043::INFO::2012-12-03 10:24:52,295::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10043::INFO::2012-12-03 10:24:52,295::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00228500366211', 'lastCheck': 1354523087.1285591, 'code': 0, 'valid': True}} >Thread-10043::DEBUG::2012-12-03 10:24:52,295::task::1151::TaskManager.Task::(prepare) Task=`4b874fa6-abbb-4f0f-a3c1-0637e3d1ee77`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00228500366211', 'lastCheck': 1354523087.1285591, 'code': 0, 'valid': True}} >Thread-10043::DEBUG::2012-12-03 10:24:52,295::task::568::TaskManager.Task::(_updateState) Task=`4b874fa6-abbb-4f0f-a3c1-0637e3d1ee77`::moving from state preparing -> state finished >Thread-10043::DEBUG::2012-12-03 10:24:52,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10043::DEBUG::2012-12-03 10:24:52,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10043::DEBUG::2012-12-03 10:24:52,295::task::957::TaskManager.Task::(_decref) Task=`4b874fa6-abbb-4f0f-a3c1-0637e3d1ee77`::ref 0 aborting False >Thread-10049::DEBUG::2012-12-03 10:25:03,231::task::568::TaskManager.Task::(_updateState) Task=`41465961-84e0-4095-bff9-7854ba795b41`::moving from state init -> state preparing >Thread-10049::INFO::2012-12-03 10:25:03,231::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10049::INFO::2012-12-03 10:25:03,231::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00612902641296', 'lastCheck': 1354523097.146971, 'code': 0, 'valid': True}} >Thread-10049::DEBUG::2012-12-03 10:25:03,231::task::1151::TaskManager.Task::(prepare) Task=`41465961-84e0-4095-bff9-7854ba795b41`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00612902641296', 'lastCheck': 1354523097.146971, 'code': 0, 'valid': True}} >Thread-10049::DEBUG::2012-12-03 10:25:03,232::task::568::TaskManager.Task::(_updateState) Task=`41465961-84e0-4095-bff9-7854ba795b41`::moving from state preparing -> state finished >Thread-10049::DEBUG::2012-12-03 10:25:03,232::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10049::DEBUG::2012-12-03 10:25:03,232::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10049::DEBUG::2012-12-03 10:25:03,232::task::957::TaskManager.Task::(_decref) Task=`41465961-84e0-4095-bff9-7854ba795b41`::ref 0 aborting False >Thread-10055::DEBUG::2012-12-03 10:25:14,184::task::568::TaskManager.Task::(_updateState) Task=`435220bc-f8a0-4eb0-9e86-f7cb5c73f63d`::moving from state init -> state preparing >Thread-10055::INFO::2012-12-03 10:25:14,184::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10055::INFO::2012-12-03 10:25:14,184::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00250506401062', 'lastCheck': 1354523107.1675389, 'code': 0, 'valid': True}} >Thread-10055::DEBUG::2012-12-03 10:25:14,184::task::1151::TaskManager.Task::(prepare) Task=`435220bc-f8a0-4eb0-9e86-f7cb5c73f63d`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00250506401062', 'lastCheck': 1354523107.1675389, 'code': 0, 'valid': True}} >Thread-10055::DEBUG::2012-12-03 10:25:14,184::task::568::TaskManager.Task::(_updateState) Task=`435220bc-f8a0-4eb0-9e86-f7cb5c73f63d`::moving from state preparing -> state finished >Thread-10055::DEBUG::2012-12-03 10:25:14,184::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10055::DEBUG::2012-12-03 10:25:14,185::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10055::DEBUG::2012-12-03 10:25:14,185::task::957::TaskManager.Task::(_decref) Task=`435220bc-f8a0-4eb0-9e86-f7cb5c73f63d`::ref 0 aborting False >Thread-10061::DEBUG::2012-12-03 10:25:25,244::task::568::TaskManager.Task::(_updateState) Task=`a1dd289d-5049-42da-a833-6075ec00a575`::moving from state init -> state preparing >Thread-10061::INFO::2012-12-03 10:25:25,244::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10061::INFO::2012-12-03 10:25:25,244::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00256085395813', 'lastCheck': 1354523117.183754, 'code': 0, 'valid': True}} >Thread-10061::DEBUG::2012-12-03 10:25:25,244::task::1151::TaskManager.Task::(prepare) Task=`a1dd289d-5049-42da-a833-6075ec00a575`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00256085395813', 'lastCheck': 1354523117.183754, 'code': 0, 'valid': True}} >Thread-10061::DEBUG::2012-12-03 10:25:25,244::task::568::TaskManager.Task::(_updateState) Task=`a1dd289d-5049-42da-a833-6075ec00a575`::moving from state preparing -> state finished >Thread-10061::DEBUG::2012-12-03 10:25:25,244::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10061::DEBUG::2012-12-03 10:25:25,244::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10061::DEBUG::2012-12-03 10:25:25,245::task::957::TaskManager.Task::(_decref) Task=`a1dd289d-5049-42da-a833-6075ec00a575`::ref 0 aborting False >Thread-10067::DEBUG::2012-12-03 10:25:36,239::task::568::TaskManager.Task::(_updateState) Task=`0a1815ab-faf2-4299-9128-a14826a26680`::moving from state init -> state preparing >Thread-10067::INFO::2012-12-03 10:25:36,239::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10067::INFO::2012-12-03 10:25:36,240::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00249600410461', 'lastCheck': 1354523127.1976781, 'code': 0, 'valid': True}} >Thread-10067::DEBUG::2012-12-03 10:25:36,240::task::1151::TaskManager.Task::(prepare) Task=`0a1815ab-faf2-4299-9128-a14826a26680`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00249600410461', 'lastCheck': 1354523127.1976781, 'code': 0, 'valid': True}} >Thread-10067::DEBUG::2012-12-03 10:25:36,240::task::568::TaskManager.Task::(_updateState) Task=`0a1815ab-faf2-4299-9128-a14826a26680`::moving from state preparing -> state finished >Thread-10067::DEBUG::2012-12-03 10:25:36,240::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10067::DEBUG::2012-12-03 10:25:36,240::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10067::DEBUG::2012-12-03 10:25:36,240::task::957::TaskManager.Task::(_decref) Task=`0a1815ab-faf2-4299-9128-a14826a26680`::ref 0 aborting False >Thread-10073::DEBUG::2012-12-03 10:25:47,278::task::568::TaskManager.Task::(_updateState) Task=`be057378-3c6b-49f7-89bf-2258322fa6cc`::moving from state init -> state preparing >Thread-10073::INFO::2012-12-03 10:25:47,278::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10073::INFO::2012-12-03 10:25:47,279::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00442600250244', 'lastCheck': 1354523147.2300639, 'code': 0, 'valid': True}} >Thread-10073::DEBUG::2012-12-03 10:25:47,279::task::1151::TaskManager.Task::(prepare) Task=`be057378-3c6b-49f7-89bf-2258322fa6cc`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00442600250244', 'lastCheck': 1354523147.2300639, 'code': 0, 'valid': True}} >Thread-10073::DEBUG::2012-12-03 10:25:47,279::task::568::TaskManager.Task::(_updateState) Task=`be057378-3c6b-49f7-89bf-2258322fa6cc`::moving from state preparing -> state finished >Thread-10073::DEBUG::2012-12-03 10:25:47,279::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10073::DEBUG::2012-12-03 10:25:47,279::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10073::DEBUG::2012-12-03 10:25:47,279::task::957::TaskManager.Task::(_decref) Task=`be057378-3c6b-49f7-89bf-2258322fa6cc`::ref 0 aborting False >Thread-10079::DEBUG::2012-12-03 10:25:58,279::task::568::TaskManager.Task::(_updateState) Task=`7bd1e3b8-e6b1-43f2-8c3f-e56b687dd7b7`::moving from state init -> state preparing >Thread-10079::INFO::2012-12-03 10:25:58,279::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10079::INFO::2012-12-03 10:25:58,279::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00217008590698', 'lastCheck': 1354523157.242161, 'code': 0, 'valid': True}} >Thread-10079::DEBUG::2012-12-03 10:25:58,280::task::1151::TaskManager.Task::(prepare) Task=`7bd1e3b8-e6b1-43f2-8c3f-e56b687dd7b7`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00217008590698', 'lastCheck': 1354523157.242161, 'code': 0, 'valid': True}} >Thread-10079::DEBUG::2012-12-03 10:25:58,280::task::568::TaskManager.Task::(_updateState) Task=`7bd1e3b8-e6b1-43f2-8c3f-e56b687dd7b7`::moving from state preparing -> state finished >Thread-10079::DEBUG::2012-12-03 10:25:58,280::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10079::DEBUG::2012-12-03 10:25:58,280::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10079::DEBUG::2012-12-03 10:25:58,280::task::957::TaskManager.Task::(_decref) Task=`7bd1e3b8-e6b1-43f2-8c3f-e56b687dd7b7`::ref 0 aborting False >Thread-10085::DEBUG::2012-12-03 10:26:09,280::task::568::TaskManager.Task::(_updateState) Task=`36281d99-0646-492f-818f-a18a30dab559`::moving from state init -> state preparing >Thread-10085::INFO::2012-12-03 10:26:09,280::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10085::INFO::2012-12-03 10:26:09,281::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00203800201416', 'lastCheck': 1354523167.258533, 'code': 0, 'valid': True}} >Thread-10085::DEBUG::2012-12-03 10:26:09,281::task::1151::TaskManager.Task::(prepare) Task=`36281d99-0646-492f-818f-a18a30dab559`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00203800201416', 'lastCheck': 1354523167.258533, 'code': 0, 'valid': True}} >Thread-10085::DEBUG::2012-12-03 10:26:09,281::task::568::TaskManager.Task::(_updateState) Task=`36281d99-0646-492f-818f-a18a30dab559`::moving from state preparing -> state finished >Thread-10085::DEBUG::2012-12-03 10:26:09,281::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10085::DEBUG::2012-12-03 10:26:09,281::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10085::DEBUG::2012-12-03 10:26:09,281::task::957::TaskManager.Task::(_decref) Task=`36281d99-0646-492f-818f-a18a30dab559`::ref 0 aborting False >Thread-10091::DEBUG::2012-12-03 10:26:20,230::task::568::TaskManager.Task::(_updateState) Task=`87b38d1e-d20d-4886-bfe9-98238e3665ee`::moving from state init -> state preparing >Thread-10091::INFO::2012-12-03 10:26:20,231::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10091::INFO::2012-12-03 10:26:20,231::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00330209732056', 'lastCheck': 1354523177.272716, 'code': 0, 'valid': True}} >Thread-10091::DEBUG::2012-12-03 10:26:20,231::task::1151::TaskManager.Task::(prepare) Task=`87b38d1e-d20d-4886-bfe9-98238e3665ee`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00330209732056', 'lastCheck': 1354523177.272716, 'code': 0, 'valid': True}} >Thread-10091::DEBUG::2012-12-03 10:26:20,231::task::568::TaskManager.Task::(_updateState) Task=`87b38d1e-d20d-4886-bfe9-98238e3665ee`::moving from state preparing -> state finished >Thread-10091::DEBUG::2012-12-03 10:26:20,231::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10091::DEBUG::2012-12-03 10:26:20,231::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10091::DEBUG::2012-12-03 10:26:20,231::task::957::TaskManager.Task::(_decref) Task=`87b38d1e-d20d-4886-bfe9-98238e3665ee`::ref 0 aborting False >Thread-10097::DEBUG::2012-12-03 10:26:31,207::task::568::TaskManager.Task::(_updateState) Task=`328cb294-374b-41fe-9132-2945cf111da3`::moving from state init -> state preparing >Thread-10097::INFO::2012-12-03 10:26:31,207::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10097::INFO::2012-12-03 10:26:31,207::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00203609466553', 'lastCheck': 1354523187.2838571, 'code': 0, 'valid': True}} >Thread-10097::DEBUG::2012-12-03 10:26:31,208::task::1151::TaskManager.Task::(prepare) Task=`328cb294-374b-41fe-9132-2945cf111da3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00203609466553', 'lastCheck': 1354523187.2838571, 'code': 0, 'valid': True}} >Thread-10097::DEBUG::2012-12-03 10:26:31,208::task::568::TaskManager.Task::(_updateState) Task=`328cb294-374b-41fe-9132-2945cf111da3`::moving from state preparing -> state finished >Thread-10097::DEBUG::2012-12-03 10:26:31,208::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10097::DEBUG::2012-12-03 10:26:31,208::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10097::DEBUG::2012-12-03 10:26:31,208::task::957::TaskManager.Task::(_decref) Task=`328cb294-374b-41fe-9132-2945cf111da3`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 10:26:37,284::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:26:37,284::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:26:37,284::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:26:37,489::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:26:37,491::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:26:37,491::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10160::WARNING::2012-12-03 10:26:37,518::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 10:26:37,528::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:26:37,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:26:37,532::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:26:37,535::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:26:37,535::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:26:37,535::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-10165::DEBUG::2012-12-03 10:26:42,190::task::568::TaskManager.Task::(_updateState) Task=`1cf79a4c-f169-4ab8-9757-127329beada6`::moving from state init -> state preparing >Thread-10165::INFO::2012-12-03 10:26:42,191::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10165::INFO::2012-12-03 10:26:42,191::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00162100791931', 'lastCheck': 1354523197.5473549, 'code': 0, 'valid': True}} >Thread-10165::DEBUG::2012-12-03 10:26:42,191::task::1151::TaskManager.Task::(prepare) Task=`1cf79a4c-f169-4ab8-9757-127329beada6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00162100791931', 'lastCheck': 1354523197.5473549, 'code': 0, 'valid': True}} >Thread-10165::DEBUG::2012-12-03 10:26:42,191::task::568::TaskManager.Task::(_updateState) Task=`1cf79a4c-f169-4ab8-9757-127329beada6`::moving from state preparing -> state finished >Thread-10165::DEBUG::2012-12-03 10:26:42,191::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10165::DEBUG::2012-12-03 10:26:42,191::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10165::DEBUG::2012-12-03 10:26:42,192::task::957::TaskManager.Task::(_decref) Task=`1cf79a4c-f169-4ab8-9757-127329beada6`::ref 0 aborting False >Thread-10171::DEBUG::2012-12-03 10:26:54,175::task::568::TaskManager.Task::(_updateState) Task=`588588ba-efe7-4983-a0d0-e69aaf61b53e`::moving from state init -> state preparing >Thread-10171::INFO::2012-12-03 10:26:54,175::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10171::INFO::2012-12-03 10:26:54,176::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00235295295715', 'lastCheck': 1354523207.560766, 'code': 0, 'valid': True}} >Thread-10171::DEBUG::2012-12-03 10:26:54,176::task::1151::TaskManager.Task::(prepare) Task=`588588ba-efe7-4983-a0d0-e69aaf61b53e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00235295295715', 'lastCheck': 1354523207.560766, 'code': 0, 'valid': True}} >Thread-10171::DEBUG::2012-12-03 10:26:54,176::task::568::TaskManager.Task::(_updateState) Task=`588588ba-efe7-4983-a0d0-e69aaf61b53e`::moving from state preparing -> state finished >Thread-10171::DEBUG::2012-12-03 10:26:54,176::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10171::DEBUG::2012-12-03 10:26:54,176::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10171::DEBUG::2012-12-03 10:26:54,176::task::957::TaskManager.Task::(_decref) Task=`588588ba-efe7-4983-a0d0-e69aaf61b53e`::ref 0 aborting False >Thread-10177::DEBUG::2012-12-03 10:27:05,285::task::568::TaskManager.Task::(_updateState) Task=`24a19618-afbf-4c23-bf00-022013af8bce`::moving from state init -> state preparing >Thread-10177::INFO::2012-12-03 10:27:05,286::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10177::INFO::2012-12-03 10:27:05,286::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00243997573853', 'lastCheck': 1354523217.576421, 'code': 0, 'valid': True}} >Thread-10177::DEBUG::2012-12-03 10:27:05,286::task::1151::TaskManager.Task::(prepare) Task=`24a19618-afbf-4c23-bf00-022013af8bce`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00243997573853', 'lastCheck': 1354523217.576421, 'code': 0, 'valid': True}} >Thread-10177::DEBUG::2012-12-03 10:27:05,286::task::568::TaskManager.Task::(_updateState) Task=`24a19618-afbf-4c23-bf00-022013af8bce`::moving from state preparing -> state finished >Thread-10177::DEBUG::2012-12-03 10:27:05,286::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10177::DEBUG::2012-12-03 10:27:05,286::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10177::DEBUG::2012-12-03 10:27:05,287::task::957::TaskManager.Task::(_decref) Task=`24a19618-afbf-4c23-bf00-022013af8bce`::ref 0 aborting False >Thread-10183::DEBUG::2012-12-03 10:27:16,764::task::568::TaskManager.Task::(_updateState) Task=`fc247afa-745b-4508-9d77-9e72f45f44d5`::moving from state init -> state preparing >Thread-10183::INFO::2012-12-03 10:27:16,764::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10183::INFO::2012-12-03 10:27:16,764::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00151085853577', 'lastCheck': 1354523227.5915561, 'code': 0, 'valid': True}} >Thread-10183::DEBUG::2012-12-03 10:27:16,765::task::1151::TaskManager.Task::(prepare) Task=`fc247afa-745b-4508-9d77-9e72f45f44d5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00151085853577', 'lastCheck': 1354523227.5915561, 'code': 0, 'valid': True}} >Thread-10183::DEBUG::2012-12-03 10:27:16,765::task::568::TaskManager.Task::(_updateState) Task=`fc247afa-745b-4508-9d77-9e72f45f44d5`::moving from state preparing -> state finished >Thread-10183::DEBUG::2012-12-03 10:27:16,765::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10183::DEBUG::2012-12-03 10:27:16,765::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10183::DEBUG::2012-12-03 10:27:16,765::task::957::TaskManager.Task::(_decref) Task=`fc247afa-745b-4508-9d77-9e72f45f44d5`::ref 0 aborting False >Thread-10189::DEBUG::2012-12-03 10:27:27,998::task::568::TaskManager.Task::(_updateState) Task=`131b2113-6b83-4367-8d47-e17fd5c40757`::moving from state init -> state preparing >Thread-10189::INFO::2012-12-03 10:27:27,998::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10189::INFO::2012-12-03 10:27:27,999::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00223398208618', 'lastCheck': 1354523247.6152711, 'code': 0, 'valid': True}} >Thread-10189::DEBUG::2012-12-03 10:27:27,999::task::1151::TaskManager.Task::(prepare) Task=`131b2113-6b83-4367-8d47-e17fd5c40757`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00223398208618', 'lastCheck': 1354523247.6152711, 'code': 0, 'valid': True}} >Thread-10189::DEBUG::2012-12-03 10:27:27,999::task::568::TaskManager.Task::(_updateState) Task=`131b2113-6b83-4367-8d47-e17fd5c40757`::moving from state preparing -> state finished >Thread-10189::DEBUG::2012-12-03 10:27:27,999::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10189::DEBUG::2012-12-03 10:27:27,999::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10189::DEBUG::2012-12-03 10:27:27,999::task::957::TaskManager.Task::(_decref) Task=`131b2113-6b83-4367-8d47-e17fd5c40757`::ref 0 aborting False >Thread-10195::DEBUG::2012-12-03 10:27:39,289::task::568::TaskManager.Task::(_updateState) Task=`679cff86-bf88-44f1-973a-aa26345139c4`::moving from state init -> state preparing >Thread-10195::INFO::2012-12-03 10:27:39,289::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10195::INFO::2012-12-03 10:27:39,290::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00473380088806', 'lastCheck': 1354523257.6321859, 'code': 0, 'valid': True}} >Thread-10195::DEBUG::2012-12-03 10:27:39,290::task::1151::TaskManager.Task::(prepare) Task=`679cff86-bf88-44f1-973a-aa26345139c4`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00473380088806', 'lastCheck': 1354523257.6321859, 'code': 0, 'valid': True}} >Thread-10195::DEBUG::2012-12-03 10:27:39,290::task::568::TaskManager.Task::(_updateState) Task=`679cff86-bf88-44f1-973a-aa26345139c4`::moving from state preparing -> state finished >Thread-10195::DEBUG::2012-12-03 10:27:39,290::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10195::DEBUG::2012-12-03 10:27:39,290::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10195::DEBUG::2012-12-03 10:27:39,290::task::957::TaskManager.Task::(_decref) Task=`679cff86-bf88-44f1-973a-aa26345139c4`::ref 0 aborting False >Thread-10201::DEBUG::2012-12-03 10:27:50,323::task::568::TaskManager.Task::(_updateState) Task=`c9835c0a-e37e-48da-9b34-ee3d2b8b29e7`::moving from state init -> state preparing >Thread-10201::INFO::2012-12-03 10:27:50,323::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10201::INFO::2012-12-03 10:27:50,323::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00210905075073', 'lastCheck': 1354523267.6443329, 'code': 0, 'valid': True}} >Thread-10201::DEBUG::2012-12-03 10:27:50,323::task::1151::TaskManager.Task::(prepare) Task=`c9835c0a-e37e-48da-9b34-ee3d2b8b29e7`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00210905075073', 'lastCheck': 1354523267.6443329, 'code': 0, 'valid': True}} >Thread-10201::DEBUG::2012-12-03 10:27:50,323::task::568::TaskManager.Task::(_updateState) Task=`c9835c0a-e37e-48da-9b34-ee3d2b8b29e7`::moving from state preparing -> state finished >Thread-10201::DEBUG::2012-12-03 10:27:50,323::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10201::DEBUG::2012-12-03 10:27:50,324::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10201::DEBUG::2012-12-03 10:27:50,324::task::957::TaskManager.Task::(_decref) Task=`c9835c0a-e37e-48da-9b34-ee3d2b8b29e7`::ref 0 aborting False >Thread-10207::DEBUG::2012-12-03 10:28:01,426::task::568::TaskManager.Task::(_updateState) Task=`917710e4-738e-42e9-9e54-67802665b035`::moving from state init -> state preparing >Thread-10207::INFO::2012-12-03 10:28:01,426::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10207::INFO::2012-12-03 10:28:01,426::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00264501571655', 'lastCheck': 1354523277.6593289, 'code': 0, 'valid': True}} >Thread-10207::DEBUG::2012-12-03 10:28:01,427::task::1151::TaskManager.Task::(prepare) Task=`917710e4-738e-42e9-9e54-67802665b035`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00264501571655', 'lastCheck': 1354523277.6593289, 'code': 0, 'valid': True}} >Thread-10207::DEBUG::2012-12-03 10:28:01,427::task::568::TaskManager.Task::(_updateState) Task=`917710e4-738e-42e9-9e54-67802665b035`::moving from state preparing -> state finished >Thread-10207::DEBUG::2012-12-03 10:28:01,427::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10207::DEBUG::2012-12-03 10:28:01,427::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10207::DEBUG::2012-12-03 10:28:01,427::task::957::TaskManager.Task::(_decref) Task=`917710e4-738e-42e9-9e54-67802665b035`::ref 0 aborting False >Thread-10213::DEBUG::2012-12-03 10:28:12,469::task::568::TaskManager.Task::(_updateState) Task=`2bfd9c56-e9b7-4c6e-8547-bb097e8fbac8`::moving from state init -> state preparing >Thread-10213::INFO::2012-12-03 10:28:12,469::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10213::INFO::2012-12-03 10:28:12,469::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00179600715637', 'lastCheck': 1354523287.681004, 'code': 0, 'valid': True}} >Thread-10213::DEBUG::2012-12-03 10:28:12,469::task::1151::TaskManager.Task::(prepare) Task=`2bfd9c56-e9b7-4c6e-8547-bb097e8fbac8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00179600715637', 'lastCheck': 1354523287.681004, 'code': 0, 'valid': True}} >Thread-10213::DEBUG::2012-12-03 10:28:12,469::task::568::TaskManager.Task::(_updateState) Task=`2bfd9c56-e9b7-4c6e-8547-bb097e8fbac8`::moving from state preparing -> state finished >Thread-10213::DEBUG::2012-12-03 10:28:12,470::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10213::DEBUG::2012-12-03 10:28:12,470::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10213::DEBUG::2012-12-03 10:28:12,470::task::957::TaskManager.Task::(_decref) Task=`2bfd9c56-e9b7-4c6e-8547-bb097e8fbac8`::ref 0 aborting False >Thread-10219::DEBUG::2012-12-03 10:28:23,943::task::568::TaskManager.Task::(_updateState) Task=`967cc9f9-c2d2-4793-af7a-e6daf51b26d9`::moving from state init -> state preparing >Thread-10219::INFO::2012-12-03 10:28:23,943::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10219::INFO::2012-12-03 10:28:23,943::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00251007080078', 'lastCheck': 1354523297.693511, 'code': 0, 'valid': True}} >Thread-10219::DEBUG::2012-12-03 10:28:23,943::task::1151::TaskManager.Task::(prepare) Task=`967cc9f9-c2d2-4793-af7a-e6daf51b26d9`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00251007080078', 'lastCheck': 1354523297.693511, 'code': 0, 'valid': True}} >Thread-10219::DEBUG::2012-12-03 10:28:23,943::task::568::TaskManager.Task::(_updateState) Task=`967cc9f9-c2d2-4793-af7a-e6daf51b26d9`::moving from state preparing -> state finished >Thread-10219::DEBUG::2012-12-03 10:28:23,944::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10219::DEBUG::2012-12-03 10:28:23,944::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10219::DEBUG::2012-12-03 10:28:23,944::task::957::TaskManager.Task::(_decref) Task=`967cc9f9-c2d2-4793-af7a-e6daf51b26d9`::ref 0 aborting False >Thread-10225::DEBUG::2012-12-03 10:28:34,943::task::568::TaskManager.Task::(_updateState) Task=`b0edd8a0-cf94-4493-8c9e-72293fd03c40`::moving from state init -> state preparing >Thread-10225::INFO::2012-12-03 10:28:34,943::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10225::INFO::2012-12-03 10:28:34,943::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00243306159973', 'lastCheck': 1354523307.7054811, 'code': 0, 'valid': True}} >Thread-10225::DEBUG::2012-12-03 10:28:34,943::task::1151::TaskManager.Task::(prepare) Task=`b0edd8a0-cf94-4493-8c9e-72293fd03c40`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00243306159973', 'lastCheck': 1354523307.7054811, 'code': 0, 'valid': True}} >Thread-10225::DEBUG::2012-12-03 10:28:34,943::task::568::TaskManager.Task::(_updateState) Task=`b0edd8a0-cf94-4493-8c9e-72293fd03c40`::moving from state preparing -> state finished >Thread-10225::DEBUG::2012-12-03 10:28:34,943::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10225::DEBUG::2012-12-03 10:28:34,943::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10225::DEBUG::2012-12-03 10:28:34,944::task::957::TaskManager.Task::(_decref) Task=`b0edd8a0-cf94-4493-8c9e-72293fd03c40`::ref 0 aborting False >Thread-10231::DEBUG::2012-12-03 10:28:46,011::task::568::TaskManager.Task::(_updateState) Task=`7304e61c-a55b-46e6-9287-8514127b9a2c`::moving from state init -> state preparing >Thread-10231::INFO::2012-12-03 10:28:46,011::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10231::INFO::2012-12-03 10:28:46,011::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00471782684326', 'lastCheck': 1354523317.721808, 'code': 0, 'valid': True}} >Thread-10231::DEBUG::2012-12-03 10:28:46,012::task::1151::TaskManager.Task::(prepare) Task=`7304e61c-a55b-46e6-9287-8514127b9a2c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00471782684326', 'lastCheck': 1354523317.721808, 'code': 0, 'valid': True}} >Thread-10231::DEBUG::2012-12-03 10:28:46,012::task::568::TaskManager.Task::(_updateState) Task=`7304e61c-a55b-46e6-9287-8514127b9a2c`::moving from state preparing -> state finished >Thread-10231::DEBUG::2012-12-03 10:28:46,012::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10231::DEBUG::2012-12-03 10:28:46,012::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10231::DEBUG::2012-12-03 10:28:46,012::task::957::TaskManager.Task::(_decref) Task=`7304e61c-a55b-46e6-9287-8514127b9a2c`::ref 0 aborting False >Thread-10237::DEBUG::2012-12-03 10:28:56,953::task::568::TaskManager.Task::(_updateState) Task=`9a5d8443-4117-4cb7-8b7e-956d9627605b`::moving from state init -> state preparing >Thread-10237::INFO::2012-12-03 10:28:56,953::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10237::INFO::2012-12-03 10:28:56,953::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00232100486755', 'lastCheck': 1354523327.7343869, 'code': 0, 'valid': True}} >Thread-10237::DEBUG::2012-12-03 10:28:56,954::task::1151::TaskManager.Task::(prepare) Task=`9a5d8443-4117-4cb7-8b7e-956d9627605b`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00232100486755', 'lastCheck': 1354523327.7343869, 'code': 0, 'valid': True}} >Thread-10237::DEBUG::2012-12-03 10:28:56,954::task::568::TaskManager.Task::(_updateState) Task=`9a5d8443-4117-4cb7-8b7e-956d9627605b`::moving from state preparing -> state finished >Thread-10237::DEBUG::2012-12-03 10:28:56,954::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10237::DEBUG::2012-12-03 10:28:56,954::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10237::DEBUG::2012-12-03 10:28:56,954::task::957::TaskManager.Task::(_decref) Task=`9a5d8443-4117-4cb7-8b7e-956d9627605b`::ref 0 aborting False >Thread-10243::DEBUG::2012-12-03 10:29:08,924::task::568::TaskManager.Task::(_updateState) Task=`a9155e1e-7e3d-47dc-905d-dcd5c6928083`::moving from state init -> state preparing >Thread-10243::INFO::2012-12-03 10:29:08,924::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10243::INFO::2012-12-03 10:29:08,924::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00275111198425', 'lastCheck': 1354523347.7668891, 'code': 0, 'valid': True}} >Thread-10243::DEBUG::2012-12-03 10:29:08,924::task::1151::TaskManager.Task::(prepare) Task=`a9155e1e-7e3d-47dc-905d-dcd5c6928083`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00275111198425', 'lastCheck': 1354523347.7668891, 'code': 0, 'valid': True}} >Thread-10243::DEBUG::2012-12-03 10:29:08,924::task::568::TaskManager.Task::(_updateState) Task=`a9155e1e-7e3d-47dc-905d-dcd5c6928083`::moving from state preparing -> state finished >Thread-10243::DEBUG::2012-12-03 10:29:08,925::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10243::DEBUG::2012-12-03 10:29:08,925::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10243::DEBUG::2012-12-03 10:29:08,925::task::957::TaskManager.Task::(_decref) Task=`a9155e1e-7e3d-47dc-905d-dcd5c6928083`::ref 0 aborting False >Thread-10249::DEBUG::2012-12-03 10:29:20,288::task::568::TaskManager.Task::(_updateState) Task=`9459cb58-1e8a-41bf-8638-98c9b5dea761`::moving from state init -> state preparing >Thread-10249::INFO::2012-12-03 10:29:20,288::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10249::INFO::2012-12-03 10:29:20,288::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00242304801941', 'lastCheck': 1354523357.7805641, 'code': 0, 'valid': True}} >Thread-10249::DEBUG::2012-12-03 10:29:20,288::task::1151::TaskManager.Task::(prepare) Task=`9459cb58-1e8a-41bf-8638-98c9b5dea761`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00242304801941', 'lastCheck': 1354523357.7805641, 'code': 0, 'valid': True}} >Thread-10249::DEBUG::2012-12-03 10:29:20,288::task::568::TaskManager.Task::(_updateState) Task=`9459cb58-1e8a-41bf-8638-98c9b5dea761`::moving from state preparing -> state finished >Thread-10249::DEBUG::2012-12-03 10:29:20,289::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10249::DEBUG::2012-12-03 10:29:20,289::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10249::DEBUG::2012-12-03 10:29:20,289::task::957::TaskManager.Task::(_decref) Task=`9459cb58-1e8a-41bf-8638-98c9b5dea761`::ref 0 aborting False >Thread-10255::DEBUG::2012-12-03 10:29:31,300::task::568::TaskManager.Task::(_updateState) Task=`ee44b52a-4126-4082-84b5-d5c2f0667e22`::moving from state init -> state preparing >Thread-10255::INFO::2012-12-03 10:29:31,300::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10255::INFO::2012-12-03 10:29:31,300::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00246691703796', 'lastCheck': 1354523367.793611, 'code': 0, 'valid': True}} >Thread-10255::DEBUG::2012-12-03 10:29:31,300::task::1151::TaskManager.Task::(prepare) Task=`ee44b52a-4126-4082-84b5-d5c2f0667e22`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00246691703796', 'lastCheck': 1354523367.793611, 'code': 0, 'valid': True}} >Thread-10255::DEBUG::2012-12-03 10:29:31,300::task::568::TaskManager.Task::(_updateState) Task=`ee44b52a-4126-4082-84b5-d5c2f0667e22`::moving from state preparing -> state finished >Thread-10255::DEBUG::2012-12-03 10:29:31,301::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10255::DEBUG::2012-12-03 10:29:31,301::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10255::DEBUG::2012-12-03 10:29:31,301::task::957::TaskManager.Task::(_decref) Task=`ee44b52a-4126-4082-84b5-d5c2f0667e22`::ref 0 aborting False >Thread-10261::DEBUG::2012-12-03 10:29:42,245::task::568::TaskManager.Task::(_updateState) Task=`34b54beb-35ff-4808-af9f-cbaea62b54d6`::moving from state init -> state preparing >Thread-10261::INFO::2012-12-03 10:29:42,245::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10261::INFO::2012-12-03 10:29:42,246::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00559592247009', 'lastCheck': 1354523377.8097579, 'code': 0, 'valid': True}} >Thread-10261::DEBUG::2012-12-03 10:29:42,246::task::1151::TaskManager.Task::(prepare) Task=`34b54beb-35ff-4808-af9f-cbaea62b54d6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00559592247009', 'lastCheck': 1354523377.8097579, 'code': 0, 'valid': True}} >Thread-10261::DEBUG::2012-12-03 10:29:42,246::task::568::TaskManager.Task::(_updateState) Task=`34b54beb-35ff-4808-af9f-cbaea62b54d6`::moving from state preparing -> state finished >Thread-10261::DEBUG::2012-12-03 10:29:42,246::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10261::DEBUG::2012-12-03 10:29:42,246::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10261::DEBUG::2012-12-03 10:29:42,246::task::957::TaskManager.Task::(_decref) Task=`34b54beb-35ff-4808-af9f-cbaea62b54d6`::ref 0 aborting False >Thread-10267::DEBUG::2012-12-03 10:29:53,246::task::568::TaskManager.Task::(_updateState) Task=`c2692698-bcc8-4167-9778-18ed2d947f59`::moving from state init -> state preparing >Thread-10267::INFO::2012-12-03 10:29:53,246::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10267::INFO::2012-12-03 10:29:53,246::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00245714187622', 'lastCheck': 1354523387.8229899, 'code': 0, 'valid': True}} >Thread-10267::DEBUG::2012-12-03 10:29:53,246::task::1151::TaskManager.Task::(prepare) Task=`c2692698-bcc8-4167-9778-18ed2d947f59`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00245714187622', 'lastCheck': 1354523387.8229899, 'code': 0, 'valid': True}} >Thread-10267::DEBUG::2012-12-03 10:29:53,246::task::568::TaskManager.Task::(_updateState) Task=`c2692698-bcc8-4167-9778-18ed2d947f59`::moving from state preparing -> state finished >Thread-10267::DEBUG::2012-12-03 10:29:53,247::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10267::DEBUG::2012-12-03 10:29:53,247::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10267::DEBUG::2012-12-03 10:29:53,247::task::957::TaskManager.Task::(_decref) Task=`c2692698-bcc8-4167-9778-18ed2d947f59`::ref 0 aborting False >Thread-10273::DEBUG::2012-12-03 10:30:04,229::task::568::TaskManager.Task::(_updateState) Task=`034023f5-48f7-43c8-b222-227f24e7a2dc`::moving from state init -> state preparing >Thread-10273::INFO::2012-12-03 10:30:04,229::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10273::INFO::2012-12-03 10:30:04,230::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0026969909668', 'lastCheck': 1354523397.835408, 'code': 0, 'valid': True}} >Thread-10273::DEBUG::2012-12-03 10:30:04,230::task::1151::TaskManager.Task::(prepare) Task=`034023f5-48f7-43c8-b222-227f24e7a2dc`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0026969909668', 'lastCheck': 1354523397.835408, 'code': 0, 'valid': True}} >Thread-10273::DEBUG::2012-12-03 10:30:04,230::task::568::TaskManager.Task::(_updateState) Task=`034023f5-48f7-43c8-b222-227f24e7a2dc`::moving from state preparing -> state finished >Thread-10273::DEBUG::2012-12-03 10:30:04,230::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10273::DEBUG::2012-12-03 10:30:04,230::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10273::DEBUG::2012-12-03 10:30:04,230::task::957::TaskManager.Task::(_decref) Task=`034023f5-48f7-43c8-b222-227f24e7a2dc`::ref 0 aborting False >Thread-10279::DEBUG::2012-12-03 10:30:15,167::task::568::TaskManager.Task::(_updateState) Task=`9860ad37-92f1-4b7a-8f4d-40a4673ca458`::moving from state init -> state preparing >Thread-10279::INFO::2012-12-03 10:30:15,167::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10279::INFO::2012-12-03 10:30:15,167::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00248098373413', 'lastCheck': 1354523407.854923, 'code': 0, 'valid': True}} >Thread-10279::DEBUG::2012-12-03 10:30:15,167::task::1151::TaskManager.Task::(prepare) Task=`9860ad37-92f1-4b7a-8f4d-40a4673ca458`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00248098373413', 'lastCheck': 1354523407.854923, 'code': 0, 'valid': True}} >Thread-10279::DEBUG::2012-12-03 10:30:15,168::task::568::TaskManager.Task::(_updateState) Task=`9860ad37-92f1-4b7a-8f4d-40a4673ca458`::moving from state preparing -> state finished >Thread-10279::DEBUG::2012-12-03 10:30:15,168::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10279::DEBUG::2012-12-03 10:30:15,168::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10279::DEBUG::2012-12-03 10:30:15,168::task::957::TaskManager.Task::(_decref) Task=`9860ad37-92f1-4b7a-8f4d-40a4673ca458`::ref 0 aborting False >Thread-10285::DEBUG::2012-12-03 10:30:26,256::task::568::TaskManager.Task::(_updateState) Task=`5cb3c7a4-9828-4262-b26d-9e254074257f`::moving from state init -> state preparing >Thread-10285::INFO::2012-12-03 10:30:26,256::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10285::INFO::2012-12-03 10:30:26,256::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00317096710205', 'lastCheck': 1354523417.8690331, 'code': 0, 'valid': True}} >Thread-10285::DEBUG::2012-12-03 10:30:26,256::task::1151::TaskManager.Task::(prepare) Task=`5cb3c7a4-9828-4262-b26d-9e254074257f`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00317096710205', 'lastCheck': 1354523417.8690331, 'code': 0, 'valid': True}} >Thread-10285::DEBUG::2012-12-03 10:30:26,257::task::568::TaskManager.Task::(_updateState) Task=`5cb3c7a4-9828-4262-b26d-9e254074257f`::moving from state preparing -> state finished >Thread-10285::DEBUG::2012-12-03 10:30:26,257::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10285::DEBUG::2012-12-03 10:30:26,257::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10285::DEBUG::2012-12-03 10:30:26,257::task::957::TaskManager.Task::(_decref) Task=`5cb3c7a4-9828-4262-b26d-9e254074257f`::ref 0 aborting False >Thread-10291::DEBUG::2012-12-03 10:30:37,440::task::568::TaskManager.Task::(_updateState) Task=`0ee9fcd7-96e4-46f1-b3ca-53b962261de3`::moving from state init -> state preparing >Thread-10291::INFO::2012-12-03 10:30:37,440::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10291::INFO::2012-12-03 10:30:37,440::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00427103042603', 'lastCheck': 1354523427.8864219, 'code': 0, 'valid': True}} >Thread-10291::DEBUG::2012-12-03 10:30:37,440::task::1151::TaskManager.Task::(prepare) Task=`0ee9fcd7-96e4-46f1-b3ca-53b962261de3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00427103042603', 'lastCheck': 1354523427.8864219, 'code': 0, 'valid': True}} >Thread-10291::DEBUG::2012-12-03 10:30:37,440::task::568::TaskManager.Task::(_updateState) Task=`0ee9fcd7-96e4-46f1-b3ca-53b962261de3`::moving from state preparing -> state finished >Thread-10291::DEBUG::2012-12-03 10:30:37,441::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10291::DEBUG::2012-12-03 10:30:37,441::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10291::DEBUG::2012-12-03 10:30:37,441::task::957::TaskManager.Task::(_decref) Task=`0ee9fcd7-96e4-46f1-b3ca-53b962261de3`::ref 0 aborting False >Thread-10297::DEBUG::2012-12-03 10:30:48,668::task::568::TaskManager.Task::(_updateState) Task=`f8adaf01-9e38-41ba-b96d-065f7c4a50a5`::moving from state init -> state preparing >Thread-10297::INFO::2012-12-03 10:30:48,669::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10297::INFO::2012-12-03 10:30:48,669::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0026228427887', 'lastCheck': 1354523447.9144089, 'code': 0, 'valid': True}} >Thread-10297::DEBUG::2012-12-03 10:30:48,669::task::1151::TaskManager.Task::(prepare) Task=`f8adaf01-9e38-41ba-b96d-065f7c4a50a5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0026228427887', 'lastCheck': 1354523447.9144089, 'code': 0, 'valid': True}} >Thread-10297::DEBUG::2012-12-03 10:30:48,669::task::568::TaskManager.Task::(_updateState) Task=`f8adaf01-9e38-41ba-b96d-065f7c4a50a5`::moving from state preparing -> state finished >Thread-10297::DEBUG::2012-12-03 10:30:48,669::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10297::DEBUG::2012-12-03 10:30:48,669::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10297::DEBUG::2012-12-03 10:30:48,669::task::957::TaskManager.Task::(_decref) Task=`f8adaf01-9e38-41ba-b96d-065f7c4a50a5`::ref 0 aborting False >Thread-10303::DEBUG::2012-12-03 10:30:59,964::task::568::TaskManager.Task::(_updateState) Task=`ca62a2eb-349d-44a8-bfe9-f6be6c53515a`::moving from state init -> state preparing >Thread-10303::INFO::2012-12-03 10:30:59,964::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10303::INFO::2012-12-03 10:30:59,964::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00275301933289', 'lastCheck': 1354523457.9287369, 'code': 0, 'valid': True}} >Thread-10303::DEBUG::2012-12-03 10:30:59,965::task::1151::TaskManager.Task::(prepare) Task=`ca62a2eb-349d-44a8-bfe9-f6be6c53515a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00275301933289', 'lastCheck': 1354523457.9287369, 'code': 0, 'valid': True}} >Thread-10303::DEBUG::2012-12-03 10:30:59,965::task::568::TaskManager.Task::(_updateState) Task=`ca62a2eb-349d-44a8-bfe9-f6be6c53515a`::moving from state preparing -> state finished >Thread-10303::DEBUG::2012-12-03 10:30:59,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10303::DEBUG::2012-12-03 10:30:59,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10303::DEBUG::2012-12-03 10:30:59,965::task::957::TaskManager.Task::(_decref) Task=`ca62a2eb-349d-44a8-bfe9-f6be6c53515a`::ref 0 aborting False >Thread-10309::DEBUG::2012-12-03 10:31:11,184::task::568::TaskManager.Task::(_updateState) Task=`9efb067b-7f86-48d1-875c-73ebc4598b97`::moving from state init -> state preparing >Thread-10309::INFO::2012-12-03 10:31:11,184::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10309::INFO::2012-12-03 10:31:11,184::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00148391723633', 'lastCheck': 1354523467.9422359, 'code': 0, 'valid': True}} >Thread-10309::DEBUG::2012-12-03 10:31:11,184::task::1151::TaskManager.Task::(prepare) Task=`9efb067b-7f86-48d1-875c-73ebc4598b97`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00148391723633', 'lastCheck': 1354523467.9422359, 'code': 0, 'valid': True}} >Thread-10309::DEBUG::2012-12-03 10:31:11,184::task::568::TaskManager.Task::(_updateState) Task=`9efb067b-7f86-48d1-875c-73ebc4598b97`::moving from state preparing -> state finished >Thread-10309::DEBUG::2012-12-03 10:31:11,185::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10309::DEBUG::2012-12-03 10:31:11,185::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10309::DEBUG::2012-12-03 10:31:11,185::task::957::TaskManager.Task::(_decref) Task=`9efb067b-7f86-48d1-875c-73ebc4598b97`::ref 0 aborting False >Thread-10315::DEBUG::2012-12-03 10:31:22,245::task::568::TaskManager.Task::(_updateState) Task=`c3220f59-b46d-4269-8976-856d2028fbdf`::moving from state init -> state preparing >Thread-10315::INFO::2012-12-03 10:31:22,245::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10315::INFO::2012-12-03 10:31:22,245::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00162696838379', 'lastCheck': 1354523477.953052, 'code': 0, 'valid': True}} >Thread-10315::DEBUG::2012-12-03 10:31:22,245::task::1151::TaskManager.Task::(prepare) Task=`c3220f59-b46d-4269-8976-856d2028fbdf`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00162696838379', 'lastCheck': 1354523477.953052, 'code': 0, 'valid': True}} >Thread-10315::DEBUG::2012-12-03 10:31:22,246::task::568::TaskManager.Task::(_updateState) Task=`c3220f59-b46d-4269-8976-856d2028fbdf`::moving from state preparing -> state finished >Thread-10315::DEBUG::2012-12-03 10:31:22,246::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10315::DEBUG::2012-12-03 10:31:22,246::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10315::DEBUG::2012-12-03 10:31:22,246::task::957::TaskManager.Task::(_decref) Task=`c3220f59-b46d-4269-8976-856d2028fbdf`::ref 0 aborting False >Thread-10321::DEBUG::2012-12-03 10:31:33,301::task::568::TaskManager.Task::(_updateState) Task=`03be6049-d931-4f14-8082-b49287d0cca5`::moving from state init -> state preparing >Thread-10321::INFO::2012-12-03 10:31:33,302::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10321::INFO::2012-12-03 10:31:33,302::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00143504142761', 'lastCheck': 1354523487.9628561, 'code': 0, 'valid': True}} >Thread-10321::DEBUG::2012-12-03 10:31:33,302::task::1151::TaskManager.Task::(prepare) Task=`03be6049-d931-4f14-8082-b49287d0cca5`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00143504142761', 'lastCheck': 1354523487.9628561, 'code': 0, 'valid': True}} >Thread-10321::DEBUG::2012-12-03 10:31:33,302::task::568::TaskManager.Task::(_updateState) Task=`03be6049-d931-4f14-8082-b49287d0cca5`::moving from state preparing -> state finished >Thread-10321::DEBUG::2012-12-03 10:31:33,302::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10321::DEBUG::2012-12-03 10:31:33,302::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10321::DEBUG::2012-12-03 10:31:33,302::task::957::TaskManager.Task::(_decref) Task=`03be6049-d931-4f14-8082-b49287d0cca5`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 10:31:37,963::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:31:37,963::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:31:37,963::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:31:38,174::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:31:38,175::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:31:38,175::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:31:38,212::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:31:38,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:31:38,216::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:31:38,218::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:31:38,219::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:31:38,219::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-10389::DEBUG::2012-12-03 10:31:44,249::task::568::TaskManager.Task::(_updateState) Task=`3adca228-7b3c-4f5e-890d-00761d21b9e7`::moving from state init -> state preparing >Thread-10389::INFO::2012-12-03 10:31:44,249::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10389::INFO::2012-12-03 10:31:44,250::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00217390060425', 'lastCheck': 1354523498.23105, 'code': 0, 'valid': True}} >Thread-10389::DEBUG::2012-12-03 10:31:44,250::task::1151::TaskManager.Task::(prepare) Task=`3adca228-7b3c-4f5e-890d-00761d21b9e7`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00217390060425', 'lastCheck': 1354523498.23105, 'code': 0, 'valid': True}} >Thread-10389::DEBUG::2012-12-03 10:31:44,250::task::568::TaskManager.Task::(_updateState) Task=`3adca228-7b3c-4f5e-890d-00761d21b9e7`::moving from state preparing -> state finished >Thread-10389::DEBUG::2012-12-03 10:31:44,250::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10389::DEBUG::2012-12-03 10:31:44,250::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10389::DEBUG::2012-12-03 10:31:44,250::task::957::TaskManager.Task::(_decref) Task=`3adca228-7b3c-4f5e-890d-00761d21b9e7`::ref 0 aborting False >Thread-10395::DEBUG::2012-12-03 10:31:55,205::task::568::TaskManager.Task::(_updateState) Task=`802ea738-f80d-4dc1-a716-3f5d7d99c853`::moving from state init -> state preparing >Thread-10395::INFO::2012-12-03 10:31:55,205::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10395::INFO::2012-12-03 10:31:55,205::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00391721725464', 'lastCheck': 1354523508.2464659, 'code': 0, 'valid': True}} >Thread-10395::DEBUG::2012-12-03 10:31:55,205::task::1151::TaskManager.Task::(prepare) Task=`802ea738-f80d-4dc1-a716-3f5d7d99c853`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00391721725464', 'lastCheck': 1354523508.2464659, 'code': 0, 'valid': True}} >Thread-10395::DEBUG::2012-12-03 10:31:55,205::task::568::TaskManager.Task::(_updateState) Task=`802ea738-f80d-4dc1-a716-3f5d7d99c853`::moving from state preparing -> state finished >Thread-10395::DEBUG::2012-12-03 10:31:55,205::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10395::DEBUG::2012-12-03 10:31:55,205::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10395::DEBUG::2012-12-03 10:31:55,206::task::957::TaskManager.Task::(_decref) Task=`802ea738-f80d-4dc1-a716-3f5d7d99c853`::ref 0 aborting False >Thread-10401::DEBUG::2012-12-03 10:32:06,158::task::568::TaskManager.Task::(_updateState) Task=`b46de16a-13f3-4056-a30c-0b575a8a96d8`::moving from state init -> state preparing >Thread-10401::INFO::2012-12-03 10:32:06,158::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10401::INFO::2012-12-03 10:32:06,159::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00245118141174', 'lastCheck': 1354523518.2600031, 'code': 0, 'valid': True}} >Thread-10401::DEBUG::2012-12-03 10:32:06,159::task::1151::TaskManager.Task::(prepare) Task=`b46de16a-13f3-4056-a30c-0b575a8a96d8`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00245118141174', 'lastCheck': 1354523518.2600031, 'code': 0, 'valid': True}} >Thread-10401::DEBUG::2012-12-03 10:32:06,159::task::568::TaskManager.Task::(_updateState) Task=`b46de16a-13f3-4056-a30c-0b575a8a96d8`::moving from state preparing -> state finished >Thread-10401::DEBUG::2012-12-03 10:32:06,159::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10401::DEBUG::2012-12-03 10:32:06,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10401::DEBUG::2012-12-03 10:32:06,159::task::957::TaskManager.Task::(_decref) Task=`b46de16a-13f3-4056-a30c-0b575a8a96d8`::ref 0 aborting False >Thread-10407::DEBUG::2012-12-03 10:32:17,090::task::568::TaskManager.Task::(_updateState) Task=`82117537-68a9-4e34-8070-1a9e6f57fb0e`::moving from state init -> state preparing >Thread-10407::INFO::2012-12-03 10:32:17,090::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10407::INFO::2012-12-03 10:32:17,090::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00214791297913', 'lastCheck': 1354523528.277251, 'code': 0, 'valid': True}} >Thread-10407::DEBUG::2012-12-03 10:32:17,090::task::1151::TaskManager.Task::(prepare) Task=`82117537-68a9-4e34-8070-1a9e6f57fb0e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00214791297913', 'lastCheck': 1354523528.277251, 'code': 0, 'valid': True}} >Thread-10407::DEBUG::2012-12-03 10:32:17,091::task::568::TaskManager.Task::(_updateState) Task=`82117537-68a9-4e34-8070-1a9e6f57fb0e`::moving from state preparing -> state finished >Thread-10407::DEBUG::2012-12-03 10:32:17,091::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10407::DEBUG::2012-12-03 10:32:17,091::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10407::DEBUG::2012-12-03 10:32:17,091::task::957::TaskManager.Task::(_decref) Task=`82117537-68a9-4e34-8070-1a9e6f57fb0e`::ref 0 aborting False >Thread-10413::DEBUG::2012-12-03 10:32:28,074::task::568::TaskManager.Task::(_updateState) Task=`32211c2d-969f-4234-85d7-96018c5ebe65`::moving from state init -> state preparing >Thread-10413::INFO::2012-12-03 10:32:28,074::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10413::INFO::2012-12-03 10:32:28,074::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00249695777893', 'lastCheck': 1354523538.290421, 'code': 0, 'valid': True}} >Thread-10413::DEBUG::2012-12-03 10:32:28,074::task::1151::TaskManager.Task::(prepare) Task=`32211c2d-969f-4234-85d7-96018c5ebe65`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00249695777893', 'lastCheck': 1354523538.290421, 'code': 0, 'valid': True}} >Thread-10413::DEBUG::2012-12-03 10:32:28,074::task::568::TaskManager.Task::(_updateState) Task=`32211c2d-969f-4234-85d7-96018c5ebe65`::moving from state preparing -> state finished >Thread-10413::DEBUG::2012-12-03 10:32:28,074::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10413::DEBUG::2012-12-03 10:32:28,075::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10413::DEBUG::2012-12-03 10:32:28,075::task::957::TaskManager.Task::(_decref) Task=`32211c2d-969f-4234-85d7-96018c5ebe65`::ref 0 aborting False >Thread-10419::DEBUG::2012-12-03 10:32:39,009::task::568::TaskManager.Task::(_updateState) Task=`107b664b-72e6-49a7-bd01-00a9e1496cc0`::moving from state init -> state preparing >Thread-10419::INFO::2012-12-03 10:32:39,009::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10419::INFO::2012-12-03 10:32:39,009::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00484490394592', 'lastCheck': 1354523558.319864, 'code': 0, 'valid': True}} >Thread-10419::DEBUG::2012-12-03 10:32:39,010::task::1151::TaskManager.Task::(prepare) Task=`107b664b-72e6-49a7-bd01-00a9e1496cc0`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00484490394592', 'lastCheck': 1354523558.319864, 'code': 0, 'valid': True}} >Thread-10419::DEBUG::2012-12-03 10:32:39,010::task::568::TaskManager.Task::(_updateState) Task=`107b664b-72e6-49a7-bd01-00a9e1496cc0`::moving from state preparing -> state finished >Thread-10419::DEBUG::2012-12-03 10:32:39,010::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10419::DEBUG::2012-12-03 10:32:39,010::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10419::DEBUG::2012-12-03 10:32:39,010::task::957::TaskManager.Task::(_decref) Task=`107b664b-72e6-49a7-bd01-00a9e1496cc0`::ref 0 aborting False >Thread-10425::DEBUG::2012-12-03 10:32:49,925::task::568::TaskManager.Task::(_updateState) Task=`a90186bb-8ac7-4d0b-8445-e419f76b7eb3`::moving from state init -> state preparing >Thread-10425::INFO::2012-12-03 10:32:49,925::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10425::INFO::2012-12-03 10:32:49,925::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00224685668945', 'lastCheck': 1354523568.333277, 'code': 0, 'valid': True}} >Thread-10425::DEBUG::2012-12-03 10:32:49,926::task::1151::TaskManager.Task::(prepare) Task=`a90186bb-8ac7-4d0b-8445-e419f76b7eb3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00224685668945', 'lastCheck': 1354523568.333277, 'code': 0, 'valid': True}} >Thread-10425::DEBUG::2012-12-03 10:32:49,926::task::568::TaskManager.Task::(_updateState) Task=`a90186bb-8ac7-4d0b-8445-e419f76b7eb3`::moving from state preparing -> state finished >Thread-10425::DEBUG::2012-12-03 10:32:49,926::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10425::DEBUG::2012-12-03 10:32:49,926::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10425::DEBUG::2012-12-03 10:32:49,926::task::957::TaskManager.Task::(_decref) Task=`a90186bb-8ac7-4d0b-8445-e419f76b7eb3`::ref 0 aborting False >Thread-10431::DEBUG::2012-12-03 10:33:00,844::task::568::TaskManager.Task::(_updateState) Task=`6ad7de6c-bc05-45a4-9ee8-112054da0557`::moving from state init -> state preparing >Thread-10431::INFO::2012-12-03 10:33:00,845::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10431::INFO::2012-12-03 10:33:00,845::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0030300617218', 'lastCheck': 1354523578.346278, 'code': 0, 'valid': True}} >Thread-10431::DEBUG::2012-12-03 10:33:00,845::task::1151::TaskManager.Task::(prepare) Task=`6ad7de6c-bc05-45a4-9ee8-112054da0557`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0030300617218', 'lastCheck': 1354523578.346278, 'code': 0, 'valid': True}} >Thread-10431::DEBUG::2012-12-03 10:33:00,845::task::568::TaskManager.Task::(_updateState) Task=`6ad7de6c-bc05-45a4-9ee8-112054da0557`::moving from state preparing -> state finished >Thread-10431::DEBUG::2012-12-03 10:33:00,845::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10431::DEBUG::2012-12-03 10:33:00,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10431::DEBUG::2012-12-03 10:33:00,845::task::957::TaskManager.Task::(_decref) Task=`6ad7de6c-bc05-45a4-9ee8-112054da0557`::ref 0 aborting False >Thread-10437::DEBUG::2012-12-03 10:33:11,782::task::568::TaskManager.Task::(_updateState) Task=`74633cc9-818c-4563-a2cd-8615a1ee4589`::moving from state init -> state preparing >Thread-10437::INFO::2012-12-03 10:33:11,782::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10437::INFO::2012-12-03 10:33:11,782::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00282096862793', 'lastCheck': 1354523588.3649361, 'code': 0, 'valid': True}} >Thread-10437::DEBUG::2012-12-03 10:33:11,782::task::1151::TaskManager.Task::(prepare) Task=`74633cc9-818c-4563-a2cd-8615a1ee4589`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00282096862793', 'lastCheck': 1354523588.3649361, 'code': 0, 'valid': True}} >Thread-10437::DEBUG::2012-12-03 10:33:11,783::task::568::TaskManager.Task::(_updateState) Task=`74633cc9-818c-4563-a2cd-8615a1ee4589`::moving from state preparing -> state finished >Thread-10437::DEBUG::2012-12-03 10:33:11,783::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10437::DEBUG::2012-12-03 10:33:11,783::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10437::DEBUG::2012-12-03 10:33:11,783::task::957::TaskManager.Task::(_decref) Task=`74633cc9-818c-4563-a2cd-8615a1ee4589`::ref 0 aborting False >Thread-10443::DEBUG::2012-12-03 10:33:22,852::task::568::TaskManager.Task::(_updateState) Task=`9aada2f6-e5ef-4a72-b594-61464b1e9d85`::moving from state init -> state preparing >Thread-10443::INFO::2012-12-03 10:33:22,852::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10443::INFO::2012-12-03 10:33:22,852::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00215101242065', 'lastCheck': 1354523598.379333, 'code': 0, 'valid': True}} >Thread-10443::DEBUG::2012-12-03 10:33:22,852::task::1151::TaskManager.Task::(prepare) Task=`9aada2f6-e5ef-4a72-b594-61464b1e9d85`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00215101242065', 'lastCheck': 1354523598.379333, 'code': 0, 'valid': True}} >Thread-10443::DEBUG::2012-12-03 10:33:22,852::task::568::TaskManager.Task::(_updateState) Task=`9aada2f6-e5ef-4a72-b594-61464b1e9d85`::moving from state preparing -> state finished >Thread-10443::DEBUG::2012-12-03 10:33:22,853::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10443::DEBUG::2012-12-03 10:33:22,853::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10443::DEBUG::2012-12-03 10:33:22,853::task::957::TaskManager.Task::(_decref) Task=`9aada2f6-e5ef-4a72-b594-61464b1e9d85`::ref 0 aborting False >Thread-10449::DEBUG::2012-12-03 10:33:33,800::task::568::TaskManager.Task::(_updateState) Task=`326db1e2-b8c4-4987-90fb-add305dd2934`::moving from state init -> state preparing >Thread-10449::INFO::2012-12-03 10:33:33,800::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10449::INFO::2012-12-03 10:33:33,801::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00238084793091', 'lastCheck': 1354523608.3928671, 'code': 0, 'valid': True}} >Thread-10449::DEBUG::2012-12-03 10:33:33,801::task::1151::TaskManager.Task::(prepare) Task=`326db1e2-b8c4-4987-90fb-add305dd2934`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00238084793091', 'lastCheck': 1354523608.3928671, 'code': 0, 'valid': True}} >Thread-10449::DEBUG::2012-12-03 10:33:33,801::task::568::TaskManager.Task::(_updateState) Task=`326db1e2-b8c4-4987-90fb-add305dd2934`::moving from state preparing -> state finished >Thread-10449::DEBUG::2012-12-03 10:33:33,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10449::DEBUG::2012-12-03 10:33:33,801::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10449::DEBUG::2012-12-03 10:33:33,801::task::957::TaskManager.Task::(_decref) Task=`326db1e2-b8c4-4987-90fb-add305dd2934`::ref 0 aborting False >Thread-10455::DEBUG::2012-12-03 10:33:44,711::task::568::TaskManager.Task::(_updateState) Task=`98afad51-043f-4228-b446-57b7db928327`::moving from state init -> state preparing >Thread-10455::INFO::2012-12-03 10:33:44,711::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10455::INFO::2012-12-03 10:33:44,711::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00348305702209', 'lastCheck': 1354523618.4063981, 'code': 0, 'valid': True}} >Thread-10455::DEBUG::2012-12-03 10:33:44,711::task::1151::TaskManager.Task::(prepare) Task=`98afad51-043f-4228-b446-57b7db928327`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00348305702209', 'lastCheck': 1354523618.4063981, 'code': 0, 'valid': True}} >Thread-10455::DEBUG::2012-12-03 10:33:44,711::task::568::TaskManager.Task::(_updateState) Task=`98afad51-043f-4228-b446-57b7db928327`::moving from state preparing -> state finished >Thread-10455::DEBUG::2012-12-03 10:33:44,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10455::DEBUG::2012-12-03 10:33:44,712::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10455::DEBUG::2012-12-03 10:33:44,712::task::957::TaskManager.Task::(_decref) Task=`98afad51-043f-4228-b446-57b7db928327`::ref 0 aborting False >Thread-10461::DEBUG::2012-12-03 10:33:55,636::task::568::TaskManager.Task::(_updateState) Task=`2f0d6e56-0875-4423-b1d8-32d33ff40836`::moving from state init -> state preparing >Thread-10461::INFO::2012-12-03 10:33:55,636::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10461::INFO::2012-12-03 10:33:55,636::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00219702720642', 'lastCheck': 1354523628.419781, 'code': 0, 'valid': True}} >Thread-10461::DEBUG::2012-12-03 10:33:55,637::task::1151::TaskManager.Task::(prepare) Task=`2f0d6e56-0875-4423-b1d8-32d33ff40836`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00219702720642', 'lastCheck': 1354523628.419781, 'code': 0, 'valid': True}} >Thread-10461::DEBUG::2012-12-03 10:33:55,637::task::568::TaskManager.Task::(_updateState) Task=`2f0d6e56-0875-4423-b1d8-32d33ff40836`::moving from state preparing -> state finished >Thread-10461::DEBUG::2012-12-03 10:33:55,637::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10461::DEBUG::2012-12-03 10:33:55,637::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10461::DEBUG::2012-12-03 10:33:55,637::task::957::TaskManager.Task::(_decref) Task=`2f0d6e56-0875-4423-b1d8-32d33ff40836`::ref 0 aborting False >Thread-10467::DEBUG::2012-12-03 10:34:06,570::task::568::TaskManager.Task::(_updateState) Task=`6a209543-4064-4867-986a-d53057e7d3af`::moving from state init -> state preparing >Thread-10467::INFO::2012-12-03 10:34:06,570::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10467::INFO::2012-12-03 10:34:06,570::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00254988670349', 'lastCheck': 1354523638.4332681, 'code': 0, 'valid': True}} >Thread-10467::DEBUG::2012-12-03 10:34:06,571::task::1151::TaskManager.Task::(prepare) Task=`6a209543-4064-4867-986a-d53057e7d3af`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00254988670349', 'lastCheck': 1354523638.4332681, 'code': 0, 'valid': True}} >Thread-10467::DEBUG::2012-12-03 10:34:06,571::task::568::TaskManager.Task::(_updateState) Task=`6a209543-4064-4867-986a-d53057e7d3af`::moving from state preparing -> state finished >Thread-10467::DEBUG::2012-12-03 10:34:06,571::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10467::DEBUG::2012-12-03 10:34:06,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10467::DEBUG::2012-12-03 10:34:06,571::task::957::TaskManager.Task::(_decref) Task=`6a209543-4064-4867-986a-d53057e7d3af`::ref 0 aborting False >Thread-10473::DEBUG::2012-12-03 10:34:17,501::task::568::TaskManager.Task::(_updateState) Task=`a00aa7e2-6f87-4b61-9286-31fe1bb25552`::moving from state init -> state preparing >Thread-10473::INFO::2012-12-03 10:34:17,502::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10473::INFO::2012-12-03 10:34:17,502::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00213813781738', 'lastCheck': 1354523648.4492421, 'code': 0, 'valid': True}} >Thread-10473::DEBUG::2012-12-03 10:34:17,502::task::1151::TaskManager.Task::(prepare) Task=`a00aa7e2-6f87-4b61-9286-31fe1bb25552`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00213813781738', 'lastCheck': 1354523648.4492421, 'code': 0, 'valid': True}} >Thread-10473::DEBUG::2012-12-03 10:34:17,502::task::568::TaskManager.Task::(_updateState) Task=`a00aa7e2-6f87-4b61-9286-31fe1bb25552`::moving from state preparing -> state finished >Thread-10473::DEBUG::2012-12-03 10:34:17,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10473::DEBUG::2012-12-03 10:34:17,502::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10473::DEBUG::2012-12-03 10:34:17,503::task::957::TaskManager.Task::(_decref) Task=`a00aa7e2-6f87-4b61-9286-31fe1bb25552`::ref 0 aborting False >Thread-10479::DEBUG::2012-12-03 10:34:28,455::task::568::TaskManager.Task::(_updateState) Task=`94237794-1484-4a57-9282-284e38549bf4`::moving from state init -> state preparing >Thread-10479::INFO::2012-12-03 10:34:28,456::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10479::INFO::2012-12-03 10:34:28,456::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00194621086121', 'lastCheck': 1354523658.4596879, 'code': 0, 'valid': True}} >Thread-10479::DEBUG::2012-12-03 10:34:28,456::task::1151::TaskManager.Task::(prepare) Task=`94237794-1484-4a57-9282-284e38549bf4`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00194621086121', 'lastCheck': 1354523658.4596879, 'code': 0, 'valid': True}} >Thread-10479::DEBUG::2012-12-03 10:34:28,456::task::568::TaskManager.Task::(_updateState) Task=`94237794-1484-4a57-9282-284e38549bf4`::moving from state preparing -> state finished >Thread-10479::DEBUG::2012-12-03 10:34:28,456::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10479::DEBUG::2012-12-03 10:34:28,456::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10479::DEBUG::2012-12-03 10:34:28,456::task::957::TaskManager.Task::(_decref) Task=`94237794-1484-4a57-9282-284e38549bf4`::ref 0 aborting False >Thread-10485::DEBUG::2012-12-03 10:34:39,404::task::568::TaskManager.Task::(_updateState) Task=`5f976421-f3a8-4de0-bcef-bf4322464181`::moving from state init -> state preparing >Thread-10485::INFO::2012-12-03 10:34:39,404::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10485::INFO::2012-12-03 10:34:39,404::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00532197952271', 'lastCheck': 1354523678.488817, 'code': 0, 'valid': True}} >Thread-10485::DEBUG::2012-12-03 10:34:39,404::task::1151::TaskManager.Task::(prepare) Task=`5f976421-f3a8-4de0-bcef-bf4322464181`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00532197952271', 'lastCheck': 1354523678.488817, 'code': 0, 'valid': True}} >Thread-10485::DEBUG::2012-12-03 10:34:39,404::task::568::TaskManager.Task::(_updateState) Task=`5f976421-f3a8-4de0-bcef-bf4322464181`::moving from state preparing -> state finished >Thread-10485::DEBUG::2012-12-03 10:34:39,404::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10485::DEBUG::2012-12-03 10:34:39,404::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10485::DEBUG::2012-12-03 10:34:39,405::task::957::TaskManager.Task::(_decref) Task=`5f976421-f3a8-4de0-bcef-bf4322464181`::ref 0 aborting False >Thread-10491::DEBUG::2012-12-03 10:34:50,356::task::568::TaskManager.Task::(_updateState) Task=`ce204db1-1c3d-4470-83d4-16916763d07a`::moving from state init -> state preparing >Thread-10491::INFO::2012-12-03 10:34:50,356::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10491::INFO::2012-12-03 10:34:50,356::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00273489952087', 'lastCheck': 1354523688.5019169, 'code': 0, 'valid': True}} >Thread-10491::DEBUG::2012-12-03 10:34:50,357::task::1151::TaskManager.Task::(prepare) Task=`ce204db1-1c3d-4470-83d4-16916763d07a`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00273489952087', 'lastCheck': 1354523688.5019169, 'code': 0, 'valid': True}} >Thread-10491::DEBUG::2012-12-03 10:34:50,357::task::568::TaskManager.Task::(_updateState) Task=`ce204db1-1c3d-4470-83d4-16916763d07a`::moving from state preparing -> state finished >Thread-10491::DEBUG::2012-12-03 10:34:50,357::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10491::DEBUG::2012-12-03 10:34:50,357::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10491::DEBUG::2012-12-03 10:34:50,357::task::957::TaskManager.Task::(_decref) Task=`ce204db1-1c3d-4470-83d4-16916763d07a`::ref 0 aborting False >Thread-10497::DEBUG::2012-12-03 10:35:01,302::task::568::TaskManager.Task::(_updateState) Task=`17db5b6e-86fb-4433-96b2-934885cb0b9c`::moving from state init -> state preparing >Thread-10497::INFO::2012-12-03 10:35:01,302::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10497::INFO::2012-12-03 10:35:01,303::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00265598297119', 'lastCheck': 1354523698.5143931, 'code': 0, 'valid': True}} >Thread-10497::DEBUG::2012-12-03 10:35:01,303::task::1151::TaskManager.Task::(prepare) Task=`17db5b6e-86fb-4433-96b2-934885cb0b9c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00265598297119', 'lastCheck': 1354523698.5143931, 'code': 0, 'valid': True}} >Thread-10497::DEBUG::2012-12-03 10:35:01,303::task::568::TaskManager.Task::(_updateState) Task=`17db5b6e-86fb-4433-96b2-934885cb0b9c`::moving from state preparing -> state finished >Thread-10497::DEBUG::2012-12-03 10:35:01,303::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10497::DEBUG::2012-12-03 10:35:01,303::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10497::DEBUG::2012-12-03 10:35:01,303::task::957::TaskManager.Task::(_decref) Task=`17db5b6e-86fb-4433-96b2-934885cb0b9c`::ref 0 aborting False >Thread-10503::DEBUG::2012-12-03 10:35:12,291::task::568::TaskManager.Task::(_updateState) Task=`ca329599-0333-4ec4-9e41-9928cb06bf1c`::moving from state init -> state preparing >Thread-10503::INFO::2012-12-03 10:35:12,292::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10503::INFO::2012-12-03 10:35:12,292::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00242614746094', 'lastCheck': 1354523708.532356, 'code': 0, 'valid': True}} >Thread-10503::DEBUG::2012-12-03 10:35:12,292::task::1151::TaskManager.Task::(prepare) Task=`ca329599-0333-4ec4-9e41-9928cb06bf1c`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00242614746094', 'lastCheck': 1354523708.532356, 'code': 0, 'valid': True}} >Thread-10503::DEBUG::2012-12-03 10:35:12,292::task::568::TaskManager.Task::(_updateState) Task=`ca329599-0333-4ec4-9e41-9928cb06bf1c`::moving from state preparing -> state finished >Thread-10503::DEBUG::2012-12-03 10:35:12,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10503::DEBUG::2012-12-03 10:35:12,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10503::DEBUG::2012-12-03 10:35:12,292::task::957::TaskManager.Task::(_decref) Task=`ca329599-0333-4ec4-9e41-9928cb06bf1c`::ref 0 aborting False >Thread-10509::DEBUG::2012-12-03 10:35:23,324::task::568::TaskManager.Task::(_updateState) Task=`26638a78-8cb9-4d60-a117-010176bcb7a3`::moving from state init -> state preparing >Thread-10509::INFO::2012-12-03 10:35:23,324::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10509::INFO::2012-12-03 10:35:23,325::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00238490104675', 'lastCheck': 1354523718.5454791, 'code': 0, 'valid': True}} >Thread-10509::DEBUG::2012-12-03 10:35:23,325::task::1151::TaskManager.Task::(prepare) Task=`26638a78-8cb9-4d60-a117-010176bcb7a3`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00238490104675', 'lastCheck': 1354523718.5454791, 'code': 0, 'valid': True}} >Thread-10509::DEBUG::2012-12-03 10:35:23,325::task::568::TaskManager.Task::(_updateState) Task=`26638a78-8cb9-4d60-a117-010176bcb7a3`::moving from state preparing -> state finished >Thread-10509::DEBUG::2012-12-03 10:35:23,325::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10509::DEBUG::2012-12-03 10:35:23,325::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10509::DEBUG::2012-12-03 10:35:23,325::task::957::TaskManager.Task::(_decref) Task=`26638a78-8cb9-4d60-a117-010176bcb7a3`::ref 0 aborting False >Thread-10515::DEBUG::2012-12-03 10:35:34,297::task::568::TaskManager.Task::(_updateState) Task=`323c1999-c05d-41dd-bd3a-aa752090a33e`::moving from state init -> state preparing >Thread-10515::INFO::2012-12-03 10:35:34,297::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10515::INFO::2012-12-03 10:35:34,297::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00197887420654', 'lastCheck': 1354523728.5580189, 'code': 0, 'valid': True}} >Thread-10515::DEBUG::2012-12-03 10:35:34,297::task::1151::TaskManager.Task::(prepare) Task=`323c1999-c05d-41dd-bd3a-aa752090a33e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00197887420654', 'lastCheck': 1354523728.5580189, 'code': 0, 'valid': True}} >Thread-10515::DEBUG::2012-12-03 10:35:34,298::task::568::TaskManager.Task::(_updateState) Task=`323c1999-c05d-41dd-bd3a-aa752090a33e`::moving from state preparing -> state finished >Thread-10515::DEBUG::2012-12-03 10:35:34,298::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10515::DEBUG::2012-12-03 10:35:34,298::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10515::DEBUG::2012-12-03 10:35:34,298::task::957::TaskManager.Task::(_decref) Task=`323c1999-c05d-41dd-bd3a-aa752090a33e`::ref 0 aborting False >Thread-10521::DEBUG::2012-12-03 10:35:45,236::task::568::TaskManager.Task::(_updateState) Task=`8667e359-f4f8-4f37-a3b4-6c7aaec1b196`::moving from state init -> state preparing >Thread-10521::INFO::2012-12-03 10:35:45,236::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10521::INFO::2012-12-03 10:35:45,236::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00482392311096', 'lastCheck': 1354523738.574688, 'code': 0, 'valid': True}} >Thread-10521::DEBUG::2012-12-03 10:35:45,237::task::1151::TaskManager.Task::(prepare) Task=`8667e359-f4f8-4f37-a3b4-6c7aaec1b196`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00482392311096', 'lastCheck': 1354523738.574688, 'code': 0, 'valid': True}} >Thread-10521::DEBUG::2012-12-03 10:35:45,237::task::568::TaskManager.Task::(_updateState) Task=`8667e359-f4f8-4f37-a3b4-6c7aaec1b196`::moving from state preparing -> state finished >Thread-10521::DEBUG::2012-12-03 10:35:45,237::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10521::DEBUG::2012-12-03 10:35:45,237::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10521::DEBUG::2012-12-03 10:35:45,237::task::957::TaskManager.Task::(_decref) Task=`8667e359-f4f8-4f37-a3b4-6c7aaec1b196`::ref 0 aborting False >Thread-10527::DEBUG::2012-12-03 10:35:56,246::task::568::TaskManager.Task::(_updateState) Task=`cd76c4ef-de99-4017-813d-7dc2f18783b0`::moving from state init -> state preparing >Thread-10527::INFO::2012-12-03 10:35:56,246::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10527::INFO::2012-12-03 10:35:56,246::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00351309776306', 'lastCheck': 1354523748.5896349, 'code': 0, 'valid': True}} >Thread-10527::DEBUG::2012-12-03 10:35:56,246::task::1151::TaskManager.Task::(prepare) Task=`cd76c4ef-de99-4017-813d-7dc2f18783b0`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00351309776306', 'lastCheck': 1354523748.5896349, 'code': 0, 'valid': True}} >Thread-10527::DEBUG::2012-12-03 10:35:56,247::task::568::TaskManager.Task::(_updateState) Task=`cd76c4ef-de99-4017-813d-7dc2f18783b0`::moving from state preparing -> state finished >Thread-10527::DEBUG::2012-12-03 10:35:56,247::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10527::DEBUG::2012-12-03 10:35:56,247::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10527::DEBUG::2012-12-03 10:35:56,247::task::957::TaskManager.Task::(_decref) Task=`cd76c4ef-de99-4017-813d-7dc2f18783b0`::ref 0 aborting False >Thread-10533::DEBUG::2012-12-03 10:36:07,214::task::568::TaskManager.Task::(_updateState) Task=`f6b35b1f-bf95-4529-b2e1-d044c009677e`::moving from state init -> state preparing >Thread-10533::INFO::2012-12-03 10:36:07,214::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10533::INFO::2012-12-03 10:36:07,214::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00235080718994', 'lastCheck': 1354523758.6024029, 'code': 0, 'valid': True}} >Thread-10533::DEBUG::2012-12-03 10:36:07,214::task::1151::TaskManager.Task::(prepare) Task=`f6b35b1f-bf95-4529-b2e1-d044c009677e`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00235080718994', 'lastCheck': 1354523758.6024029, 'code': 0, 'valid': True}} >Thread-10533::DEBUG::2012-12-03 10:36:07,214::task::568::TaskManager.Task::(_updateState) Task=`f6b35b1f-bf95-4529-b2e1-d044c009677e`::moving from state preparing -> state finished >Thread-10533::DEBUG::2012-12-03 10:36:07,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10533::DEBUG::2012-12-03 10:36:07,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10533::DEBUG::2012-12-03 10:36:07,215::task::957::TaskManager.Task::(_decref) Task=`f6b35b1f-bf95-4529-b2e1-d044c009677e`::ref 0 aborting False >Thread-10539::DEBUG::2012-12-03 10:36:18,343::task::568::TaskManager.Task::(_updateState) Task=`08a42b84-22fa-42b8-be89-1c318bd84df6`::moving from state init -> state preparing >Thread-10539::INFO::2012-12-03 10:36:18,343::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10539::INFO::2012-12-03 10:36:18,343::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0021378993988', 'lastCheck': 1354523768.6197569, 'code': 0, 'valid': True}} >Thread-10539::DEBUG::2012-12-03 10:36:18,343::task::1151::TaskManager.Task::(prepare) Task=`08a42b84-22fa-42b8-be89-1c318bd84df6`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0021378993988', 'lastCheck': 1354523768.6197569, 'code': 0, 'valid': True}} >Thread-10539::DEBUG::2012-12-03 10:36:18,344::task::568::TaskManager.Task::(_updateState) Task=`08a42b84-22fa-42b8-be89-1c318bd84df6`::moving from state preparing -> state finished >Thread-10539::DEBUG::2012-12-03 10:36:18,344::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10539::DEBUG::2012-12-03 10:36:18,344::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10539::DEBUG::2012-12-03 10:36:18,344::task::957::TaskManager.Task::(_decref) Task=`08a42b84-22fa-42b8-be89-1c318bd84df6`::ref 0 aborting False >Thread-10545::DEBUG::2012-12-03 10:36:29,601::task::568::TaskManager.Task::(_updateState) Task=`2e00c79e-f7a7-40fc-a9c3-40efc0ed9aad`::moving from state init -> state preparing >Thread-10545::INFO::2012-12-03 10:36:29,601::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10545::INFO::2012-12-03 10:36:29,602::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00331783294678', 'lastCheck': 1354523788.645371, 'code': 0, 'valid': True}} >Thread-10545::DEBUG::2012-12-03 10:36:29,602::task::1151::TaskManager.Task::(prepare) Task=`2e00c79e-f7a7-40fc-a9c3-40efc0ed9aad`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.00331783294678', 'lastCheck': 1354523788.645371, 'code': 0, 'valid': True}} >Thread-10545::DEBUG::2012-12-03 10:36:29,602::task::568::TaskManager.Task::(_updateState) Task=`2e00c79e-f7a7-40fc-a9c3-40efc0ed9aad`::moving from state preparing -> state finished >Thread-10545::DEBUG::2012-12-03 10:36:29,602::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10545::DEBUG::2012-12-03 10:36:29,602::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10545::DEBUG::2012-12-03 10:36:29,602::task::957::TaskManager.Task::(_decref) Task=`2e00c79e-f7a7-40fc-a9c3-40efc0ed9aad`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 10:36:38,645::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:36:38,645::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:36:38,646::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:36:38,851::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:36:38,852::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:36:38,852::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-10593::WARNING::2012-12-03 10:36:38,872::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10600::WARNING::2012-12-03 10:36:38,875::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10601::WARNING::2012-12-03 10:36:38,876::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10603::WARNING::2012-12-03 10:36:38,877::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10605::WARNING::2012-12-03 10:36:38,878::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10606::WARNING::2012-12-03 10:36:38,879::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10607::WARNING::2012-12-03 10:36:38,879::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10608::WARNING::2012-12-03 10:36:38,880::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10609::WARNING::2012-12-03 10:36:38,880::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10610::WARNING::2012-12-03 10:36:38,881::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10611::WARNING::2012-12-03 10:36:38,881::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-10612::WARNING::2012-12-03 10:36:38,881::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 10:36:38,897::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:36:38,897::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:36:38,902::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:36:38,905::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:36:38,905::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:36:38,905::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-10613::DEBUG::2012-12-03 10:36:40,883::task::568::TaskManager.Task::(_updateState) Task=`9651d86c-2313-4c89-807b-217f567fc923`::moving from state init -> state preparing >Thread-10613::INFO::2012-12-03 10:36:40,883::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-10613::INFO::2012-12-03 10:36:40,883::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0014910697937', 'lastCheck': 1354523798.9163129, 'code': 0, 'valid': True}} >Thread-10613::DEBUG::2012-12-03 10:36:40,883::task::1151::TaskManager.Task::(prepare) Task=`9651d86c-2313-4c89-807b-217f567fc923`::finished: {'a00efcc1-5685-4dbe-ae70-440e9f403b76': {'delay': '0.0014910697937', 'lastCheck': 1354523798.9163129, 'code': 0, 'valid': True}} >Thread-10613::DEBUG::2012-12-03 10:36:40,883::task::568::TaskManager.Task::(_updateState) Task=`9651d86c-2313-4c89-807b-217f567fc923`::moving from state preparing -> state finished >Thread-10613::DEBUG::2012-12-03 10:36:40,883::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-10613::DEBUG::2012-12-03 10:36:40,883::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-10613::DEBUG::2012-12-03 10:36:40,884::task::957::TaskManager.Task::(_decref) Task=`9651d86c-2313-4c89-807b-217f567fc923`::ref 0 aborting False >Thread-276::DEBUG::2012-12-03 10:41:39,360::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:41:39,360::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:41:39,360::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:41:39,567::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:41:39,568::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:41:39,569::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:41:39,606::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:41:39,606::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:41:39,610::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:41:39,612::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:41:39,612::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:41:39,612::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:46:40,076::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:46:40,077::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:46:40,080::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:46:40,297::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:46:40,300::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:46:40,301::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:46:40,339::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:46:40,339::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:46:40,343::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:46:40,345::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:46:40,346::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:46:40,346::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:51:40,750::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:51:40,750::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:51:40,751::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:51:40,965::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:51:40,967::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:51:40,967::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:51:41,006::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:51:41,006::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:51:41,010::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:51:41,013::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:51:41,013::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:51:41,013::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 10:56:41,471::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:56:41,471::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 10:56:41,471::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 10:56:41,678::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 10:56:41,680::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 10:56:41,680::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 10:56:41,716::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 10:56:41,716::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 10:56:41,719::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 10:56:41,721::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 10:56:41,721::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 10:56:41,722::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:01:42,174::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:01:42,175::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:01:42,178::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:01:42,399::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:01:42,402::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:01:42,402::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:01:42,438::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:01:42,439::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:01:42,442::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:01:42,445::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:01:42,445::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:01:42,446::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:06:42,878::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:06:42,878::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:06:42,879::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:06:43,094::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:06:43,096::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:06:43,096::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:06:43,134::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:06:43,134::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:06:43,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:06:43,140::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:06:43,140::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:06:43,140::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:11:43,552::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:11:43,553::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:11:43,553::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:11:43,756::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:11:43,757::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:11:43,758::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:11:43,794::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:11:43,794::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:11:43,796::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:11:43,799::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:11:43,799::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:11:43,799::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:16:44,264::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:16:44,265::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:16:44,265::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:16:44,471::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:16:44,472::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:16:44,473::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11107::WARNING::2012-12-03 11:16:44,499::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 11:16:44,510::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:16:44,510::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:16:44,514::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:16:44,517::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:16:44,517::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:16:44,517::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:21:44,951::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:21:44,952::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:21:44,952::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:21:45,156::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:21:45,157::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:21:45,157::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11173::WARNING::2012-12-03 11:21:45,185::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 11:21:45,194::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:21:45,195::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:21:45,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:21:45,200::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:21:45,200::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:21:45,200::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:26:45,620::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:26:45,620::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:26:45,621::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:26:45,827::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:26:45,829::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:26:45,829::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11214::WARNING::2012-12-03 11:26:45,848::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11215::WARNING::2012-12-03 11:26:45,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11217::WARNING::2012-12-03 11:26:45,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11222::WARNING::2012-12-03 11:26:45,852::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11223::WARNING::2012-12-03 11:26:45,852::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11226::WARNING::2012-12-03 11:26:45,854::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11229::WARNING::2012-12-03 11:26:45,855::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11228::WARNING::2012-12-03 11:26:45,855::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11230::WARNING::2012-12-03 11:26:45,856::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11231::WARNING::2012-12-03 11:26:45,856::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11232::WARNING::2012-12-03 11:26:45,857::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 11:26:45,875::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:26:45,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:26:45,877::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:26:45,880::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:26:45,880::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:26:45,880::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:31:46,278::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:31:46,278::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:31:46,279::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:31:46,483::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:31:46,485::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:31:46,485::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11294::WARNING::2012-12-03 11:31:46,511::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11295::WARNING::2012-12-03 11:31:46,512::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 11:31:46,522::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:31:46,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:31:46,524::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:31:46,527::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:31:46,527::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:31:46,527::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:36:46,961::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:36:46,961::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:36:46,961::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:36:47,166::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:36:47,168::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:36:47,168::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11353::WARNING::2012-12-03 11:36:47,194::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 11:36:47,207::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:36:47,208::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:36:47,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:36:47,214::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:36:47,214::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:36:47,215::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:41:47,627::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:41:47,627::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:41:47,628::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:41:47,834::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:41:47,835::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:41:47,835::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:41:47,872::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:41:47,873::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:41:47,877::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:41:47,879::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:41:47,880::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:41:47,880::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:46:48,324::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:46:48,324::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:46:48,325::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:46:48,532::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:46:48,533::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:46:48,534::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:46:48,572::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:46:48,572::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:46:48,576::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:46:48,578::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:46:48,579::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:46:48,579::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:51:49,034::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:51:49,035::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:51:49,035::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:51:49,243::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:51:49,245::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:51:49,245::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:51:49,283::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:51:49,283::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:51:49,287::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:51:49,289::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:51:49,290::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:51:49,290::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 11:56:49,707::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:56:49,708::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 11:56:49,708::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 11:56:49,916::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 11:56:49,917::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 11:56:49,917::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 11:56:49,954::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 11:56:49,954::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 11:56:49,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 11:56:49,960::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 11:56:49,960::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 11:56:49,960::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:01:50,379::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:01:50,380::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:01:50,383::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:01:50,613::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:01:50,615::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:01:50,616::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:01:50,653::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:01:50,654::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:01:50,657::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:01:50,660::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:01:50,660::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:01:50,660::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:06:51,106::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:06:51,107::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:06:51,107::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:06:51,314::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:06:51,316::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:06:51,316::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11729::WARNING::2012-12-03 12:06:51,344::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11731::WARNING::2012-12-03 12:06:51,344::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 12:06:51,354::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:06:51,354::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:06:51,357::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:06:51,360::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:06:51,360::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:06:51,360::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:11:51,805::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:11:51,805::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:11:51,805::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:11:52,016::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:11:52,018::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:11:52,018::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:11:52,054::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:11:52,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:11:52,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:11:52,059::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:11:52,059::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:11:52,059::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:16:52,473::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:16:52,474::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:16:52,474::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:16:52,682::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:16:52,684::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:16:52,684::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-11834::WARNING::2012-12-03 12:16:52,702::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11835::WARNING::2012-12-03 12:16:52,703::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11836::WARNING::2012-12-03 12:16:52,703::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11837::WARNING::2012-12-03 12:16:52,703::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11839::WARNING::2012-12-03 12:16:52,704::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11841::WARNING::2012-12-03 12:16:52,706::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11842::WARNING::2012-12-03 12:16:52,707::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11843::WARNING::2012-12-03 12:16:52,707::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11844::WARNING::2012-12-03 12:16:52,708::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11845::WARNING::2012-12-03 12:16:52,708::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-11846::WARNING::2012-12-03 12:16:52,709::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 12:16:52,722::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:16:52,722::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:16:52,725::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:16:52,727::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:16:52,728::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:16:52,728::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:21:53,183::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:21:53,183::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:21:53,184::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:21:53,393::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:21:53,395::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:21:53,395::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:21:53,433::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:21:53,433::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:21:53,437::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:21:53,439::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:21:53,439::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:21:53,440::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:26:53,830::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:26:53,830::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:26:53,830::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:26:54,037::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:26:54,039::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:26:54,039::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:26:54,077::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:26:54,077::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:26:54,080::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:26:54,083::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:26:54,083::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:26:54,083::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:31:54,490::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:31:54,490::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:31:54,490::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:31:54,696::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:31:54,698::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:31:54,698::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:31:54,733::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:31:54,734::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:31:54,736::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:31:54,738::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:31:54,739::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:31:54,739::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:36:55,162::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:36:55,162::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:36:55,162::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:36:55,367::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:36:55,369::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:36:55,369::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:36:55,406::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:36:55,407::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:36:55,410::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:36:55,413::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:36:55,413::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:36:55,413::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:41:55,814::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:41:55,815::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:41:55,815::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:41:56,021::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:41:56,023::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:41:56,023::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:41:56,060::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:41:56,060::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:41:56,063::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:41:56,065::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:41:56,065::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:41:56,065::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:46:56,464::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:46:56,464::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:46:56,465::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:46:56,671::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:46:56,672::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:46:56,673::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:46:56,709::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:46:56,709::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:46:56,712::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:46:56,714::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:46:56,714::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:46:56,714::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:51:57,136::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:51:57,136::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:51:57,137::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:51:57,341::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:51:57,343::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:51:57,343::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 12:51:57,380::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:51:57,380::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:51:57,383::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:51:57,385::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:51:57,385::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:51:57,385::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 12:56:57,783::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:56:57,783::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 12:56:57,784::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 12:56:57,987::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 12:56:57,989::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 12:56:57,989::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-12344::WARNING::2012-12-03 12:56:58,014::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-12348::WARNING::2012-12-03 12:56:58,016::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-12349::WARNING::2012-12-03 12:56:58,016::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 12:56:58,027::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 12:56:58,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 12:56:58,031::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 12:56:58,033::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 12:56:58,033::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 12:56:58,033::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:01:58,458::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:01:58,458::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:01:58,461::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:01:58,680::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:01:58,682::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:01:58,683::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:01:58,721::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:01:58,721::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:01:58,725::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:01:58,731::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:01:58,731::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:01:58,731::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:06:59,204::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:06:59,204::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:06:59,205::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:06:59,413::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:06:59,414::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:06:59,415::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:06:59,450::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:06:59,450::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:06:59,453::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:06:59,455::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:06:59,455::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:06:59,455::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:11:59,856::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:11:59,857::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:11:59,857::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:12:00,064::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:12:00,066::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:12:00,066::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-12516::WARNING::2012-12-03 13:12:00,084::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-12517::WARNING::2012-12-03 13:12:00,084::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-12519::WARNING::2012-12-03 13:12:00,086::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-12521::WARNING::2012-12-03 13:12:00,087::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 13:12:00,103::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:12:00,103::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:12:00,105::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:12:00,108::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:12:00,108::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:12:00,108::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:17:00,543::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:17:00,543::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:17:00,544::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:17:00,754::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:17:00,755::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:17:00,756::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:17:00,793::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:17:00,793::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:17:00,796::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:17:00,798::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:17:00,798::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:17:00,798::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:22:01,229::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:22:01,229::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:22:01,230::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:22:01,433::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:22:01,435::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:22:01,435::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:22:01,472::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:22:01,472::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:22:01,475::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:22:01,478::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:22:01,478::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:22:01,479::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:27:01,876::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:27:01,876::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:27:01,877::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:27:02,084::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:27:02,086::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:27:02,086::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-12720::WARNING::2012-12-03 13:27:02,113::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 13:27:02,123::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:27:02,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:27:02,127::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:27:02,130::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:27:02,130::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:27:02,130::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:32:02,524::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:32:02,524::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:32:02,525::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:32:02,729::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:32:02,730::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:32:02,731::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-12782::WARNING::2012-12-03 13:32:02,757::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 13:32:02,770::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:32:02,770::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:32:02,773::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:32:02,776::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:32:02,777::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:32:02,777::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:37:03,193::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:37:03,193::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:37:03,194::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:37:03,397::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:37:03,398::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:37:03,398::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:37:03,436::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:37:03,436::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:37:03,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:37:03,444::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:37:03,444::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:37:03,444::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:42:03,830::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:42:03,831::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:42:03,834::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:42:04,093::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:42:04,095::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:42:04,095::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-12906::WARNING::2012-12-03 13:42:04,120::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 13:42:04,130::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:42:04,131::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:42:04,133::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:42:04,136::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:42:04,136::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:42:04,136::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:47:04,543::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:47:04,543::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:47:04,543::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:47:04,754::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:47:04,756::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:47:04,756::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:47:04,793::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:47:04,793::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:47:04,796::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:47:04,798::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:47:04,798::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:47:04,798::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:52:05,217::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:52:05,217::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:52:05,217::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:52:05,424::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:52:05,425::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:52:05,426::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:52:05,464::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:52:05,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:52:05,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:52:05,470::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:52:05,471::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:52:05,471::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 13:57:05,949::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:57:05,949::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 13:57:05,950::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 13:57:06,159::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 13:57:06,161::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 13:57:06,161::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 13:57:06,199::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 13:57:06,200::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 13:57:06,203::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 13:57:06,205::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 13:57:06,206::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 13:57:06,206::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:02:06,590::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:02:06,591::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:02:06,594::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:02:06,820::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:02:06,822::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:02:06,822::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-13144::WARNING::2012-12-03 14:02:06,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13145::WARNING::2012-12-03 14:02:06,845::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13146::WARNING::2012-12-03 14:02:06,846::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 14:02:06,860::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:02:06,860::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:02:06,863::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:02:06,865::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:02:06,865::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:02:06,865::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:07:07,272::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:07:07,272::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:07:07,273::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:07:07,481::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:07:07,483::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:07:07,483::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:07:07,519::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:07:07,520::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:07:07,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:07:07,524::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:07:07,524::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:07:07,525::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:12:07,952::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:12:07,953::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:12:07,953::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:12:08,161::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:12:08,163::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:12:08,164::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:12:08,200::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:12:08,201::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:12:08,204::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:12:08,206::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:12:08,207::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:12:08,207::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:17:08,609::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:17:08,609::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:17:08,609::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:17:08,820::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:17:08,822::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:17:08,822::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-13322::WARNING::2012-12-03 14:17:08,840::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13324::WARNING::2012-12-03 14:17:08,841::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13326::WARNING::2012-12-03 14:17:08,842::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13328::WARNING::2012-12-03 14:17:08,843::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13329::WARNING::2012-12-03 14:17:08,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13330::WARNING::2012-12-03 14:17:08,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13331::WARNING::2012-12-03 14:17:08,845::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13332::WARNING::2012-12-03 14:17:08,846::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13333::WARNING::2012-12-03 14:17:08,846::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13334::WARNING::2012-12-03 14:17:08,847::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13335::WARNING::2012-12-03 14:17:08,847::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13336::WARNING::2012-12-03 14:17:08,848::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13337::WARNING::2012-12-03 14:17:08,848::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13338::WARNING::2012-12-03 14:17:08,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13339::WARNING::2012-12-03 14:17:08,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13340::WARNING::2012-12-03 14:17:08,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13341::WARNING::2012-12-03 14:17:08,850::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13342::WARNING::2012-12-03 14:17:08,851::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13343::WARNING::2012-12-03 14:17:08,852::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 14:17:08,908::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:17:08,908::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:17:09,117::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:17:09,118::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:17:09,119::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:17:09,153::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:17:09,154::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:17:09,157::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:17:09,160::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:17:09,160::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:17:09,160::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-03 14:17:09,160::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-03 14:17:09,160::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-03 14:17:19,173::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-03 14:22:09,540::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:22:09,541::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:22:09,541::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:22:09,747::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:22:09,748::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:22:09,748::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:22:09,784::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:22:09,784::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:22:09,787::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:22:09,790::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:22:09,790::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:22:09,790::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:27:10,234::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:27:10,234::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:27:10,234::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:27:10,440::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:27:10,442::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:27:10,442::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:27:10,479::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:27:10,479::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:27:10,482::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:27:10,484::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:27:10,484::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:27:10,484::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:32:10,878::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:32:10,879::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:32:10,879::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:32:11,087::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:32:11,088::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:32:11,088::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-13572::WARNING::2012-12-03 14:32:11,107::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13580::WARNING::2012-12-03 14:32:11,111::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13579::WARNING::2012-12-03 14:32:11,112::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-13591::WARNING::2012-12-03 14:32:11,117::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 14:32:11,128::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:32:11,128::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:32:11,132::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:32:11,134::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:32:11,134::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:32:11,134::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:37:11,526::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:37:11,526::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:37:11,527::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:37:11,734::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:37:11,735::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:37:11,736::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:37:11,770::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:37:11,770::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:37:11,773::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:37:11,775::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:37:11,775::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:37:11,776::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:42:12,261::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:42:12,261::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:42:12,262::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:42:12,469::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:42:12,470::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:42:12,471::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:42:12,507::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:42:12,507::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:42:12,510::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:42:12,513::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:42:12,513::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:42:12,513::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:47:12,932::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:47:12,932::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:47:12,932::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:47:13,135::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:47:13,136::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:47:13,137::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:47:13,175::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:47:13,176::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:47:13,179::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:47:13,182::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:47:13,182::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:47:13,182::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:52:13,580::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:52:13,580::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:52:13,580::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:52:13,784::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:52:13,785::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:52:13,786::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:52:13,822::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:52:13,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:52:13,831::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:52:13,834::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:52:13,834::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:52:13,834::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 14:57:14,301::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:57:14,301::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 14:57:14,302::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 14:57:14,505::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 14:57:14,507::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 14:57:14,507::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 14:57:14,543::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 14:57:14,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 14:57:14,546::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 14:57:14,549::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 14:57:14,549::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 14:57:14,549::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:02:14,962::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:02:14,962::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:02:14,965::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:02:15,184::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:02:15,186::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:02:15,186::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:02:15,225::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:02:15,226::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:02:15,229::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:02:15,232::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:02:15,232::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:02:15,232::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:07:15,632::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:07:15,632::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:07:15,633::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:07:15,838::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:07:15,839::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:07:15,840::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:07:15,876::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:07:15,876::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:07:15,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:07:15,881::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:07:15,881::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:07:15,882::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:12:16,310::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:12:16,311::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:12:16,311::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:12:16,524::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:12:16,526::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:12:16,526::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:12:16,567::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:12:16,567::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:12:16,570::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:12:16,573::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:12:16,573::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:12:16,574::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:17:16,961::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:17:16,961::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:17:16,961::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:17:17,165::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:17:17,167::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:17:17,167::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:17:17,204::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:17:17,204::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:17:17,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:17:17,209::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:17:17,210::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:17:17,210::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:22:17,601::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:22:17,601::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:22:17,602::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:22:17,806::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:22:17,807::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:22:17,807::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14204::WARNING::2012-12-03 15:22:17,833::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14205::WARNING::2012-12-03 15:22:17,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14207::WARNING::2012-12-03 15:22:17,835::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14209::WARNING::2012-12-03 15:22:17,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14210::WARNING::2012-12-03 15:22:17,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14208::WARNING::2012-12-03 15:22:17,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14211::WARNING::2012-12-03 15:22:17,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 15:22:17,861::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:22:17,861::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:22:17,864::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:22:17,866::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:22:17,866::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:22:17,866::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:27:18,295::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:27:18,296::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:27:18,296::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:27:18,501::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:27:18,502::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:27:18,503::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14264::WARNING::2012-12-03 15:27:18,526::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14266::WARNING::2012-12-03 15:27:18,527::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14267::WARNING::2012-12-03 15:27:18,527::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14268::WARNING::2012-12-03 15:27:18,528::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14269::WARNING::2012-12-03 15:27:18,528::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14270::WARNING::2012-12-03 15:27:18,529::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14271::WARNING::2012-12-03 15:27:18,529::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14272::WARNING::2012-12-03 15:27:18,530::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 15:27:18,548::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:27:18,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:27:18,551::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:27:18,556::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:27:18,556::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:27:18,556::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:32:19,044::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:32:19,044::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:32:19,045::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:32:19,250::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:32:19,252::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:32:19,252::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:32:19,292::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:32:19,292::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:32:19,296::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:32:19,299::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:32:19,299::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:32:19,299::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:37:19,712::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:37:19,712::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:37:19,713::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:37:19,917::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:37:19,918::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:37:19,919::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14391::WARNING::2012-12-03 15:37:19,944::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14392::WARNING::2012-12-03 15:37:19,944::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14394::WARNING::2012-12-03 15:37:19,945::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 15:37:19,961::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:37:19,962::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:37:19,966::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:37:19,969::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:37:19,969::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:37:19,969::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:42:20,381::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:42:20,381::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:42:20,382::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:42:20,585::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:42:20,587::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:42:20,587::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:42:20,625::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:42:20,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:42:20,628::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:42:20,631::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:42:20,631::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:42:20,631::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:47:21,095::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:47:21,095::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:47:21,096::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:47:21,300::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:47:21,302::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:47:21,302::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:47:21,341::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:47:21,341::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:47:21,344::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:47:21,346::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:47:21,346::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:47:21,346::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:52:21,780::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:52:21,781::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:52:21,781::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:52:21,987::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:52:21,988::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:52:21,989::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:52:22,027::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:52:22,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:52:22,031::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:52:22,033::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:52:22,033::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:52:22,034::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 15:57:22,432::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:57:22,433::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 15:57:22,433::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 15:57:22,639::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 15:57:22,640::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 15:57:22,640::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 15:57:22,676::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 15:57:22,676::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 15:57:22,679::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 15:57:22,681::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 15:57:22,681::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 15:57:22,682::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:02:23,184::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:02:23,184::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:02:23,188::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:02:23,410::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:02:23,412::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:02:23,412::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14696::WARNING::2012-12-03 16:02:23,434::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14698::WARNING::2012-12-03 16:02:23,435::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14699::WARNING::2012-12-03 16:02:23,436::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14700::WARNING::2012-12-03 16:02:23,436::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14703::WARNING::2012-12-03 16:02:23,438::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:02:23,454::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:02:23,454::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:02:23,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:02:23,460::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:02:23,460::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:02:23,460::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:07:23,882::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:07:23,882::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:07:23,883::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:07:24,091::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:07:24,093::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:07:24,093::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14765::WARNING::2012-12-03 16:07:24,118::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:07:24,132::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:07:24,132::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:07:24,136::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:07:24,139::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:07:24,139::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:07:24,139::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:12:24,594::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:12:24,594::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:12:24,595::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:12:24,809::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:12:24,810::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:12:24,811::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14830::WARNING::2012-12-03 16:12:24,838::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14831::WARNING::2012-12-03 16:12:24,838::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:12:24,903::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:12:24,903::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:12:24,906::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:12:24,908::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:12:24,908::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:12:24,908::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:17:25,392::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:17:25,392::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:17:25,393::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:17:25,602::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:17:25,603::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:17:25,604::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 16:17:25,641::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:17:25,641::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:17:25,644::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:17:25,646::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:17:25,646::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:17:25,646::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:22:26,084::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:22:26,085::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:22:26,085::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:22:26,294::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:22:26,296::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:22:26,296::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14946::WARNING::2012-12-03 16:22:26,320::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-14953::WARNING::2012-12-03 16:22:26,324::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:22:26,338::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:22:26,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:22:26,341::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:22:26,343::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:22:26,344::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:22:26,344::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:27:26,773::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:27:26,773::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:27:26,773::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:27:26,982::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:27:26,984::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:27:26,984::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15017::WARNING::2012-12-03 16:27:27,011::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:27:27,022::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:27:27,023::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:27:27,026::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:27:27,030::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:27:27,030::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:27:27,030::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:32:27,459::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:32:27,459::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:32:27,460::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:32:27,666::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:32:27,667::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:32:27,668::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 16:32:27,705::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:32:27,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:32:27,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:32:27,712::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:32:27,712::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:32:27,712::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:37:28,154::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:37:28,154::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:37:28,155::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:37:28,366::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:37:28,368::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:37:28,368::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15138::WARNING::2012-12-03 16:37:28,393::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:37:28,405::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:37:28,405::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:37:28,409::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:37:28,412::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:37:28,412::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:37:28,412::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:42:28,820::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:42:28,821::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:42:28,821::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:42:29,025::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:42:29,027::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:42:29,027::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 16:42:29,063::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:42:29,064::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:42:29,067::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:42:29,070::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:42:29,070::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:42:29,070::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:47:29,573::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:47:29,573::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:47:29,573::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:47:29,777::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:47:29,778::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:47:29,779::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 16:47:29,816::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:47:29,816::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:47:29,819::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:47:29,822::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:47:29,822::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:47:29,822::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:52:30,275::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:52:30,275::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:52:30,275::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:52:30,485::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:52:30,487::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:52:30,487::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15324::WARNING::2012-12-03 16:52:30,513::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 16:52:30,524::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:52:30,525::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:52:30,527::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:52:30,529::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:52:30,529::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:52:30,530::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 16:57:30,952::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:57:30,952::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 16:57:30,952::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 16:57:31,158::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 16:57:31,159::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 16:57:31,160::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 16:57:31,196::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 16:57:31,196::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 16:57:31,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 16:57:31,201::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 16:57:31,201::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 16:57:31,201::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:02:31,628::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:02:31,629::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:02:31,632::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:02:31,856::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:02:31,858::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:02:31,859::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15430::WARNING::2012-12-03 17:02:31,877::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15431::WARNING::2012-12-03 17:02:31,878::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15433::WARNING::2012-12-03 17:02:31,879::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15438::WARNING::2012-12-03 17:02:31,882::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15440::WARNING::2012-12-03 17:02:31,882::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15441::WARNING::2012-12-03 17:02:31,883::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15442::WARNING::2012-12-03 17:02:31,883::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15443::WARNING::2012-12-03 17:02:31,884::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15444::WARNING::2012-12-03 17:02:31,885::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15445::WARNING::2012-12-03 17:02:31,885::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15447::WARNING::2012-12-03 17:02:31,886::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15451::WARNING::2012-12-03 17:02:31,888::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15450::WARNING::2012-12-03 17:02:31,888::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 17:02:31,902::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:02:31,902::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:02:31,905::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:02:31,911::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:02:31,912::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:02:31,912::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:07:32,334::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:07:32,335::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:07:32,335::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:07:32,542::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:07:32,543::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:07:32,543::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:07:32,581::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:07:32,581::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:07:32,585::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:07:32,588::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:07:32,588::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:07:32,588::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:12:33,037::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:12:33,037::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:12:33,038::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:12:33,245::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:12:33,247::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:12:33,247::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:12:33,286::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:12:33,286::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:12:33,289::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:12:33,292::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:12:33,292::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:12:33,293::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:17:33,692::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:17:33,692::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:17:33,692::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:17:33,895::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:17:33,897::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:17:33,897::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:17:33,933::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:17:33,933::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:17:33,936::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:17:33,939::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:17:33,939::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:17:33,939::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:22:34,439::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:22:34,439::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:22:34,440::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:22:34,643::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:22:34,644::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:22:34,645::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:22:34,684::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:22:34,684::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:22:34,688::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:22:34,690::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:22:34,690::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:22:34,691::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:27:35,157::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:27:35,157::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:27:35,157::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:27:35,362::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:27:35,363::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:27:35,363::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:27:35,400::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:27:35,400::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:27:35,403::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:27:35,405::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:27:35,405::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:27:35,406::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:32:35,807::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:32:35,807::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:32:35,808::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:32:36,017::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:32:36,019::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:32:36,019::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15820::WARNING::2012-12-03 17:32:36,045::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15822::WARNING::2012-12-03 17:32:36,047::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 17:32:36,058::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:32:36,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:32:36,062::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:32:36,064::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:32:36,064::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:32:36,064::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:37:36,474::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:37:36,475::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:37:36,475::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:37:36,685::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:37:36,687::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:37:36,687::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:37:36,724::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:37:36,725::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:37:36,727::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:37:36,730::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:37:36,730::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:37:36,730::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:42:37,215::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:42:37,216::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:42:37,219::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:42:37,432::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:42:37,434::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:42:37,434::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:42:37,471::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:42:37,471::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:42:37,474::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:42:37,476::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:42:37,476::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:42:37,477::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:47:37,884::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:47:37,884::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:47:37,885::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:47:38,094::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:47:38,095::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:47:38,096::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15984::WARNING::2012-12-03 17:47:38,113::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_iso2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15986::WARNING::2012-12-03 17:47:38,114::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15987::WARNING::2012-12-03 17:47:38,115::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15988::WARNING::2012-12-03 17:47:38,115::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15990::WARNING::2012-12-03 17:47:38,116::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15989::WARNING::2012-12-03 17:47:38,117::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15991::WARNING::2012-12-03 17:47:38,118::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15992::WARNING::2012-12-03 17:47:38,118::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15993::WARNING::2012-12-03 17:47:38,119::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15994::WARNING::2012-12-03 17:47:38,119::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15995::WARNING::2012-12-03 17:47:38,119::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15996::WARNING::2012-12-03 17:47:38,120::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15997::WARNING::2012-12-03 17:47:38,121::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15998::WARNING::2012-12-03 17:47:38,121::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-15999::WARNING::2012-12-03 17:47:38,122::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16000::WARNING::2012-12-03 17:47:38,123::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16001::WARNING::2012-12-03 17:47:38,123::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16002::WARNING::2012-12-03 17:47:38,123::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16003::WARNING::2012-12-03 17:47:38,124::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16004::WARNING::2012-12-03 17:47:38,125::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16005::WARNING::2012-12-03 17:47:38,125::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16006::WARNING::2012-12-03 17:47:38,126::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16007::WARNING::2012-12-03 17:47:38,126::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16008::WARNING::2012-12-03 17:47:38,127::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16009::WARNING::2012-12-03 17:47:38,127::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 17:47:38,131::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:47:38,132::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:47:38,342::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:47:38,343::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:47:38,344::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:47:38,380::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:47:38,380::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:47:38,383::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:47:38,385::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:47:38,385::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:47:38,385::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-03 17:47:38,386::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-03 17:47:38,386::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-03 17:47:48,400::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-03 17:52:38,773::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:52:38,774::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:52:38,774::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:52:38,977::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:52:38,978::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:52:38,979::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:52:39,016::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:52:39,017::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:52:39,020::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:52:39,023::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:52:39,023::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:52:39,023::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 17:57:39,461::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:57:39,462::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 17:57:39,462::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 17:57:39,676::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 17:57:39,678::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 17:57:39,679::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 17:57:39,714::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 17:57:39,714::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 17:57:39,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 17:57:39,719::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 17:57:39,719::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 17:57:39,719::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:02:40,157::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:02:40,157::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:02:40,158::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:02:40,374::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:02:40,376::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:02:40,376::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 18:02:40,412::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:02:40,413::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:02:40,416::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:02:40,419::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:02:40,419::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:02:40,419::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:07:40,809::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:07:40,810::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:07:40,813::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:07:41,021::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:07:41,022::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:07:41,022::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16317::WARNING::2012-12-03 18:07:41,048::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:07:41,062::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:07:41,062::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:07:41,067::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:07:41,070::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:07:41,070::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:07:41,070::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:12:41,502::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:12:41,503::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:12:41,503::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:12:41,712::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:12:41,714::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:12:41,714::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16373::WARNING::2012-12-03 18:12:41,738::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16374::WARNING::2012-12-03 18:12:41,739::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16376::WARNING::2012-12-03 18:12:41,740::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16377::WARNING::2012-12-03 18:12:41,741::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16378::WARNING::2012-12-03 18:12:41,741::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16380::WARNING::2012-12-03 18:12:41,742::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16379::WARNING::2012-12-03 18:12:41,742::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:12:41,762::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:12:41,762::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:12:41,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:12:41,774::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:12:41,774::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:12:41,774::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:17:42,232::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:17:42,233::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:17:42,233::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:17:42,437::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:17:42,438::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:17:42,439::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 18:17:42,475::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:17:42,475::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:17:42,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:17:42,484::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:17:42,484::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:17:42,484::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:22:42,906::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:22:42,907::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:22:42,907::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:22:43,113::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:22:43,114::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:22:43,114::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16502::WARNING::2012-12-03 18:22:43,142::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16503::WARNING::2012-12-03 18:22:43,142::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:22:43,154::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:22:43,155::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:22:43,159::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:22:43,161::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:22:43,162::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:22:43,162::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:27:43,582::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:27:43,582::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:27:43,583::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:27:43,786::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:27:43,788::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:27:43,788::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16547::WARNING::2012-12-03 18:27:43,807::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16555::WARNING::2012-12-03 18:27:43,811::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16559::WARNING::2012-12-03 18:27:43,813::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16561::WARNING::2012-12-03 18:27:43,813::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16562::WARNING::2012-12-03 18:27:43,814::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16564::WARNING::2012-12-03 18:27:43,815::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:27:43,829::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:27:43,829::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:27:43,832::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:27:43,834::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:27:43,834::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:27:43,834::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:32:44,284::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:32:44,284::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:32:44,285::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:32:44,494::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:32:44,496::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:32:44,496::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 18:32:44,532::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:32:44,532::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:32:44,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:32:44,538::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:32:44,539::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:32:44,539::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:37:44,953::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:37:44,953::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:37:44,954::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:37:45,155::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:37:45,156::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:37:45,156::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16686::WARNING::2012-12-03 18:37:45,181::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:37:45,196::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:37:45,196::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:37:45,199::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:37:45,202::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:37:45,202::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:37:45,202::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:42:45,614::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:42:45,614::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:42:45,614::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:42:45,814::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:42:45,815::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:42:45,816::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16729::WARNING::2012-12-03 18:42:45,833::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16732::WARNING::2012-12-03 18:42:45,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16733::WARNING::2012-12-03 18:42:45,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16735::WARNING::2012-12-03 18:42:45,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16738::WARNING::2012-12-03 18:42:45,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16737::WARNING::2012-12-03 18:42:45,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16741::WARNING::2012-12-03 18:42:45,839::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16742::WARNING::2012-12-03 18:42:45,840::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16744::WARNING::2012-12-03 18:42:45,841::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16745::WARNING::2012-12-03 18:42:45,841::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16746::WARNING::2012-12-03 18:42:45,842::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16747::WARNING::2012-12-03 18:42:45,842::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16748::WARNING::2012-12-03 18:42:45,843::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16750::WARNING::2012-12-03 18:42:45,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16749::WARNING::2012-12-03 18:42:45,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16751::WARNING::2012-12-03 18:42:45,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16752::WARNING::2012-12-03 18:42:45,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16753::WARNING::2012-12-03 18:42:45,845::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:42:45,899::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:42:45,899::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:42:46,104::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:42:46,106::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:42:46,106::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 18:42:46,143::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:42:46,143::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:42:46,146::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:42:46,148::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:42:46,148::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:42:46,148::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-03 18:42:46,148::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-03 18:42:46,149::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-03 18:42:56,164::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-03 18:47:46,569::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:47:46,569::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:47:46,569::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:47:46,772::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:47:46,773::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:47:46,774::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16870::WARNING::2012-12-03 18:47:46,798::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16872::WARNING::2012-12-03 18:47:46,799::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16873::WARNING::2012-12-03 18:47:46,799::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16874::WARNING::2012-12-03 18:47:46,800::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16875::WARNING::2012-12-03 18:47:46,800::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-16877::WARNING::2012-12-03 18:47:46,802::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:47:46,812::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:47:46,812::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:47:46,815::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:47:46,817::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:47:46,817::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:47:46,817::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:52:47,243::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:52:47,244::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:52:47,244::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:52:47,447::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:52:47,448::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:52:47,449::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 18:52:47,484::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:52:47,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:52:47,487::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:52:47,489::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:52:47,489::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:52:47,489::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 18:57:47,908::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:57:47,909::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 18:57:47,909::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 18:57:48,113::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 18:57:48,115::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 18:57:48,115::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-16999::WARNING::2012-12-03 18:57:48,141::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 18:57:48,153::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 18:57:48,153::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 18:57:48,157::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 18:57:48,160::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 18:57:48,160::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 18:57:48,160::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:02:48,587::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:02:48,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:02:48,591::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:02:48,807::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:02:48,809::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:02:48,809::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17043::WARNING::2012-12-03 19:02:48,828::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17042::WARNING::2012-12-03 19:02:48,828::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17047::WARNING::2012-12-03 19:02:48,830::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17055::WARNING::2012-12-03 19:02:48,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 19:02:48,852::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:02:48,853::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:02:48,858::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:02:48,862::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:02:48,862::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:02:48,862::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:07:49,385::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:07:49,385::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:07:49,386::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:07:49,588::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:07:49,589::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:07:49,589::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:07:49,625::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:07:49,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:07:49,628::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:07:49,630::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:07:49,631::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:07:49,631::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:12:50,064::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:12:50,064::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:12:50,065::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:12:50,275::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:12:50,277::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:12:50,278::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17186::WARNING::2012-12-03 19:12:50,306::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 19:12:50,316::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:12:50,316::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:12:50,321::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:12:50,324::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:12:50,324::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:12:50,324::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:17:50,766::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:17:50,766::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:17:50,766::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:17:50,978::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:17:50,979::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:17:50,979::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:17:51,016::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:17:51,016::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:17:51,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:17:51,021::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:17:51,021::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:17:51,021::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:22:51,438::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:22:51,439::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:22:51,439::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:22:51,644::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:22:51,646::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:22:51,646::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:22:51,683::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:22:51,683::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:22:51,686::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:22:51,688::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:22:51,688::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:22:51,688::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:27:52,230::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:27:52,231::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:27:52,231::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:27:52,435::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:27:52,436::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:27:52,437::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:27:52,473::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:27:52,474::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:27:52,477::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:27:52,480::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:27:52,480::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:27:52,480::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:32:52,915::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:32:52,915::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:32:52,916::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:32:53,118::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:32:53,120::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:32:53,120::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17428::WARNING::2012-12-03 19:32:53,144::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17431::WARNING::2012-12-03 19:32:53,146::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17435::WARNING::2012-12-03 19:32:53,148::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 19:32:53,159::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:32:53,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:32:53,162::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:32:53,165::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:32:53,165::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:32:53,165::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:37:53,585::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:37:53,586::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:37:53,586::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:37:53,790::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:37:53,792::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:37:53,792::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17488::WARNING::2012-12-03 19:37:53,816::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17489::WARNING::2012-12-03 19:37:53,817::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17491::WARNING::2012-12-03 19:37:53,818::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17492::WARNING::2012-12-03 19:37:53,819::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 19:37:53,832::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:37:53,832::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:37:53,834::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:37:53,837::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:37:53,837::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:37:53,837::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:42:54,298::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:42:54,298::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:42:54,298::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:42:54,504::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:42:54,505::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:42:54,506::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17557::WARNING::2012-12-03 19:42:54,534::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 19:42:54,544::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:42:54,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:42:54,549::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:42:54,551::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:42:54,551::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:42:54,551::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:47:55,011::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:47:55,012::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:47:55,012::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:47:55,217::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:47:55,218::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:47:55,219::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:47:55,256::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:47:55,256::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:47:55,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:47:55,262::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:47:55,262::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:47:55,262::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:52:55,764::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:52:55,764::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:52:55,765::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:52:55,967::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:52:55,969::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:52:55,969::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:52:56,003::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:52:56,003::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:52:56,005::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:52:56,008::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:52:56,008::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:52:56,008::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 19:57:56,449::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:57:56,449::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 19:57:56,449::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 19:57:56,653::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 19:57:56,655::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 19:57:56,655::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 19:57:56,692::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 19:57:56,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 19:57:56,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 19:57:56,699::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 19:57:56,699::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 19:57:56,699::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:02:57,196::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:02:57,196::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:02:57,200::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:02:57,423::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:02:57,425::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:02:57,425::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17786::WARNING::2012-12-03 20:02:57,444::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17798::WARNING::2012-12-03 20:02:57,449::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17800::WARNING::2012-12-03 20:02:57,450::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17801::WARNING::2012-12-03 20:02:57,451::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17803::WARNING::2012-12-03 20:02:57,452::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17805::WARNING::2012-12-03 20:02:57,453::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17807::WARNING::2012-12-03 20:02:57,454::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:02:57,467::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:02:57,467::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:02:57,470::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:02:57,472::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:02:57,472::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:02:57,472::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:07:57,914::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:07:57,914::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:07:57,914::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:07:58,124::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:07:58,126::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:07:58,126::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17867::WARNING::2012-12-03 20:07:58,153::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:07:58,164::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:07:58,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:07:58,168::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:07:58,171::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:07:58,171::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:07:58,171::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:12:58,588::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:12:58,588::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:12:58,589::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:12:58,803::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:12:58,805::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:12:58,805::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-17908::WARNING::2012-12-03 20:12:58,824::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17909::WARNING::2012-12-03 20:12:58,824::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17910::WARNING::2012-12-03 20:12:58,825::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17912::WARNING::2012-12-03 20:12:58,826::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17914::WARNING::2012-12-03 20:12:58,827::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17916::WARNING::2012-12-03 20:12:58,828::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17917::WARNING::2012-12-03 20:12:58,829::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17918::WARNING::2012-12-03 20:12:58,830::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17919::WARNING::2012-12-03 20:12:58,830::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17920::WARNING::2012-12-03 20:12:58,831::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17921::WARNING::2012-12-03 20:12:58,831::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17922::WARNING::2012-12-03 20:12:58,832::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17923::WARNING::2012-12-03 20:12:58,832::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17925::WARNING::2012-12-03 20:12:58,833::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17924::WARNING::2012-12-03 20:12:58,833::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17926::WARNING::2012-12-03 20:12:58,833::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17927::WARNING::2012-12-03 20:12:58,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17928::WARNING::2012-12-03 20:12:58,834::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17929::WARNING::2012-12-03 20:12:58,835::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17930::WARNING::2012-12-03 20:12:58,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-17931::WARNING::2012-12-03 20:12:58,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:12:58,870::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:12:58,871::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:12:59,077::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:12:59,078::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:12:59,078::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 20:12:59,112::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:12:59,113::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:12:59,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:12:59,118::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:12:59,118::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:12:59,118::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-03 20:12:59,118::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-03 20:12:59,118::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-03 20:13:09,131::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-03 20:17:59,532::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:17:59,533::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:17:59,533::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:17:59,738::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:17:59,739::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:17:59,739::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 20:17:59,779::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:17:59,779::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:17:59,782::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:17:59,784::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:17:59,784::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:17:59,784::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:23:00,293::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:23:00,293::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:23:00,294::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:23:00,500::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:23:00,501::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:23:00,502::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 20:23:00,536::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:23:00,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:23:00,539::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:23:00,542::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:23:00,542::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:23:00,542::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:28:00,981::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:28:00,981::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:28:00,981::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:28:01,184::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:28:01,186::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:28:01,186::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18175::WARNING::2012-12-03 20:28:01,211::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:28:01,223::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:28:01,224::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:28:01,227::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:28:01,230::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:28:01,230::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:28:01,230::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:33:01,623::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:33:01,623::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:33:01,623::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:33:01,826::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:33:01,827::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:33:01,828::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 20:33:01,863::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:33:01,864::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:33:01,866::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:33:01,868::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:33:01,869::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:33:01,869::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:38:02,363::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:38:02,363::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:38:02,364::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:38:02,570::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:38:02,572::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:38:02,572::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 20:38:02,609::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:38:02,610::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:38:02,613::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:38:02,616::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:38:02,616::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:38:02,616::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:43:03,042::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:43:03,042::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:43:03,043::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:43:03,247::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:43:03,248::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:43:03,249::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18343::WARNING::2012-12-03 20:43:03,267::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18344::WARNING::2012-12-03 20:43:03,268::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18345::WARNING::2012-12-03 20:43:03,268::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18347::WARNING::2012-12-03 20:43:03,270::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18346::WARNING::2012-12-03 20:43:03,270::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18349::WARNING::2012-12-03 20:43:03,270::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18348::WARNING::2012-12-03 20:43:03,271::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18350::WARNING::2012-12-03 20:43:03,271::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18352::WARNING::2012-12-03 20:43:03,272::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18351::WARNING::2012-12-03 20:43:03,272::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18353::WARNING::2012-12-03 20:43:03,273::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18354::WARNING::2012-12-03 20:43:03,274::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18355::WARNING::2012-12-03 20:43:03,274::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18356::WARNING::2012-12-03 20:43:03,275::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18357::WARNING::2012-12-03 20:43:03,275::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18358::WARNING::2012-12-03 20:43:03,276::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18359::WARNING::2012-12-03 20:43:03,276::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18360::WARNING::2012-12-03 20:43:03,277::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18361::WARNING::2012-12-03 20:43:03,277::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18362::WARNING::2012-12-03 20:43:03,278::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18363::WARNING::2012-12-03 20:43:03,279::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18364::WARNING::2012-12-03 20:43:03,279::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18365::WARNING::2012-12-03 20:43:03,279::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:43:03,328::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:43:03,328::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:43:03,532::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:43:03,534::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:43:03,534::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 20:43:03,570::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:43:03,570::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:43:03,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:43:03,576::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:43:03,576::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:43:03,577::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-03 20:43:03,577::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-03 20:43:03,577::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-03 20:43:13,590::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-03 20:48:04,001::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:48:04,002::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:48:04,002::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:48:04,206::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:48:04,207::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:48:04,208::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18479::WARNING::2012-12-03 20:48:04,232::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18483::WARNING::2012-12-03 20:48:04,234::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:48:04,245::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:48:04,245::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:48:04,248::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:48:04,250::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:48:04,250::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:48:04,250::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:53:04,705::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:53:04,705::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:53:04,705::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:53:04,910::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:53:04,911::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:53:04,912::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18539::WARNING::2012-12-03 20:53:04,933::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18543::WARNING::2012-12-03 20:53:04,934::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18544::WARNING::2012-12-03 20:53:04,935::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18546::WARNING::2012-12-03 20:53:04,936::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18547::WARNING::2012-12-03 20:53:04,937::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18549::WARNING::2012-12-03 20:53:04,938::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:53:04,952::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:53:04,952::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:53:04,956::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:53:04,959::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:53:04,959::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:53:04,959::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 20:58:05,397::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:58:05,397::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 20:58:05,397::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 20:58:05,602::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 20:58:05,604::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 20:58:05,604::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18608::WARNING::2012-12-03 20:58:05,630::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18611::WARNING::2012-12-03 20:58:05,632::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 20:58:05,645::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 20:58:05,645::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 20:58:05,648::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 20:58:05,650::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 20:58:05,650::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 20:58:05,650::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:03:06,146::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:03:06,147::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:03:06,150::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:03:06,368::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:03:06,371::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:03:06,372::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:03:06,408::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:03:06,409::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:03:06,412::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:03:06,415::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:03:06,415::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:03:06,415::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:08:06,820::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:08:06,820::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:08:06,820::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:08:07,026::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:08:07,028::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:08:07,028::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-18730::WARNING::2012-12-03 21:08:07,053::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18733::WARNING::2012-12-03 21:08:07,054::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-18734::WARNING::2012-12-03 21:08:07,055::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 21:08:07,067::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:08:07,068::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:08:07,071::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:08:07,074::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:08:07,074::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:08:07,074::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:13:07,498::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:13:07,498::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:13:07,499::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:13:07,714::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:13:07,716::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:13:07,716::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:13:07,755::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:13:07,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:13:07,759::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:13:07,762::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:13:07,762::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:13:07,762::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:18:08,254::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:18:08,254::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:18:08,254::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:18:08,461::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:18:08,463::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:18:08,463::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:18:08,500::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:18:08,501::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:18:08,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:18:08,506::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:18:08,507::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:18:08,507::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:23:08,954::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:23:08,954::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:23:08,954::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:23:09,159::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:23:09,161::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:23:09,161::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:23:09,199::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:23:09,200::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:23:09,203::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:23:09,206::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:23:09,206::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:23:09,206::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:28:09,614::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:28:09,614::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:28:09,617::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:28:09,836::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:28:09,837::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:28:09,838::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:28:09,874::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:28:09,874::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:28:09,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:28:09,881::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:28:09,881::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:28:09,881::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:33:10,288::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:33:10,288::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:33:10,289::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:33:10,497::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:33:10,500::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:33:10,500::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:33:10,537::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:33:10,537::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:33:10,540::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:33:10,543::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:33:10,543::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:33:10,543::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:38:11,015::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:38:11,016::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:38:11,016::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:38:11,227::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:38:11,229::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:38:11,229::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19100::WARNING::2012-12-03 21:38:11,253::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19104::WARNING::2012-12-03 21:38:11,255::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19109::WARNING::2012-12-03 21:38:11,258::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 21:38:11,266::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:38:11,267::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:38:11,269::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:38:11,272::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:38:11,272::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:38:11,272::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:43:11,703::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:43:11,703::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:43:11,704::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:43:11,917::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:43:11,918::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:43:11,919::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:43:11,954::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:43:11,955::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:43:11,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:43:11,959::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:43:11,960::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:43:11,960::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:48:12,392::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:48:12,392::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:48:12,393::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:48:12,600::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:48:12,601::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:48:12,602::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19231::WARNING::2012-12-03 21:48:12,628::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 21:48:12,638::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:48:12,638::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:48:12,642::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:48:12,645::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:48:12,645::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:48:12,645::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:53:13,065::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:53:13,065::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:53:13,066::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:53:13,276::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:53:13,277::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:53:13,278::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:53:13,314::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:53:13,314::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:53:13,317::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:53:13,319::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:53:13,319::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:53:13,319::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 21:58:13,738::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:58:13,739::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 21:58:13,742::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 21:58:13,959::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 21:58:13,961::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 21:58:13,961::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 21:58:13,996::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 21:58:13,996::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 21:58:13,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 21:58:14,001::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 21:58:14,001::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 21:58:14,002::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:03:14,435::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:03:14,435::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:03:14,436::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:03:14,662::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:03:14,664::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:03:14,664::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:03:14,701::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:03:14,702::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:03:14,705::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:03:14,708::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:03:14,708::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:03:14,708::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:08:15,152::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:08:15,152::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:08:15,152::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:08:15,362::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:08:15,364::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:08:15,364::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:08:15,401::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:08:15,401::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:08:15,404::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:08:15,406::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:08:15,406::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:08:15,407::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:13:15,810::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:13:15,810::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:13:15,814::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:13:16,039::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:13:16,041::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:13:16,041::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:13:16,079::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:13:16,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:13:16,082::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:13:16,085::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:13:16,085::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:13:16,085::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:18:16,492::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:18:16,492::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:18:16,492::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:18:16,697::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:18:16,698::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:18:16,699::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:18:16,735::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:18:16,735::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:18:16,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:18:16,740::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:18:16,740::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:18:16,741::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:23:17,193::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:23:17,194::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:23:17,194::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:23:17,400::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:23:17,402::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:23:17,403::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:23:17,441::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:23:17,441::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:23:17,445::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:23:17,447::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:23:17,448::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:23:17,448::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:28:17,873::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:28:17,873::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:28:17,874::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:28:18,077::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:28:18,078::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:28:18,079::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19723::WARNING::2012-12-03 22:28:18,104::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 22:28:18,118::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:28:18,118::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:28:18,121::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:28:18,124::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:28:18,124::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:28:18,125::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:33:18,602::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:33:18,602::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:33:18,602::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:33:18,809::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:33:18,810::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:33:18,811::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19783::WARNING::2012-12-03 22:33:18,835::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19784::WARNING::2012-12-03 22:33:18,836::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19786::WARNING::2012-12-03 22:33:18,837::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19788::WARNING::2012-12-03 22:33:18,838::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19790::WARNING::2012-12-03 22:33:18,839::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 22:33:18,856::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:33:18,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:33:18,862::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:33:18,866::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:33:18,867::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:33:18,867::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:38:19,315::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:38:19,315::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:38:19,316::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:38:19,518::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:38:19,519::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:38:19,520::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:38:19,557::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:38:19,558::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:38:19,560::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:38:19,562::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:38:19,563::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:38:19,563::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:43:19,994::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:43:19,994::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:43:19,994::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:43:20,200::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:43:20,201::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:43:20,202::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:43:20,240::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:43:20,240::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:43:20,244::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:43:20,246::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:43:20,247::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:43:20,247::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:48:20,642::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:48:20,642::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:48:20,642::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:48:20,847::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:48:20,848::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:48:20,848::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-19958::WARNING::2012-12-03 22:48:20,867::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19957::WARNING::2012-12-03 22:48:20,867::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19964::WARNING::2012-12-03 22:48:20,870::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-19965::WARNING::2012-12-03 22:48:20,871::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 22:48:20,885::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:48:20,885::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:48:20,887::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:48:20,890::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:48:20,890::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:48:20,890::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:53:21,348::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:53:21,348::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:53:21,349::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:53:21,556::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:53:21,557::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:53:21,558::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:53:21,596::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:53:21,596::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:53:21,600::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:53:21,603::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:53:21,603::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:53:21,603::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 22:58:22,024::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:58:22,024::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 22:58:22,025::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 22:58:22,228::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 22:58:22,230::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 22:58:22,230::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 22:58:22,267::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 22:58:22,267::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 22:58:22,271::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 22:58:22,273::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 22:58:22,274::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 22:58:22,274::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:03:22,687::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:03:22,687::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:03:22,687::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:03:22,903::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:03:22,905::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:03:22,905::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20161::WARNING::2012-12-03 23:03:22,932::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-20160::WARNING::2012-12-03 23:03:22,932::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 23:03:22,944::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:03:22,944::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:03:22,946::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:03:22,949::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:03:22,949::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:03:22,949::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:08:23,404::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:08:23,404::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:08:23,404::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:08:23,613::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:08:23,615::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:08:23,615::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:08:23,652::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:08:23,653::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:08:23,656::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:08:23,659::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:08:23,659::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:08:23,659::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:13:24,090::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:13:24,090::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:13:24,094::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:13:24,308::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:13:24,309::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:13:24,310::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:13:24,347::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:13:24,348::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:13:24,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:13:24,352::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:13:24,353::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:13:24,353::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:18:24,865::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:18:24,865::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:18:24,865::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:18:25,069::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:18:25,070::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:18:25,070::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:18:25,106::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:18:25,106::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:18:25,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:18:25,111::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:18:25,112::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:18:25,112::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:23:25,536::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:23:25,537::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:23:25,537::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:23:25,747::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:23:25,749::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:23:25,750::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:23:25,786::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:23:25,786::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:23:25,788::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:23:25,790::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:23:25,790::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:23:25,790::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:28:26,227::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:28:26,227::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:28:26,228::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:28:26,436::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:28:26,437::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:28:26,438::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20448::WARNING::2012-12-03 23:28:26,453::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_vered_iso2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-20449::WARNING::2012-12-03 23:28:26,454::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1510 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-20451::WARNING::2012-12-03 23:28:26,455::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 23:28:26,474::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:28:26,475::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:28:26,477::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:28:26,479::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:28:26,480::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:28:26,480::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:33:26,961::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:33:26,961::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:33:26,961::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:33:27,166::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:33:27,168::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:33:27,169::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:33:27,205::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:33:27,206::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:33:27,208::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:33:27,211::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:33:27,211::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:33:27,211::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:38:27,628::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:38:27,628::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:38:27,628::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:38:27,833::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:38:27,835::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:38:27,835::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20594::WARNING::2012-12-03 23:38:27,861::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 23:38:27,874::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:38:27,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:38:27,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:38:27,881::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:38:27,881::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:38:27,882::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:43:28,360::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:43:28,360::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:43:28,361::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:43:28,567::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:43:28,569::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:43:28,569::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:43:28,605::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:43:28,605::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:43:28,607::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:43:28,610::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:43:28,610::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:43:28,610::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:48:29,059::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:48:29,059::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:48:29,060::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:48:29,269::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:48:29,271::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:48:29,271::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:48:29,309::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:48:29,310::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:48:29,313::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:48:29,316::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:48:29,316::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:48:29,316::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:53:29,710::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:53:29,710::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:53:29,711::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:53:29,921::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:53:29,922::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:53:29,923::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20777::WARNING::2012-12-03 23:53:29,948::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-03 23:53:29,962::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:53:29,962::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:53:29,965::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:53:29,967::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:53:29,967::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:53:29,967::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-03 23:58:30,369::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:58:30,369::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-03 23:58:30,370::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-03 23:58:30,575::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-03 23:58:30,576::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-03 23:58:30,577::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-03 23:58:30,613::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-03 23:58:30,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-03 23:58:30,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-03 23:58:30,618::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-03 23:58:30,618::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-03 23:58:30,618::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:03:31,065::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:03:31,065::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:03:31,068::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:03:31,295::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:03:31,297::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:03:31,297::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:03:31,335::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:03:31,335::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:03:31,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:03:31,341::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:03:31,341::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:03:31,341::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:08:31,766::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:08:31,766::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:08:31,766::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:08:31,974::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:08:31,975::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:08:31,976::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20967::WARNING::2012-12-04 00:08:32,003::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 00:08:32,015::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:08:32,015::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:08:32,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:08:32,021::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:08:32,021::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:08:32,021::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:13:32,447::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:13:32,447::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:13:32,448::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:13:32,655::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:13:32,657::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:13:32,657::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21026::WARNING::2012-12-04 00:13:32,683::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21028::WARNING::2012-12-04 00:13:32,684::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 00:13:32,697::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:13:32,697::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:13:32,700::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:13:32,703::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:13:32,703::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:13:32,703::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:18:33,175::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:18:33,175::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:18:33,175::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:18:33,386::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:18:33,387::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:18:33,387::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21090::WARNING::2012-12-04 00:18:33,415::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 00:18:33,426::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:18:33,426::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:18:33,428::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:18:33,431::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:18:33,431::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:18:33,431::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:23:33,906::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:23:33,906::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:23:33,907::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:23:34,112::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:23:34,114::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:23:34,114::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:23:34,151::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:23:34,151::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:23:34,155::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:23:34,158::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:23:34,158::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:23:34,158::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:28:34,581::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:28:34,581::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:28:34,582::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:28:34,786::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:28:34,787::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:28:34,788::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21194::WARNING::2012-12-04 00:28:34,805::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2328 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21196::WARNING::2012-12-04 00:28:34,806::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21198::WARNING::2012-12-04 00:28:34,807::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21199::WARNING::2012-12-04 00:28:34,807::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21200::WARNING::2012-12-04 00:28:34,808::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21201::WARNING::2012-12-04 00:28:34,808::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21203::WARNING::2012-12-04 00:28:34,810::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21204::WARNING::2012-12-04 00:28:34,811::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21205::WARNING::2012-12-04 00:28:34,811::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21206::WARNING::2012-12-04 00:28:34,811::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21207::WARNING::2012-12-04 00:28:34,812::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21208::WARNING::2012-12-04 00:28:34,812::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21209::WARNING::2012-12-04 00:28:34,813::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21210::WARNING::2012-12-04 00:28:34,814::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21211::WARNING::2012-12-04 00:28:34,814::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21212::WARNING::2012-12-04 00:28:34,815::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21213::WARNING::2012-12-04 00:28:34,815::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21214::WARNING::2012-12-04 00:28:34,816::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21215::WARNING::2012-12-04 00:28:34,817::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21216::WARNING::2012-12-04 00:28:34,817::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21217::WARNING::2012-12-04 00:28:34,818::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 00:28:34,852::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:28:34,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:28:34,855::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:28:34,857::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:28:34,857::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:28:34,857::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:33:35,301::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:33:35,301::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:33:35,301::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:33:35,503::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:33:35,505::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:33:35,505::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:33:35,541::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:33:35,542::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:33:35,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:33:35,548::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:33:35,548::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:33:35,548::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:38:35,985::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:38:35,986::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:38:35,986::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:38:36,190::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:38:36,192::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:38:36,192::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:38:36,230::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:38:36,230::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:38:36,234::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:38:36,237::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:38:36,237::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:38:36,237::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:43:36,638::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:43:36,638::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:43:36,638::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:43:36,845::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:43:36,846::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:43:36,846::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:43:36,883::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:43:36,884::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:43:36,887::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:43:36,890::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:43:36,890::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:43:36,890::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:48:37,299::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:48:37,300::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:48:37,300::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:48:37,504::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:48:37,506::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:48:37,506::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:48:37,543::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:48:37,543::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:48:37,547::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:48:37,549::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:48:37,549::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:48:37,550::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:53:38,002::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:53:38,002::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:53:38,002::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:53:38,206::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:53:38,208::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:53:38,208::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21526::WARNING::2012-12-04 00:53:38,235::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 00:53:38,245::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:53:38,245::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:53:38,249::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:53:38,251::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:53:38,251::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:53:38,252::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 00:58:38,659::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:58:38,660::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 00:58:38,660::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 00:58:38,864::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 00:58:38,866::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 00:58:38,866::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 00:58:38,903::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 00:58:38,903::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 00:58:38,907::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 00:58:38,909::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 00:58:38,910::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 00:58:38,910::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:03:39,359::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:03:39,359::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:03:39,362::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:03:39,585::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:03:39,587::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:03:39,588::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:03:39,626::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:03:39,626::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:03:39,629::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:03:39,631::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:03:39,631::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:03:39,631::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:08:40,190::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:08:40,191::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:08:40,191::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:08:40,405::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:08:40,407::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:08:40,407::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21711::WARNING::2012-12-04 01:08:40,434::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 01:08:40,444::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:08:40,444::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:08:40,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:08:40,449::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:08:40,449::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:08:40,449::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:13:40,865::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:13:40,866::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:13:40,866::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:13:41,074::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:13:41,075::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:13:41,076::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:13:41,112::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:13:41,112::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:13:41,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:13:41,117::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:13:41,117::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:13:41,117::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:18:41,545::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:18:41,545::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:18:41,546::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:18:41,755::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:18:41,756::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:18:41,757::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21835::WARNING::2012-12-04 01:18:41,784::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 01:18:41,796::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:18:41,796::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:18:41,800::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:18:41,802::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:18:41,802::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:18:41,802::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:23:42,279::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:23:42,279::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:23:42,280::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:23:42,489::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:23:42,491::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:23:42,491::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-21896::WARNING::2012-12-04 01:23:42,517::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-21898::WARNING::2012-12-04 01:23:42,519::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 01:23:42,529::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:23:42,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:23:42,532::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:23:42,535::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:23:42,535::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:23:42,535::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:28:43,040::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:28:43,041::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:28:43,041::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:28:43,248::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:28:43,250::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:28:43,250::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:28:43,286::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:28:43,287::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:28:43,291::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:28:43,293::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:28:43,293::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:28:43,294::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:33:43,712::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:33:43,713::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:33:43,713::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:33:43,923::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:33:43,925::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:33:43,925::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:33:43,963::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:33:43,964::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:33:43,967::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:33:43,970::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:33:43,970::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:33:43,970::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:38:44,459::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:38:44,460::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:38:44,460::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:38:44,666::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:38:44,668::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:38:44,668::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:38:44,703::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:38:44,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:38:44,707::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:38:44,710::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:38:44,710::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:38:44,710::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:43:45,161::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:43:45,161::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:43:45,162::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:43:45,370::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:43:45,372::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:43:45,372::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:43:45,409::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:43:45,409::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:43:45,411::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:43:45,413::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:43:45,414::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:43:45,414::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:48:45,856::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:48:45,856::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:48:45,857::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:48:46,062::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:48:46,063::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:48:46,064::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22207::WARNING::2012-12-04 01:48:46,089::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 01:48:46,102::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:48:46,102::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:48:46,106::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:48:46,109::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:48:46,109::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:48:46,110::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:53:46,552::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:53:46,553::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:53:46,556::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:53:46,774::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:53:46,775::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:53:46,776::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:53:46,813::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:53:46,813::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:53:46,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:53:46,818::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:53:46,818::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:53:46,818::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 01:58:47,261::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:58:47,261::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 01:58:47,262::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 01:58:47,473::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 01:58:47,475::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 01:58:47,475::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 01:58:47,512::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 01:58:47,512::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 01:58:47,515::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 01:58:47,518::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 01:58:47,518::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 01:58:47,519::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:03:47,969::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:03:47,969::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:03:47,973::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:03:48,202::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:03:48,204::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:03:48,204::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22392::WARNING::2012-12-04 02:03:48,231::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 02:03:48,244::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:03:48,245::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:03:48,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:03:48,253::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:03:48,253::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:03:48,253::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:08:48,741::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:08:48,741::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:08:48,742::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:08:48,948::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:08:48,950::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:08:48,950::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22454::WARNING::2012-12-04 02:08:48,975::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 02:08:48,986::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:08:48,986::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:08:48,990::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:08:48,993::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:08:48,993::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:08:48,993::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:13:49,427::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:13:49,427::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:13:49,428::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:13:49,635::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:13:49,637::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:13:49,638::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 02:13:49,674::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:13:49,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:13:49,678::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:13:49,680::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:13:49,681::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:13:49,681::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:18:50,100::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:18:50,100::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:18:50,101::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:18:50,309::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:18:50,311::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:18:50,311::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22579::WARNING::2012-12-04 02:18:50,337::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 02:18:50,347::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:18:50,347::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:18:50,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:18:50,353::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:18:50,353::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:18:50,353::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:23:50,748::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:23:50,748::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:23:50,749::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:23:50,955::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:23:50,956::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:23:50,957::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22638::WARNING::2012-12-04 02:23:50,983::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 02:23:50,997::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:23:50,997::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:23:51,001::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:23:51,004::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:23:51,004::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:23:51,004::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:28:51,428::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:28:51,428::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:28:51,429::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:28:51,638::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:28:51,640::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:28:51,640::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 02:28:51,678::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:28:51,678::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:28:51,681::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:28:51,683::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:28:51,683::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:28:51,684::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:33:52,195::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:33:52,196::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:33:52,196::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:33:52,403::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:33:52,405::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:33:52,405::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 02:33:52,442::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:33:52,442::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:33:52,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:33:52,446::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:33:52,447::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:33:52,447::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:38:52,838::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:38:52,838::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:38:52,839::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:38:53,042::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:38:53,043::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:38:53,043::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 02:38:53,083::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:38:53,083::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:38:53,087::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:38:53,089::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:38:53,089::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:38:53,089::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:43:53,487::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:43:53,488::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:43:53,488::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:43:53,692::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:43:53,694::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:43:53,694::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 02:43:53,731::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:43:53,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:43:53,735::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:43:53,737::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:43:53,737::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:43:53,737::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:48:54,241::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:48:54,241::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:48:54,242::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:48:54,444::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:48:54,446::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:48:54,446::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22948::WARNING::2012-12-04 02:48:54,472::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 02:48:54,486::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:48:54,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:48:54,490::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:48:54,493::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:48:54,493::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:48:54,493::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:53:54,891::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:53:54,891::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:53:54,892::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:53:55,098::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:53:55,100::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:53:55,100::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 02:53:55,135::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:53:55,135::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:53:55,139::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:53:55,141::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:53:55,142::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:53:55,142::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 02:58:55,563::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:58:55,563::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 02:58:55,564::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 02:58:55,768::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 02:58:55,770::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 02:58:55,770::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-23077::WARNING::2012-12-04 02:58:55,798::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 02:58:55,809::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 02:58:55,809::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 02:58:55,813::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 02:58:55,816::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 02:58:55,816::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 02:58:55,816::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:03:56,302::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:03:56,302::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:03:56,306::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:03:56,528::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:03:56,530::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:03:56,531::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:03:56,569::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:03:56,569::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:03:56,573::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:03:56,575::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:03:56,575::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:03:56,575::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:08:57,008::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:08:57,008::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:08:57,009::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:08:57,216::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:08:57,218::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:08:57,218::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:08:57,254::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:08:57,255::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:08:57,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:08:57,260::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:08:57,260::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:08:57,260::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:13:57,680::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:13:57,681::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:13:57,684::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:13:57,913::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:13:57,915::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:13:57,916::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-23254::WARNING::2012-12-04 03:13:57,940::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-23258::WARNING::2012-12-04 03:13:57,942::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-23260::WARNING::2012-12-04 03:13:57,943::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-23263::WARNING::2012-12-04 03:13:57,945::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 03:13:57,957::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:13:57,957::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:13:57,961::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:13:57,964::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:13:57,964::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:13:57,965::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:18:58,404::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:18:58,405::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:18:58,408::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:18:58,615::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:18:58,616::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:18:58,616::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-23325::WARNING::2012-12-04 03:18:58,644::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 03:18:58,654::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:18:58,654::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:18:58,657::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:18:58,659::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:18:58,660::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:18:58,660::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:23:59,087::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:23:59,087::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:23:59,088::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:23:59,300::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:23:59,302::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:23:59,302::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:23:59,343::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:23:59,344::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:23:59,349::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:23:59,351::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:23:59,352::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:23:59,352::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:28:59,858::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:28:59,858::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:28:59,859::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:29:00,069::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:29:00,071::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:29:00,071::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-23446::WARNING::2012-12-04 03:29:00,097::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 03:29:00,109::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:29:00,109::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:29:00,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:29:00,116::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:29:00,116::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:29:00,116::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:34:00,541::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:34:00,542::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:34:00,542::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:34:00,750::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:34:00,752::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:34:00,752::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:34:00,789::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:34:00,790::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:34:00,794::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:34:00,797::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:34:00,797::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:34:00,797::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:39:01,240::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:39:01,240::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:39:01,241::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:39:01,447::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:39:01,448::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:39:01,449::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:39:01,536::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:39:01,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:39:01,539::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:39:01,541::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:39:01,541::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:39:01,541::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:44:01,991::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:44:01,991::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:44:01,992::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:44:02,201::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:44:02,202::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:44:02,203::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-23635::WARNING::2012-12-04 03:44:02,230::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 03:44:02,242::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:44:02,242::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:44:02,246::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:44:02,249::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:44:02,249::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:44:02,249::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:49:02,679::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:49:02,679::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:49:02,680::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:49:02,887::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:49:02,889::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:49:02,889::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-23696::WARNING::2012-12-04 03:49:02,915::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-23697::WARNING::2012-12-04 03:49:02,916::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 03:49:02,926::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:49:02,927::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:49:02,931::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:49:02,934::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:49:02,934::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:49:02,934::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:54:03,389::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:54:03,389::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:54:03,390::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:54:03,598::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:54:03,600::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:54:03,600::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:54:03,640::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:54:03,640::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:54:03,644::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:54:03,646::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:54:03,646::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:54:03,647::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 03:59:04,089::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:59:04,090::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 03:59:04,090::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 03:59:04,299::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 03:59:04,301::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 03:59:04,301::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 03:59:04,337::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 03:59:04,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 03:59:04,342::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 03:59:04,344::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 03:59:04,344::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 03:59:04,344::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:04:04,778::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:04:04,778::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:04:04,779::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:04:04,991::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:04:04,994::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:04:04,994::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:04:05,030::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:04:05,031::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:04:05,033::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:04:05,036::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:04:05,036::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:04:05,036::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:09:05,480::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:09:05,481::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:09:05,481::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:09:05,688::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:09:05,690::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:09:05,690::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:09:05,727::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:09:05,727::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:09:05,731::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:09:05,733::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:09:05,734::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:09:05,734::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:14:06,250::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:14:06,250::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:14:06,250::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:14:06,459::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:14:06,461::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:14:06,461::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:14:06,498::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:14:06,498::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:14:06,500::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:14:06,503::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:14:06,503::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:14:06,503::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:19:06,920::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:19:06,921::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:19:06,924::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:19:07,137::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:19:07,139::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:19:07,139::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:19:07,175::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:19:07,176::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:19:07,180::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:19:07,184::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:19:07,184::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:19:07,184::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:24:07,608::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:24:07,609::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:24:07,609::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:24:07,819::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:24:07,820::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:24:07,820::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-24110::WARNING::2012-12-04 04:24:07,839::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24111::WARNING::2012-12-04 04:24:07,840::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24113::WARNING::2012-12-04 04:24:07,842::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24115::WARNING::2012-12-04 04:24:07,842::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24117::WARNING::2012-12-04 04:24:07,843::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24118::WARNING::2012-12-04 04:24:07,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24119::WARNING::2012-12-04 04:24:07,844::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24120::WARNING::2012-12-04 04:24:07,845::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24121::WARNING::2012-12-04 04:24:07,846::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24122::WARNING::2012-12-04 04:24:07,846::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24123::WARNING::2012-12-04 04:24:07,847::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24124::WARNING::2012-12-04 04:24:07,847::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24125::WARNING::2012-12-04 04:24:07,848::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24126::WARNING::2012-12-04 04:24:07,848::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24127::WARNING::2012-12-04 04:24:07,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24128::WARNING::2012-12-04 04:24:07,849::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24129::WARNING::2012-12-04 04:24:07,850::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24130::WARNING::2012-12-04 04:24:07,851::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-24131::WARNING::2012-12-04 04:24:07,851::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 04:24:07,891::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:24:07,892::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:24:07,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:24:07,896::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:24:07,897::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:24:07,897::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:29:08,329::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:29:08,330::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:29:08,330::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:29:08,538::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:29:08,540::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:29:08,540::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:29:08,577::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:29:08,577::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:29:08,581::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:29:08,584::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:29:08,584::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:29:08,584::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:34:09,060::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:34:09,060::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:34:09,061::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:34:09,267::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:34:09,269::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:34:09,269::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:34:09,308::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:34:09,309::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:34:09,313::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:34:09,315::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:34:09,316::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:34:09,316::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:39:09,817::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:39:09,817::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:39:09,817::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:39:10,022::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:39:10,023::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:39:10,024::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:39:10,061::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:39:10,061::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:39:10,065::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:39:10,068::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:39:10,068::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:39:10,068::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:44:10,510::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:44:10,510::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:44:10,511::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:44:10,718::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:44:10,720::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:44:10,720::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:44:10,758::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:44:10,759::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:44:10,762::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:44:10,766::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:44:10,766::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:44:10,766::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:49:11,191::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:49:11,191::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:49:11,192::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:49:11,399::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:49:11,401::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:49:11,401::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-24438::WARNING::2012-12-04 04:49:11,428::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 04:49:11,439::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:49:11,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:49:11,443::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:49:11,446::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:49:11,446::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:49:11,446::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:54:11,859::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:54:11,859::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:54:11,860::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:54:12,067::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:54:12,069::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:54:12,069::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-24499::WARNING::2012-12-04 04:54:12,096::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 04:54:12,109::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:54:12,109::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:54:12,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:54:12,115::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:54:12,115::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:54:12,116::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 04:59:12,585::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:59:12,585::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 04:59:12,585::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 04:59:12,791::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 04:59:12,793::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 04:59:12,793::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 04:59:12,832::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 04:59:12,832::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 04:59:12,835::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 04:59:12,838::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 04:59:12,839::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 04:59:12,839::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:04:13,285::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:04:13,285::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:04:13,286::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:04:13,504::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:04:13,506::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:04:13,507::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:04:13,544::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:04:13,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:04:13,547::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:04:13,549::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:04:13,549::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:04:13,550::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:09:13,989::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:09:13,989::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:09:13,989::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:09:14,203::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:09:14,204::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:09:14,205::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:09:14,242::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:09:14,242::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:09:14,245::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:09:14,248::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:09:14,248::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:09:14,248::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:14:14,659::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:14:14,660::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:14:14,663::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:14:14,870::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:14:14,871::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:14:14,874::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-24740::WARNING::2012-12-04 05:14:14,898::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 05:14:14,912::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:14:14,912::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:14:14,916::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:14:14,919::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:14:14,919::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:14:14,919::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:19:15,340::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:19:15,340::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:19:15,341::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:19:15,548::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:19:15,549::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:19:15,550::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-24810::WARNING::2012-12-04 05:19:15,576::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 05:19:15,591::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:19:15,591::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:19:15,594::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:19:15,596::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:19:15,596::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:19:15,596::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:24:16,088::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:24:16,088::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:24:16,089::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:24:16,294::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:24:16,296::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:24:16,297::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-24872::WARNING::2012-12-04 05:24:16,323::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 05:24:16,335::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:24:16,336::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:24:16,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:24:16,341::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:24:16,341::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:24:16,341::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:29:16,749::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:29:16,749::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:29:16,750::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:29:16,961::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:29:16,962::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:29:16,962::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:29:17,002::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:29:17,002::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:29:17,005::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:29:17,007::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:29:17,007::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:29:17,007::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:34:17,438::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:34:17,439::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:34:17,439::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:34:17,645::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:34:17,647::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:34:17,647::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:34:17,681::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:34:17,681::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:34:17,685::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:34:17,687::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:34:17,687::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:34:17,688::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:39:18,119::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:39:18,119::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:39:18,120::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:39:18,327::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:39:18,329::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:39:18,329::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:39:18,365::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:39:18,365::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:39:18,369::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:39:18,371::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:39:18,372::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:39:18,372::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:44:18,800::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:44:18,801::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:44:18,801::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:44:19,012::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:44:19,014::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:44:19,015::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:44:19,054::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:44:19,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:44:19,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:44:19,061::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:44:19,061::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:44:19,062::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:49:19,482::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:49:19,482::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:49:19,483::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:49:19,692::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:49:19,694::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:49:19,694::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:49:19,730::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:49:19,731::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:49:19,733::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:49:19,736::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:49:19,736::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:49:19,736::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:54:20,210::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:54:20,210::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:54:20,210::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:54:20,422::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:54:20,424::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:54:20,425::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:54:20,464::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:54:20,465::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:54:20,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:54:20,471::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:54:20,471::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:54:20,471::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 05:59:20,897::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:59:20,897::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 05:59:20,897::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 05:59:21,104::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 05:59:21,105::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 05:59:21,106::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 05:59:21,141::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 05:59:21,141::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 05:59:21,144::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 05:59:21,146::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 05:59:21,146::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 05:59:21,146::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:04:21,564::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:04:21,565::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:04:21,568::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:04:21,788::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:04:21,790::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:04:21,791::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:04:21,829::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:04:21,829::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:04:21,833::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:04:21,835::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:04:21,835::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:04:21,835::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:09:22,259::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:09:22,259::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:09:22,260::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:09:22,561::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:09:22,563::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:09:22,563::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:09:22,601::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:09:22,601::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:09:22,605::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:09:22,607::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:09:22,607::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:09:22,608::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:14:23,044::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:14:23,044::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:14:23,045::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:14:23,253::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:14:23,255::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:14:23,255::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:14:23,292::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:14:23,292::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:14:23,295::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:14:23,297::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:14:23,297::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:14:23,297::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:19:23,701::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:19:23,701::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:19:23,702::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:19:23,908::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:19:23,910::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:19:23,910::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:19:23,947::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:19:23,947::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:19:23,950::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:19:23,952::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:19:23,952::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:19:23,953::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:24:24,368::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:24:24,368::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:24:24,369::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:24:24,571::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:24:24,573::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:24:24,573::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:24:24,610::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:24:24,610::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:24:24,612::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:24:24,614::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:24:24,615::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:24:24,615::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:29:25,108::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:29:25,108::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:29:25,109::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:29:25,312::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:29:25,313::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:29:25,313::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:29:25,352::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:29:25,352::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:29:25,355::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:29:25,357::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:29:25,357::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:29:25,357::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:34:25,759::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:34:25,760::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:34:25,760::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:34:25,967::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:34:25,969::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:34:25,969::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:34:26,008::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:34:26,008::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:34:26,011::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:34:26,013::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:34:26,013::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:34:26,013::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:39:26,439::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:39:26,439::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:39:26,439::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:39:26,643::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:39:26,644::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:39:26,645::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:39:26,682::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:39:26,682::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:39:26,684::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:39:26,687::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:39:26,687::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:39:26,687::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:44:27,185::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:44:27,185::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:44:27,186::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:44:27,392::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:44:27,393::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:44:27,394::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:44:27,429::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:44:27,430::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:44:27,432::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:44:27,434::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:44:27,435::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:44:27,435::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:49:27,830::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:49:27,830::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:49:27,831::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:49:28,039::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:49:28,040::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:49:28,041::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-25925::WARNING::2012-12-04 06:49:28,068::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-25926::WARNING::2012-12-04 06:49:28,069::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 06:49:28,082::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:49:28,082::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:49:28,085::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:49:28,087::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:49:28,087::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:49:28,087::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:54:28,480::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:54:28,480::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:54:28,480::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:54:28,684::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:54:28,686::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:54:28,686::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:54:28,723::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:54:28,723::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:54:28,727::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:54:28,730::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:54:28,730::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:54:28,730::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 06:59:29,238::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:59:29,238::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 06:59:29,239::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 06:59:29,444::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 06:59:29,445::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 06:59:29,445::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 06:59:29,483::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 06:59:29,483::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 06:59:29,486::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 06:59:29,488::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 06:59:29,488::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 06:59:29,488::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:04:29,904::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:04:29,904::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:04:29,908::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:04:30,131::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:04:30,134::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:04:30,135::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:04:30,173::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:04:30,173::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:04:30,177::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:04:30,180::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:04:30,180::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:04:30,180::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:09:30,579::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:09:30,579::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:09:30,579::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:09:30,785::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:09:30,786::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:09:30,787::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:09:30,823::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:09:30,823::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:09:30,827::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:09:30,830::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:09:30,830::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:09:30,830::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:14:31,309::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:14:31,309::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:14:31,310::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:14:31,521::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:14:31,523::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:14:31,523::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:14:31,560::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:14:31,560::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:14:31,563::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:14:31,565::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:14:31,566::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:14:31,566::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:19:32,003::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:19:32,003::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:19:32,004::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:19:32,210::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:19:32,211::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:19:32,212::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:19:32,252::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:19:32,252::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:19:32,256::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:19:32,258::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:19:32,258::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:19:32,259::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:24:32,671::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:24:32,671::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:24:32,672::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:24:32,880::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:24:32,881::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:24:32,881::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-26361::WARNING::2012-12-04 07:24:32,907::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 07:24:32,920::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:24:32,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:24:32,924::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:24:32,926::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:24:32,926::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:24:32,927::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:29:33,399::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:29:33,399::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:29:33,400::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:29:33,605::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:29:33,606::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:29:33,607::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:29:33,643::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:29:33,644::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:29:33,647::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:29:33,650::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:29:33,650::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:29:33,650::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:34:34,104::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:34:34,105::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:34:34,108::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:34:34,335::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:34:34,338::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:34:34,339::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:34:34,376::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:34:34,376::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:34:34,380::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:34:34,383::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:34:34,383::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:34:34,383::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:39:34,814::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:39:34,814::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:39:34,815::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:39:35,027::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:39:35,029::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:39:35,029::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:39:35,066::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:39:35,066::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:39:35,070::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:39:35,073::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:39:35,073::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:39:35,073::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:44:35,518::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:44:35,518::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:44:35,519::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:44:35,728::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:44:35,729::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:44:35,729::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:44:35,766::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:44:35,766::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:44:35,769::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:44:35,772::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:44:35,772::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:44:35,772::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:49:36,218::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:49:36,218::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:49:36,218::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:49:36,421::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:49:36,422::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:49:36,423::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:49:36,459::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:49:36,460::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:49:36,463::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:49:36,466::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:49:36,466::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:49:36,466::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:54:36,922::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:54:36,922::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:54:36,922::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:54:37,124::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:54:37,126::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:54:37,126::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:54:37,161::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:54:37,161::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:54:37,164::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:54:37,166::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:54:37,167::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:54:37,167::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 07:59:37,591::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:59:37,591::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 07:59:37,591::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 07:59:37,800::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 07:59:37,802::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 07:59:37,802::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 07:59:37,840::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 07:59:37,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 07:59:37,844::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 07:59:37,846::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 07:59:37,846::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 07:59:37,847::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:04:38,298::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:04:38,299::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:04:38,302::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:04:38,522::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:04:38,525::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:04:38,526::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:04:38,565::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:04:38,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:04:38,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:04:38,570::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:04:38,570::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:04:38,571::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:09:39,016::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:09:39,016::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:09:39,017::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:09:39,660::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:09:39,662::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:09:39,662::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:09:39,699::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:09:39,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:09:39,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:09:39,706::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:09:39,706::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:09:39,706::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:14:40,149::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:14:40,150::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:14:40,150::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:14:40,368::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:14:40,370::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:14:40,370::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:14:40,408::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:14:40,408::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:14:40,411::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:14:40,414::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:14:40,414::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:14:40,414::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:19:40,832::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:19:40,833::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:19:40,833::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:19:41,045::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:19:41,046::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:19:41,046::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27024::WARNING::2012-12-04 08:19:41,065::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27028::WARNING::2012-12-04 08:19:41,067::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27032::WARNING::2012-12-04 08:19:41,069::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27034::WARNING::2012-12-04 08:19:41,069::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27033::WARNING::2012-12-04 08:19:41,070::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27037::WARNING::2012-12-04 08:19:41,071::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27038::WARNING::2012-12-04 08:19:41,072::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27040::WARNING::2012-12-04 08:19:41,073::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27041::WARNING::2012-12-04 08:19:41,074::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27042::WARNING::2012-12-04 08:19:41,074::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27043::WARNING::2012-12-04 08:19:41,075::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27044::WARNING::2012-12-04 08:19:41,075::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27045::WARNING::2012-12-04 08:19:41,076::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 08:19:41,087::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:19:41,088::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:19:41,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:19:41,095::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:19:41,095::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:19:41,095::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:24:41,534::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:24:41,534::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:24:41,534::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:24:41,739::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:24:41,741::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:24:41,741::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27086::WARNING::2012-12-04 08:24:41,760::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27088::WARNING::2012-12-04 08:24:41,761::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27089::WARNING::2012-12-04 08:24:41,762::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27091::WARNING::2012-12-04 08:24:41,763::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27092::WARNING::2012-12-04 08:24:41,763::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27094::WARNING::2012-12-04 08:24:41,764::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27098::WARNING::2012-12-04 08:24:41,766::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27100::WARNING::2012-12-04 08:24:41,767::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27102::WARNING::2012-12-04 08:24:41,768::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27104::WARNING::2012-12-04 08:24:41,769::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27103::WARNING::2012-12-04 08:24:41,770::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27106::WARNING::2012-12-04 08:24:41,770::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27105::WARNING::2012-12-04 08:24:41,770::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27107::WARNING::2012-12-04 08:24:41,771::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 08:24:41,783::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:24:41,784::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:24:41,992::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:24:41,994::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:24:41,994::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:24:42,030::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:24:42,031::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:24:42,033::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:24:42,036::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:24:42,036::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:24:42,036::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-04 08:24:42,036::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-04 08:24:42,036::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-04 08:24:52,049::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-04 08:29:42,460::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:29:42,460::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:29:42,460::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:29:42,663::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:29:42,664::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:29:42,665::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:29:42,701::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:29:42,701::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:29:42,704::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:29:42,706::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:29:42,707::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:29:42,707::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:34:43,146::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:34:43,146::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:34:43,147::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:34:43,350::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:34:43,352::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:34:43,352::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27282::WARNING::2012-12-04 08:34:43,375::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27290::WARNING::2012-12-04 08:34:43,379::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 08:34:43,392::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:34:43,392::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:34:43,397::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:34:43,399::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:34:43,399::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:34:43,400::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:39:43,839::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:39:43,840::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:39:43,840::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:39:44,045::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:39:44,046::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:39:44,047::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:39:44,084::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:39:44,084::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:39:44,087::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:39:44,090::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:39:44,090::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:39:44,090::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:44:44,490::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:44:44,490::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:44:44,490::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:44:44,694::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:44:44,695::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:44:44,696::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27409::WARNING::2012-12-04 08:44:44,720::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27411::WARNING::2012-12-04 08:44:44,721::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27413::WARNING::2012-12-04 08:44:44,722::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27416::WARNING::2012-12-04 08:44:44,723::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27417::WARNING::2012-12-04 08:44:44,724::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 08:44:44,736::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:44:44,736::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:44:44,740::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:44:44,743::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:44:44,743::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:44:44,743::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:49:45,170::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:49:45,170::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:49:45,171::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:49:45,374::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:49:45,375::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:49:45,375::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:49:45,412::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:49:45,412::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:49:45,415::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:49:45,418::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:49:45,418::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:49:45,419::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:54:45,847::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:54:45,848::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:54:45,848::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:54:46,055::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:54:46,056::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:54:46,057::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 08:54:46,093::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:54:46,093::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:54:46,096::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:54:46,098::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:54:46,098::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:54:46,098::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 08:59:46,510::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:59:46,510::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 08:59:46,511::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 08:59:46,724::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 08:59:46,725::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 08:59:46,725::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27584::WARNING::2012-12-04 08:59:46,745::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27586::WARNING::2012-12-04 08:59:46,746::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 08:59:46,761::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 08:59:46,762::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 08:59:46,764::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 08:59:46,767::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 08:59:46,767::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 08:59:46,767::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:04:47,212::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:04:47,213::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:04:47,216::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:04:47,439::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:04:47,442::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:04:47,442::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:04:47,480::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:04:47,481::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:04:47,484::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:04:47,487::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:04:47,487::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:04:47,487::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:09:47,938::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:09:47,939::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:09:47,939::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:09:48,145::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:09:48,146::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:09:48,147::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:09:48,183::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:09:48,183::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:09:48,189::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:09:48,192::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:09:48,192::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:09:48,192::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:14:48,625::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:14:48,625::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:14:48,625::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:14:48,840::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:14:48,841::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:14:48,842::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27782::WARNING::2012-12-04 09:14:48,867::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27788::WARNING::2012-12-04 09:14:48,871::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 09:14:48,882::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:14:48,882::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:14:48,886::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:14:48,889::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:14:48,890::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:14:48,890::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:19:49,305::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:19:49,305::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:19:49,305::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:19:49,512::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:19:49,514::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:19:49,514::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:19:49,551::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:19:49,551::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:19:49,555::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:19:49,557::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:19:49,558::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:19:49,558::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:24:49,990::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:24:49,990::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:24:49,991::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:24:50,198::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:24:50,200::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:24:50,200::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:24:50,235::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:24:50,235::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:24:50,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:24:50,240::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:24:50,240::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:24:50,240::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:29:50,685::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:29:50,685::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:29:50,686::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:29:50,895::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:29:50,896::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:29:50,896::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-27967::WARNING::2012-12-04 09:29:50,921::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-27972::WARNING::2012-12-04 09:29:50,924::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 09:29:50,936::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:29:50,936::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:29:50,940::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:29:50,943::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:29:50,943::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:29:50,943::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:34:51,400::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:34:51,400::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:34:51,401::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:34:51,605::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:34:51,606::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:34:51,606::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:34:51,642::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:34:51,642::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:34:51,645::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:34:51,647::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:34:51,647::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:34:51,647::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:39:52,139::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:39:52,140::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:39:52,140::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:39:52,351::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:39:52,352::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:39:52,353::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:39:52,390::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:39:52,390::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:39:52,392::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:39:52,395::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:39:52,395::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:39:52,395::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:44:52,823::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:44:52,823::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:44:52,824::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:44:53,032::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:44:53,034::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:44:53,034::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28149::WARNING::2012-12-04 09:44:53,056::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28151::WARNING::2012-12-04 09:44:53,057::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28150::WARNING::2012-12-04 09:44:53,057::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28159::WARNING::2012-12-04 09:44:53,062::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28160::WARNING::2012-12-04 09:44:53,062::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 09:44:53,076::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:44:53,076::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:44:53,079::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:44:53,082::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:44:53,082::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:44:53,083::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:49:53,513::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:49:53,513::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:49:53,513::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:49:53,719::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:49:53,721::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:49:53,721::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28202::WARNING::2012-12-04 09:49:53,740::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28203::WARNING::2012-12-04 09:49:53,740::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28205::WARNING::2012-12-04 09:49:53,741::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28206::WARNING::2012-12-04 09:49:53,742::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28207::WARNING::2012-12-04 09:49:53,742::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28209::WARNING::2012-12-04 09:49:53,744::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28210::WARNING::2012-12-04 09:49:53,744::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28211::WARNING::2012-12-04 09:49:53,745::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28212::WARNING::2012-12-04 09:49:53,745::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28213::WARNING::2012-12-04 09:49:53,746::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28214::WARNING::2012-12-04 09:49:53,746::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28215::WARNING::2012-12-04 09:49:53,747::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28216::WARNING::2012-12-04 09:49:53,748::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28217::WARNING::2012-12-04 09:49:53,748::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28218::WARNING::2012-12-04 09:49:53,749::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28219::WARNING::2012-12-04 09:49:53,749::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28220::WARNING::2012-12-04 09:49:53,750::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28221::WARNING::2012-12-04 09:49:53,750::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28222::WARNING::2012-12-04 09:49:53,751::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28223::WARNING::2012-12-04 09:49:53,751::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 09:49:53,784::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:49:53,784::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:49:53,787::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:49:53,789::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:49:53,789::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:49:53,789::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:54:54,249::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:54:54,250::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:54:54,253::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:54:54,469::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:54:54,471::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:54:54,471::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:54:54,509::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:54:54,509::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:54:54,513::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:54:54,516::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:54:54,516::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:54:54,516::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 09:59:54,957::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:59:54,958::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 09:59:54,958::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 09:59:55,164::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 09:59:55,165::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 09:59:55,166::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 09:59:55,200::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 09:59:55,200::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 09:59:55,203::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 09:59:55,205::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 09:59:55,205::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 09:59:55,206::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:04:55,612::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:04:55,612::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:04:55,612::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:04:55,830::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:04:55,832::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:04:55,832::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28407::WARNING::2012-12-04 10:04:55,863::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 10:04:55,874::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:04:55,874::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:04:55,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:04:55,881::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:04:55,881::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:04:55,881::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:09:56,323::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:09:56,323::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:09:56,323::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:09:56,531::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:09:56,532::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:09:56,533::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:09:56,571::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:09:56,571::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:09:56,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:09:56,578::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:09:56,578::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:09:56,578::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:14:57,074::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:14:57,075::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:14:57,075::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:14:57,286::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:14:57,287::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:14:57,288::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:14:57,322::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:14:57,323::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:14:57,327::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:14:57,329::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:14:57,329::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:14:57,330::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:19:57,801::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:19:57,801::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:19:57,802::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:19:58,009::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:19:58,010::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:19:58,011::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:19:58,047::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:19:58,047::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:19:58,051::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:19:58,053::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:19:58,054::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:19:58,054::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:24:58,519::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:24:58,520::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:24:58,520::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:24:58,726::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:24:58,728::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:24:58,729::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:24:58,765::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:24:58,765::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:24:58,770::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:24:58,772::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:24:58,772::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:24:58,772::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:29:59,244::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:29:59,244::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:29:59,244::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:29:59,451::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:29:59,453::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:29:59,453::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:29:59,490::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:29:59,490::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:29:59,493::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:29:59,495::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:29:59,495::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:29:59,495::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:34:59,958::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:34:59,958::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:34:59,959::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:35:00,168::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:35:00,170::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:35:00,170::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28771::WARNING::2012-12-04 10:35:00,193::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 10:35:00,211::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:35:00,211::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:35:00,215::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:35:00,218::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:35:00,218::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:35:00,218::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:40:00,673::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:40:00,673::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:40:00,673::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:40:00,878::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:40:00,880::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:40:00,880::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:40:00,915::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:40:00,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:40:00,918::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:40:00,920::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:40:00,920::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:40:00,921::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:45:01,418::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:45:01,418::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:45:01,419::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:45:01,623::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:45:01,625::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:45:01,625::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-28888::WARNING::2012-12-04 10:45:01,645::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-28889::WARNING::2012-12-04 10:45:01,645::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 10:45:01,665::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:45:01,665::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:45:01,671::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:45:01,674::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:45:01,674::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:45:01,674::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:50:02,219::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:50:02,219::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:50:02,220::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:50:02,434::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:50:02,435::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:50:02,436::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:50:02,472::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:50:02,472::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:50:02,476::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:50:02,480::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:50:02,480::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:50:02,480::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 10:55:02,911::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:55:02,911::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 10:55:02,912::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 10:55:03,121::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 10:55:03,123::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 10:55:03,123::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 10:55:03,161::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 10:55:03,161::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 10:55:03,164::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 10:55:03,167::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 10:55:03,167::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 10:55:03,168::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:00:03,569::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:00:03,570::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:00:03,570::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:00:03,777::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:00:03,778::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:00:03,778::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:00:03,815::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:00:03,815::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:00:03,819::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:00:03,822::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:00:03,822::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:00:03,822::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:05:04,287::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:05:04,288::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:05:04,291::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:05:04,510::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:05:04,512::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:05:04,513::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29152::WARNING::2012-12-04 11:05:04,539::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 11:05:04,551::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:05:04,551::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:05:04,554::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:05:04,557::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:05:04,557::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:05:04,557::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:10:05,011::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:10:05,011::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:10:05,011::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:10:05,221::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:10:05,223::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:10:05,223::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:10:05,261::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:10:05,261::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:10:05,265::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:10:05,267::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:10:05,267::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:10:05,267::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:15:05,668::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:15:05,669::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:15:05,669::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:15:05,886::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:15:05,888::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:15:05,888::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29255::WARNING::2012-12-04 11:15:05,907::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29256::WARNING::2012-12-04 11:15:05,907::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29257::WARNING::2012-12-04 11:15:05,908::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29258::WARNING::2012-12-04 11:15:05,908::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29260::WARNING::2012-12-04 11:15:05,910::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol3_somevol >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29262::WARNING::2012-12-04 11:15:05,911::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29263::WARNING::2012-12-04 11:15:05,911::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix5 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29264::WARNING::2012-12-04 11:15:05,912::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29265::WARNING::2012-12-04 11:15:05,912::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29266::WARNING::2012-12-04 11:15:05,913::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29269::WARNING::2012-12-04 11:15:05,915::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29270::WARNING::2012-12-04 11:15:05,916::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29272::WARNING::2012-12-04 11:15:05,917::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29273::WARNING::2012-12-04 11:15:05,917::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29274::WARNING::2012-12-04 11:15:05,918::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29276::WARNING::2012-12-04 11:15:05,918::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29275::WARNING::2012-12-04 11:15:05,919::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29277::WARNING::2012-12-04 11:15:05,919::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 11:15:05,935::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:15:05,936::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:15:06,144::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:15:06,146::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:15:06,146::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29334::WARNING::2012-12-04 11:15:06,172::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-29335::WARNING::2012-12-04 11:15:06,172::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 11:15:06,184::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:15:06,185::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:15:06,189::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:15:06,192::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:15:06,192::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:15:06,192::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-04 11:15:06,193::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-04 11:15:06,193::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-04 11:15:16,207::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-04 11:20:06,613::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:20:06,613::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:20:06,614::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:20:06,822::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:20:06,824::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:20:06,824::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:20:06,861::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:20:06,861::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:20:06,865::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:20:06,868::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:20:06,868::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:20:06,868::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:25:07,347::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:25:07,347::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:25:07,348::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:25:07,556::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:25:07,557::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:25:07,558::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:25:07,594::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:25:07,595::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:25:07,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:25:07,599::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:25:07,599::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:25:07,600::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:30:08,068::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:30:08,068::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:30:08,068::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:30:08,274::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:30:08,275::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:30:08,275::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29521::WARNING::2012-12-04 11:30:08,300::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 11:30:08,312::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:30:08,312::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:30:08,316::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:30:08,319::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:30:08,319::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:30:08,319::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:35:08,757::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:35:08,758::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:35:08,758::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:35:08,961::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:35:08,963::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:35:08,963::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:35:08,999::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:35:08,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:35:09,002::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:35:09,004::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:35:09,004::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:35:09,005::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:40:09,430::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:40:09,431::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:40:09,431::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:40:09,634::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:40:09,635::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:40:09,636::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29648::WARNING::2012-12-04 11:40:09,662::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 11:40:09,671::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:40:09,672::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:40:09,675::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:40:09,677::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:40:09,678::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:40:09,678::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:45:10,122::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:45:10,122::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:45:10,123::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:45:10,327::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:45:10,329::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:45:10,329::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:45:10,366::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:45:10,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:45:10,370::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:45:10,373::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:45:10,373::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:45:10,373::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:50:10,934::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:50:10,934::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:50:10,934::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:50:11,139::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:50:11,140::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:50:11,141::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29770::WARNING::2012-12-04 11:50:11,167::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 11:50:11,181::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:50:11,181::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:50:11,185::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:50:11,188::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:50:11,188::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:50:11,188::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 11:55:11,651::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:55:11,651::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 11:55:11,651::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 11:55:11,854::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 11:55:11,855::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 11:55:11,856::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 11:55:11,891::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 11:55:11,892::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 11:55:11,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 11:55:11,896::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 11:55:11,896::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 11:55:11,897::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:00:12,349::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:00:12,349::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:00:12,350::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:00:12,554::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:00:12,556::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:00:12,556::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:00:12,592::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:00:12,593::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:00:12,596::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:00:12,599::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:00:12,599::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:00:12,599::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:05:13,059::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:05:13,059::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:05:13,062::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:05:13,281::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:05:13,282::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:05:13,283::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29959::WARNING::2012-12-04 12:05:13,311::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 12:05:13,323::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:05:13,323::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:05:13,327::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:05:13,330::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:05:13,330::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:05:13,330::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:10:13,830::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:10:13,830::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:10:13,831::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:10:14,045::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:10:14,047::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:10:14,047::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:10:14,086::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:10:14,086::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:10:14,090::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:10:14,093::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:10:14,093::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:10:14,093::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:15:14,529::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:15:14,529::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:15:14,529::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:15:14,741::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:15:14,743::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:15:14,743::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30081::WARNING::2012-12-04 12:15:14,769::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-30080::WARNING::2012-12-04 12:15:14,770::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-30082::WARNING::2012-12-04 12:15:14,770::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 12:15:14,781::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:15:14,781::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:15:14,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:15:14,786::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:15:14,786::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:15:14,786::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:20:15,254::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:20:15,255::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:20:15,255::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:20:15,461::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:20:15,463::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:20:15,463::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:20:15,503::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:20:15,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:20:15,506::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:20:15,509::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:20:15,509::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:20:15,509::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:25:15,932::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:25:15,932::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:25:15,933::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:25:16,157::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:25:16,158::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:25:16,159::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:25:16,195::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:25:16,195::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:25:16,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:25:16,200::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:25:16,201::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:25:16,201::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:30:16,634::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:30:16,635::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:30:16,635::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:30:16,843::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:30:16,845::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:30:16,845::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30257::WARNING::2012-12-04 12:30:16,867::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-30266::WARNING::2012-12-04 12:30:16,871::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 12:30:16,885::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:30:16,885::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:30:16,889::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:30:16,892::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:30:16,892::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:30:16,892::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:35:17,326::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:35:17,326::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:35:17,327::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:35:17,532::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:35:17,534::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:35:17,534::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30330::WARNING::2012-12-04 12:35:17,561::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 12:35:17,571::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:35:17,571::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:35:17,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:35:17,577::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:35:17,577::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:35:17,577::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:40:18,024::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:40:18,024::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:40:18,024::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:40:18,230::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:40:18,232::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:40:18,232::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:40:18,270::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:40:18,270::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:40:18,274::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:40:18,276::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:40:18,276::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:40:18,276::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:45:18,677::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:45:18,677::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:45:18,677::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:45:18,884::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:45:18,885::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:45:18,885::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:45:18,920::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:45:18,921::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:45:18,923::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:45:18,925::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:45:18,926::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:45:18,926::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:50:19,360::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:50:19,360::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:50:19,361::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:50:19,567::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:50:19,568::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:50:19,569::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:50:19,605::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:50:19,605::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:50:19,609::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:50:19,614::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:50:19,614::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:50:19,614::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 12:55:20,051::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:55:20,052::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 12:55:20,052::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 12:55:20,257::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 12:55:20,259::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 12:55:20,259::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 12:55:20,298::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 12:55:20,298::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 12:55:20,301::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 12:55:20,304::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 12:55:20,304::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 12:55:20,305::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:00:20,737::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:00:20,737::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:00:20,737::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:00:20,942::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:00:20,943::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:00:20,944::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:00:20,980::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:00:20,981::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:00:20,983::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:00:20,986::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:00:20,986::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:00:20,986::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:05:21,409::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:05:21,409::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:05:21,412::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:05:21,634::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:05:21,637::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:05:21,638::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30702::WARNING::2012-12-04 13:05:21,665::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 13:05:21,675::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:05:21,676::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:05:21,681::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:05:21,685::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:05:21,685::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:05:21,685::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:10:22,109::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:10:22,110::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:10:22,110::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:10:22,326::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:10:22,328::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:10:22,328::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:10:22,365::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:10:22,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:10:22,369::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:10:22,371::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:10:22,372::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:10:22,372::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:15:22,826::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:15:22,826::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:15:22,827::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:15:23,037::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:15:23,039::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:15:23,039::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30820::WARNING::2012-12-04 13:15:23,062::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-30822::WARNING::2012-12-04 13:15:23,064::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-30826::WARNING::2012-12-04 13:15:23,066::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 13:15:23,078::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:15:23,078::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:15:23,081::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:15:23,084::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:15:23,085::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:15:23,085::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:20:23,508::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:20:23,508::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:20:23,509::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:20:23,717::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:20:23,718::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:20:23,719::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-30886::WARNING::2012-12-04 13:20:23,746::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 13:20:23,759::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:20:23,759::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:20:23,763::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:20:23,765::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:20:23,765::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:20:23,766::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:25:24,193::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:25:24,193::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:25:24,194::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:25:24,401::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:25:24,403::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:25:24,403::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:25:24,441::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:25:24,441::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:25:24,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:25:24,447::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:25:24,447::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:25:24,447::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:30:24,837::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:30:24,837::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:30:24,838::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:30:25,040::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:30:25,042::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:30:25,042::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:30:25,081::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:30:25,081::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:30:25,084::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:30:25,087::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:30:25,087::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:30:25,087::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:35:25,512::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:35:25,513::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:35:25,513::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:35:25,719::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:35:25,721::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:35:25,721::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:35:25,760::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:35:25,760::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:35:25,764::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:35:25,767::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:35:25,767::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:35:25,767::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:40:26,256::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:40:26,256::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:40:26,256::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:40:26,461::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:40:26,462::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:40:26,463::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:40:26,499::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:40:26,500::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:40:26,503::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:40:26,506::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:40:26,506::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:40:26,507::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:45:26,943::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:45:26,943::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:45:26,944::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:45:27,153::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:45:27,154::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:45:27,155::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31192::WARNING::2012-12-04 13:45:27,179::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 13:45:27,191::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:45:27,191::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:45:27,193::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:45:27,196::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:45:27,196::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:45:27,196::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:50:27,624::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:50:27,625::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:50:27,625::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:50:27,833::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:50:27,834::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:50:27,834::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31256::WARNING::2012-12-04 13:50:27,860::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-31258::WARNING::2012-12-04 13:50:27,862::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 13:50:27,875::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:50:27,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:50:27,878::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:50:27,881::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:50:27,881::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:50:27,881::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 13:55:28,378::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:55:28,379::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 13:55:28,382::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 13:55:28,595::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 13:55:28,596::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 13:55:28,597::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 13:55:28,634::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 13:55:28,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 13:55:28,637::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 13:55:28,640::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 13:55:28,640::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 13:55:28,641::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:00:29,114::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:00:29,115::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:00:29,115::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:00:29,322::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:00:29,323::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:00:29,324::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:00:29,361::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:00:29,361::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:00:29,364::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:00:29,367::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:00:29,367::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:00:29,368::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:05:29,806::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:05:29,806::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:05:29,807::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:05:30,028::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:05:30,031::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:05:30,031::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:05:30,068::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:05:30,068::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:05:30,071::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:05:30,074::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:05:30,074::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:05:30,074::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:10:30,503::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:10:30,504::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:10:30,507::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:10:30,718::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:10:30,720::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:10:30,721::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:10:30,756::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:10:30,757::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:10:30,759::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:10:30,761::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:10:30,761::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:10:30,762::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:15:31,230::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:15:31,231::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:15:31,231::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:15:31,436::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:15:31,438::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:15:31,438::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:15:31,474::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:15:31,475::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:15:31,478::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:15:31,480::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:15:31,480::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:15:31,481::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:20:32,048::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:20:32,048::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:20:32,049::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:20:32,259::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:20:32,261::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:20:32,261::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:20:32,296::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:20:32,296::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:20:32,299::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:20:32,302::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:20:32,302::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:20:32,302::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:25:32,732::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:25:32,732::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:25:32,733::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:25:32,939::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:25:32,941::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:25:32,941::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:25:32,974::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:25:32,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:25:32,977::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:25:32,980::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:25:32,980::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:25:32,980::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:30:33,410::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:30:33,410::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:30:33,411::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:30:33,621::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:30:33,622::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:30:33,623::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:30:33,659::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:30:33,659::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:30:33,663::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:30:33,666::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:30:33,666::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:30:33,666::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:35:34,111::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:35:34,111::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:35:34,111::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:35:34,319::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:35:34,321::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:35:34,321::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:35:34,356::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:35:34,356::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:35:34,360::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:35:34,363::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:35:34,363::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:35:34,363::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:40:34,845::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:40:34,845::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:40:34,846::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:40:35,053::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:40:35,054::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:40:35,054::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31880::WARNING::2012-12-04 14:40:35,082::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 14:40:35,092::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:40:35,092::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:40:35,095::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:40:35,098::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:40:35,098::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:40:35,098::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:45:35,520::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:45:35,521::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:45:35,521::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:45:35,729::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:45:35,731::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:45:35,731::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-31941::WARNING::2012-12-04 14:45:35,757::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 14:45:35,769::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:45:35,769::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:45:35,773::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:45:35,776::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:45:35,776::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:45:35,776::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:50:36,201::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:50:36,201::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:50:36,202::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:50:36,410::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:50:36,411::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:50:36,411::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:50:36,448::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:50:36,448::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:50:36,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:50:36,453::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:50:36,453::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:50:36,453::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 14:55:36,860::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:55:36,860::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 14:55:36,861::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 14:55:37,068::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 14:55:37,069::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 14:55:37,069::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 14:55:37,106::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 14:55:37,106::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 14:55:37,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 14:55:37,111::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 14:55:37,111::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 14:55:37,111::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:00:37,567::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:00:37,567::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:00:37,567::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:00:37,781::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:00:37,782::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:00:37,782::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32129::WARNING::2012-12-04 15:00:37,810::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 15:00:37,819::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:00:37,819::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:00:37,822::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:00:37,825::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:00:37,825::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:00:37,825::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:05:38,232::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:05:38,232::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:05:38,233::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:05:38,450::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:05:38,452::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:05:38,452::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:05:38,489::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:05:38,489::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:05:38,493::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:05:38,496::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:05:38,496::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:05:38,496::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:10:38,911::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:10:38,912::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:10:38,915::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:10:39,134::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:10:39,135::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:10:39,136::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32253::WARNING::2012-12-04 15:10:39,161::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 15:10:39,172::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:10:39,172::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:10:39,174::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:10:39,177::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:10:39,177::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:10:39,177::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:15:39,589::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:15:39,589::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:15:39,590::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:15:39,799::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:15:39,800::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:15:39,801::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:15:39,838::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:15:39,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:15:39,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:15:39,843::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:15:39,843::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:15:39,843::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:20:40,285::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:20:40,285::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:20:40,286::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:20:40,494::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:20:40,495::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:20:40,496::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:20:40,531::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:20:40,532::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:20:40,537::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:20:40,540::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:20:40,540::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:20:40,540::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:25:40,971::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:25:40,971::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:25:40,972::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:25:41,174::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:25:41,177::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:25:41,177::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:25:41,213::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:25:41,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:25:41,216::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:25:41,218::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:25:41,218::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:25:41,218::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:30:41,639::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:30:41,640::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:30:41,640::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:30:41,843::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:30:41,845::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:30:41,845::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32497::WARNING::2012-12-04 15:30:41,871::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-32499::WARNING::2012-12-04 15:30:41,872::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 15:30:41,884::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:30:41,885::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:30:41,889::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:30:41,892::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:30:41,892::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:30:41,892::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:35:42,327::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:35:42,327::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:35:42,328::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:35:42,531::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:35:42,532::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:35:42,532::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-32541::WARNING::2012-12-04 15:35:42,550::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1523 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-32542::WARNING::2012-12-04 15:35:42,550::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-32544::WARNING::2012-12-04 15:35:42,552::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-32550::WARNING::2012-12-04 15:35:42,555::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-32552::WARNING::2012-12-04 15:35:42,556::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 15:35:42,570::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:35:42,570::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:35:42,573::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:35:42,575::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:35:42,575::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:35:42,575::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:40:42,993::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:40:42,994::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:40:42,994::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:40:43,198::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:40:43,199::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:40:43,200::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:40:43,235::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:40:43,235::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:40:43,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:40:43,240::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:40:43,241::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:40:43,241::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:45:43,708::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:45:43,708::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:45:43,709::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:45:43,919::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:45:43,921::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:45:43,921::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:45:43,956::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:45:43,957::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:45:43,960::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:45:43,962::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:45:43,963::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:45:43,963::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:50:44,479::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:50:44,479::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:50:44,479::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:50:44,684::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:50:44,685::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:50:44,686::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:50:44,722::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:50:44,722::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:50:44,724::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:50:44,726::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:50:44,726::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:50:44,727::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 15:55:45,130::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:55:45,130::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 15:55:45,131::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 15:55:45,337::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 15:55:45,338::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 15:55:45,338::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 15:55:45,374::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 15:55:45,375::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 15:55:45,377::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 15:55:45,379::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 15:55:45,380::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 15:55:45,380::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:00:45,812::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:00:45,812::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:00:45,812::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:00:46,020::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:00:46,021::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:00:46,021::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:00:46,059::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:00:46,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:00:46,062::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:00:46,065::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:00:46,065::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:00:46,065::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:05:46,517::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:05:46,517::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:05:46,521::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:05:46,742::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:05:46,743::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:05:46,744::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:05:46,781::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:05:46,781::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:05:46,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:05:46,786::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:05:46,786::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:05:46,787::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:10:47,220::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:10:47,221::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:10:47,221::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:10:47,433::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:10:47,434::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:10:47,435::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:10:47,470::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:10:47,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:10:47,473::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:10:47,476::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:10:47,476::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:10:47,477::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:15:47,911::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:15:47,911::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:15:47,911::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:15:48,123::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:15:48,124::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:15:48,124::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:15:48,159::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:15:48,160::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:15:48,163::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:15:48,166::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:15:48,166::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:15:48,166::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:20:48,610::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:20:48,610::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:20:48,610::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:20:48,820::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:20:48,821::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:20:48,822::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:20:48,859::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:20:48,859::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:20:48,862::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:20:48,864::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:20:48,864::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:20:48,864::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:25:49,279::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:25:49,279::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:25:49,279::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:25:49,488::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:25:49,490::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:25:49,490::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:25:49,524::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:25:49,525::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:25:49,528::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:25:49,531::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:25:49,531::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:25:49,531::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:30:49,961::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:30:49,962::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:30:49,962::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:30:50,173::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:30:50,174::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:30:50,174::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33242::WARNING::2012-12-04 16:30:50,199::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-33244::WARNING::2012-12-04 16:30:50,200::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 16:30:50,213::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:30:50,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:30:50,216::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:30:50,219::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:30:50,219::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:30:50,220::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:35:50,618::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:35:50,618::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:35:50,618::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:35:50,823::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:35:50,825::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:35:50,825::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:35:50,862::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:35:50,862::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:35:50,865::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:35:50,868::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:35:50,868::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:35:50,868::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:40:51,292::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:40:51,292::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:40:51,293::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:40:51,498::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:40:51,500::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:40:51,500::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:40:51,537::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:40:51,537::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:40:51,541::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:40:51,543::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:40:51,544::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:40:51,544::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:45:52,087::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:45:52,087::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:45:52,087::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:45:52,293::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:45:52,294::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:45:52,294::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33428::WARNING::2012-12-04 16:45:52,321::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-33431::WARNING::2012-12-04 16:45:52,322::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 16:45:52,333::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:45:52,333::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:45:52,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:45:52,340::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:45:52,340::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:45:52,340::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:50:52,756::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:50:52,756::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:50:52,757::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:50:52,962::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:50:52,963::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:50:52,963::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:50:52,997::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:50:52,997::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:50:53,000::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:50:53,002::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:50:53,003::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:50:53,003::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 16:55:53,410::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:55:53,410::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 16:55:53,410::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 16:55:53,618::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 16:55:53,620::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 16:55:53,620::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 16:55:53,662::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 16:55:53,663::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 16:55:53,666::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 16:55:53,669::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 16:55:53,669::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 16:55:53,669::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:00:54,070::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:00:54,070::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:00:54,071::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:00:54,346::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:00:54,348::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:00:54,348::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:00:54,383::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:00:54,384::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:00:54,387::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:00:54,390::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:00:54,390::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:00:54,390::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:05:54,812::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:05:54,813::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:05:54,816::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:05:55,039::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:05:55,041::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:05:55,042::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:05:55,078::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:05:55,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:05:55,082::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:05:55,085::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:05:55,085::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:05:55,085::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:10:55,487::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:10:55,487::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:10:55,487::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:10:55,702::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:10:55,704::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:10:55,704::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:10:55,744::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:10:55,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:10:55,747::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:10:55,750::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:10:55,750::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:10:55,750::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:15:56,168::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:15:56,168::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:15:56,168::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:15:56,380::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:15:56,382::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:15:56,382::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:15:56,417::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:15:56,418::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:15:56,421::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:15:56,423::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:15:56,423::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:15:56,424::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:20:56,865::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:20:56,865::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:20:56,865::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:20:57,072::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:20:57,074::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:20:57,074::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:20:57,110::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:20:57,110::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:20:57,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:20:57,115::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:20:57,115::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:20:57,115::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:25:57,549::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:25:57,549::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:25:57,550::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:25:57,756::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:25:57,758::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:25:57,758::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-33921::WARNING::2012-12-04 17:25:57,783::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-33924::WARNING::2012-12-04 17:25:57,785::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 17:25:57,796::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:25:57,796::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:25:57,800::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:25:57,803::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:25:57,804::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:25:57,804::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:30:58,282::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:30:58,282::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:30:58,283::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:30:58,488::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:30:58,489::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:30:58,490::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:30:58,528::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:30:58,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:30:58,533::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:30:58,535::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:30:58,536::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:30:58,536::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:35:58,966::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:35:58,967::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:35:58,967::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:35:59,172::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:35:59,173::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:35:59,174::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34047::WARNING::2012-12-04 17:35:59,199::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 17:35:59,212::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:35:59,212::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:35:59,216::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:35:59,218::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:35:59,219::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:35:59,219::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:40:59,624::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:40:59,624::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:40:59,624::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:40:59,831::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:40:59,833::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:40:59,833::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34095::WARNING::2012-12-04 17:40:59,853::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34109::WARNING::2012-12-04 17:40:59,858::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34111::WARNING::2012-12-04 17:40:59,860::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 17:40:59,872::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:40:59,872::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:40:59,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:40:59,879::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:40:59,879::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:40:59,879::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:46:00,302::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:46:00,302::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:46:00,302::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:46:00,509::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:46:00,510::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:46:00,510::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34171::WARNING::2012-12-04 17:46:00,536::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34172::WARNING::2012-12-04 17:46:00,537::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34175::WARNING::2012-12-04 17:46:00,539::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 17:46:00,553::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:46:00,553::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:46:00,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:46:00,560::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:46:00,560::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:46:00,560::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:51:01,007::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:51:01,007::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:51:01,007::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:51:01,212::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:51:01,213::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:51:01,214::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 17:51:01,251::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:51:01,252::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:51:01,255::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:51:01,258::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:51:01,258::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:51:01,259::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 17:56:01,693::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:56:01,693::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 17:56:01,697::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 17:56:01,912::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 17:56:01,913::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 17:56:01,914::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34298::WARNING::2012-12-04 17:56:01,940::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34299::WARNING::2012-12-04 17:56:01,940::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 17:56:01,951::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 17:56:01,951::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 17:56:01,955::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 17:56:01,958::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 17:56:01,958::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 17:56:01,958::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:01:02,370::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:01:02,371::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:01:02,372::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:01:02,604::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:01:02,606::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:01:02,606::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:01:02,643::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:01:02,643::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:01:02,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:01:02,648::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:01:02,648::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:01:02,648::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:06:03,069::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:06:03,069::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:06:03,069::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:06:03,283::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:06:03,285::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:06:03,285::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:06:03,322::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:06:03,323::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:06:03,326::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:06:03,329::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:06:03,329::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:06:03,329::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:11:03,739::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:11:03,739::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:11:03,740::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:11:03,956::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:11:03,957::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:11:03,958::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:11:03,994::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:11:03,995::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:11:03,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:11:04,001::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:11:04,002::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:11:04,002::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:16:04,404::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:16:04,404::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:16:04,405::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:16:04,612::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:16:04,614::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:16:04,614::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34538::WARNING::2012-12-04 18:16:04,637::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34540::WARNING::2012-12-04 18:16:04,638::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34541::WARNING::2012-12-04 18:16:04,639::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34542::WARNING::2012-12-04 18:16:04,639::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34543::WARNING::2012-12-04 18:16:04,640::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34544::WARNING::2012-12-04 18:16:04,640::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34546::WARNING::2012-12-04 18:16:04,641::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34545::WARNING::2012-12-04 18:16:04,641::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34547::WARNING::2012-12-04 18:16:04,641::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 18:16:04,702::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:16:04,702::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:16:04,705::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:16:04,707::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:16:04,708::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:16:04,708::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:21:05,121::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:21:05,121::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:21:05,121::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:21:05,326::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:21:05,328::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:21:05,329::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:21:05,365::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:21:05,365::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:21:05,368::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:21:05,370::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:21:05,370::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:21:05,371::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:26:05,860::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:26:05,860::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:26:05,861::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:26:06,069::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:26:06,070::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:26:06,070::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:26:06,106::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:26:06,106::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:26:06,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:26:06,111::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:26:06,111::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:26:06,112::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:31:06,511::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:31:06,511::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:31:06,512::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:31:06,719::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:31:06,720::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:31:06,721::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:31:06,760::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:31:06,760::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:31:06,763::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:31:06,765::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:31:06,765::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:31:06,765::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:36:07,171::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:36:07,172::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:36:07,172::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:36:07,378::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:36:07,380::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:36:07,380::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:36:07,420::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:36:07,421::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:36:07,423::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:36:07,425::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:36:07,426::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:36:07,426::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:41:08,049::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:41:08,050::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:41:08,050::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:41:08,258::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:41:08,259::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:41:08,259::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:41:08,297::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:41:08,297::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:41:08,306::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:41:08,309::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:41:08,309::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:41:08,309::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:46:08,725::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:46:08,725::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:46:08,726::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:46:08,933::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:46:08,934::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:46:08,934::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 18:46:08,971::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:46:08,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:46:08,975::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:46:08,978::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:46:08,978::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:46:08,979::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:51:09,418::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:51:09,418::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:51:09,419::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:51:09,627::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:51:09,629::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:51:09,629::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-34961::WARNING::2012-12-04 18:51:09,648::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34969::WARNING::2012-12-04 18:51:09,652::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-34973::WARNING::2012-12-04 18:51:09,654::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 18:51:09,666::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:51:09,667::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:51:09,669::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:51:09,671::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:51:09,672::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:51:09,672::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 18:56:10,109::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:56:10,109::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 18:56:10,109::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 18:56:10,319::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 18:56:10,320::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 18:56:10,320::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35043::WARNING::2012-12-04 18:56:10,349::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 18:56:10,360::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 18:56:10,361::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 18:56:10,364::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 18:56:10,367::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 18:56:10,367::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 18:56:10,367::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:01:10,787::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:01:10,788::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:01:10,788::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:01:11,087::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:01:11,089::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:01:11,090::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:01:11,130::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:01:11,130::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:01:11,134::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:01:11,137::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:01:11,137::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:01:11,137::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:06:11,594::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:06:11,594::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:06:11,594::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:06:11,805::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:06:11,806::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:06:11,807::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35163::WARNING::2012-12-04 19:06:11,832::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 19:06:11,845::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:06:11,845::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:06:11,849::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:06:11,851::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:06:11,852::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:06:11,852::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:11:12,300::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:11:12,301::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:11:12,304::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:11:12,514::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:11:12,515::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:11:12,515::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:11:12,554::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:11:12,555::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:11:12,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:11:12,559::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:11:12,559::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:11:12,559::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:16:13,057::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:16:13,057::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:16:13,058::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:16:13,264::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:16:13,265::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:16:13,266::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:16:13,303::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:16:13,303::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:16:13,306::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:16:13,308::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:16:13,308::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:16:13,308::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:21:13,728::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:21:13,729::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:21:13,729::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:21:13,932::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:21:13,933::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:21:13,934::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:21:13,970::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:21:13,970::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:21:13,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:21:13,975::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:21:13,976::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:21:13,976::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:26:14,388::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:26:14,388::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:26:14,388::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:26:14,591::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:26:14,592::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:26:14,592::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35413::WARNING::2012-12-04 19:26:14,618::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 19:26:14,628::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:26:14,629::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:26:14,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:26:14,634::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:26:14,634::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:26:14,634::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:31:15,058::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:31:15,059::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:31:15,059::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:31:15,274::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:31:15,276::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:31:15,277::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:31:15,313::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:31:15,313::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:31:15,315::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:31:15,318::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:31:15,318::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:31:15,318::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:36:15,724::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:36:15,725::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:36:15,725::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:36:15,938::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:36:15,939::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:36:15,939::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:36:15,976::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:36:15,977::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:36:15,980::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:36:15,983::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:36:15,983::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:36:15,984::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:41:16,377::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:41:16,378::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:41:16,378::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:41:16,585::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:41:16,586::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:41:16,586::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:41:16,623::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:41:16,624::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:41:16,626::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:41:16,628::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:41:16,628::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:41:16,629::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:46:17,061::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:46:17,061::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:46:17,062::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:46:17,267::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:46:17,269::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:46:17,269::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:46:17,305::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:46:17,305::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:46:17,310::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:46:17,313::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:46:17,313::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:46:17,314::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:51:17,730::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:51:17,731::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:51:17,731::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:51:17,938::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:51:17,939::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:51:17,940::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:51:17,980::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:51:17,980::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:51:17,984::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:51:17,986::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:51:17,987::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:51:17,987::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 19:56:18,439::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:56:18,439::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:56:18,439::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:56:18,644::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:56:18,645::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:56:18,646::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35766::WARNING::2012-12-04 19:56:18,665::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tl.redhat.com:_export_images_rnd_amureini_data4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35768::WARNING::2012-12-04 19:56:18,666::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35769::WARNING::2012-12-04 19:56:18,667::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix3 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35771::WARNING::2012-12-04 19:56:18,668::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_115 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35772::WARNING::2012-12-04 19:56:18,668::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35774::WARNING::2012-12-04 19:56:18,669::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1604 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35775::WARNING::2012-12-04 19:56:18,670::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1016 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35777::WARNING::2012-12-04 19:56:18,671::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35779::WARNING::2012-12-04 19:56:18,672::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35781::WARNING::2012-12-04 19:56:18,673::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35783::WARNING::2012-12-04 19:56:18,674::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35784::WARNING::2012-12-04 19:56:18,675::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35786::WARNING::2012-12-04 19:56:18,676::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 19:56:18,692::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 19:56:18,692::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 19:56:18,898::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 19:56:18,900::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 19:56:18,900::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 19:56:18,936::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 19:56:18,937::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 19:56:18,939::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 19:56:18,941::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 19:56:18,942::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 19:56:18,942::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::ERROR::2012-12-04 19:56:18,942::domainMonitor::152::Storage.DomainMonitor::(_monitorDomain) Error while collecting domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 130, in _monitorDomain > domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 47, in __getattr__ > dom = self.getRealDomain() > File "/usr/share/vdsm/storage/sdc.py", line 51, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce > dom = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 147, in _findDomain > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('a00efcc1-5685-4dbe-ae70-440e9f403b76',) >Thread-276::DEBUG::2012-12-04 19:56:18,942::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Invalid >Thread-276::DEBUG::2012-12-04 19:56:28,957::domainMonitor::160::Storage.DomainMonitor::(_monitorDomain) Domain `a00efcc1-5685-4dbe-ae70-440e9f403b76` changed its status to Valid >Thread-276::DEBUG::2012-12-04 20:01:19,368::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:01:19,368::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:01:19,369::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:01:19,589::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:01:19,592::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:01:19,593::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-35908::WARNING::2012-12-04 20:01:19,619::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-35911::WARNING::2012-12-04 20:01:19,621::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 20:01:19,632::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:01:19,632::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:01:19,682::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:01:19,685::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:01:19,685::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:01:19,685::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:06:20,120::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:06:20,121::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:06:20,121::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:06:20,331::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:06:20,333::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:06:20,333::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:06:20,369::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:06:20,369::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:06:20,373::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:06:20,376::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:06:20,376::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:06:20,376::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:11:20,909::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:11:20,909::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:11:20,910::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:11:21,124::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:11:21,126::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:11:21,126::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:11:21,164::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:11:21,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:11:21,168::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:11:21,170::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:11:21,170::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:11:21,170::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:16:21,567::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:16:21,568::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:16:21,568::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:16:21,775::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:16:21,776::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:16:21,777::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-36094::WARNING::2012-12-04 20:16:21,804::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 20:16:21,815::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:16:21,815::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:16:21,818::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:16:21,821::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:16:21,821::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:16:21,821::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:21:22,217::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:21:22,218::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:21:22,218::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:21:22,423::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:21:22,425::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:21:22,425::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:21:22,460::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:21:22,460::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:21:22,465::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:21:22,467::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:21:22,467::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:21:22,467::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:26:22,967::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:26:22,967::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:26:22,967::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:26:23,176::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:26:23,177::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:26:23,178::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:26:23,215::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:26:23,216::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:26:23,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:26:23,222::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:26:23,222::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:26:23,223::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:31:23,641::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:31:23,641::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:31:23,641::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:31:23,853::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:31:23,855::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:31:23,855::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-36272::WARNING::2012-12-04 20:31:23,876::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36280::WARNING::2012-12-04 20:31:23,880::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 20:31:23,893::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:31:23,894::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:31:23,897::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:31:23,899::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:31:23,899::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:31:23,899::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:36:24,300::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:36:24,300::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:36:24,300::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:36:24,508::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:36:24,509::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:36:24,510::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:36:24,548::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:36:24,549::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:36:24,552::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:36:24,555::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:36:24,555::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:36:24,555::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:41:24,969::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:41:24,969::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:41:24,973::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:41:25,184::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:41:25,186::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:41:25,186::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:41:25,226::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:41:25,226::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:41:25,229::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:41:25,231::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:41:25,232::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:41:25,232::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:46:25,629::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:46:25,629::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:46:25,629::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:46:25,837::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:46:25,838::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:46:25,839::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-36449::WARNING::2012-12-04 20:46:25,857::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36460::WARNING::2012-12-04 20:46:25,862::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multiplass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_export2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36462::WARNING::2012-12-04 20:46:25,863::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36464::WARNING::2012-12-04 20:46:25,864::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36466::WARNING::2012-12-04 20:46:25,865::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36468::WARNING::2012-12-04 20:46:25,866::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36469::WARNING::2012-12-04 20:46:25,867::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 20:46:25,879::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:46:25,879::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:46:25,882::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:46:25,884::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:46:25,884::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:46:25,884::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:51:26,309::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:51:26,310::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:51:26,310::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:51:26,517::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:51:26,519::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:51:26,519::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 20:51:26,556::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:51:26,556::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:51:26,558::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:51:26,561::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:51:26,561::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:51:26,561::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 20:56:26,969::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:56:26,969::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 20:56:26,970::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 20:56:27,174::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 20:56:27,176::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 20:56:27,176::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-36589::WARNING::2012-12-04 20:56:27,201::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-36590::WARNING::2012-12-04 20:56:27,202::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 20:56:27,212::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 20:56:27,212::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 20:56:27,215::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 20:56:27,217::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 20:56:27,217::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 20:56:27,217::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:01:27,611::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:01:27,611::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:01:27,612::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:01:27,832::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:01:27,835::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:01:27,835::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:01:27,872::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:01:27,872::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:01:27,875::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:01:27,877::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:01:27,877::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:01:27,877::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:06:28,330::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:06:28,330::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:06:28,330::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:06:28,534::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:06:28,535::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:06:28,536::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:06:28,572::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:06:28,573::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:06:28,576::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:06:28,579::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:06:28,579::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:06:28,579::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:11:29,039::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:11:29,040::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:11:29,040::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:11:29,251::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:11:29,253::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:11:29,253::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:11:29,289::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:11:29,289::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:11:29,292::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:11:29,294::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:11:29,294::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:11:29,295::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:16:29,696::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:16:29,696::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:16:29,697::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:16:29,901::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:16:29,903::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:16:29,903::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:16:29,939::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:16:29,939::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:16:29,942::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:16:29,944::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:16:29,945::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:16:29,945::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:21:30,357::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:21:30,358::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:21:30,358::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:21:30,561::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:21:30,563::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:21:30,563::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-36899::WARNING::2012-12-04 21:21:30,589::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 21:21:30,600::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:21:30,600::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:21:30,603::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:21:30,605::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:21:30,605::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:21:30,605::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:26:31,098::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:26:31,098::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:26:31,098::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:26:31,303::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:26:31,304::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:26:31,305::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:26:31,343::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:26:31,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:26:31,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:26:31,348::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:26:31,348::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:26:31,348::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:31:31,785::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:31:31,785::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:31:31,785::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:31:31,990::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:31:31,992::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:31:31,992::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:31:32,029::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:31:32,030::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:31:32,033::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:31:32,036::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:31:32,036::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:31:32,036::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:36:32,449::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:36:32,449::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:36:32,449::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:36:32,653::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:36:32,654::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:36:32,655::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:36:32,690::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:36:32,690::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:36:32,692::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:36:32,695::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:36:32,695::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:36:32,695::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:41:33,160::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:41:33,161::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:41:33,164::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:41:33,377::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:41:33,378::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:41:33,379::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-37131::WARNING::2012-12-04 21:41:33,398::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redaht.com:_export_images_rnd_vered_data1 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-37140::WARNING::2012-12-04 21:41:33,401::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-37144::WARNING::2012-12-04 21:41:33,404::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.redhat.tlv.com:_export_images_rnd_amureini_data2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 21:41:33,415::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:41:33,415::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:41:33,418::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:41:33,420::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:41:33,420::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:41:33,421::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:46:33,907::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:46:33,907::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:46:33,907::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:46:34,114::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:46:34,115::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:46:34,115::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-37213::WARNING::2012-12-04 21:46:34,144::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 21:46:34,155::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:46:34,155::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:46:34,158::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:46:34,160::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:46:34,160::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:46:34,161::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:51:34,559::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:51:34,559::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:51:34,559::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:51:34,765::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:51:34,767::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:51:34,767::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 21:51:34,806::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:51:34,806::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:51:34,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:51:34,812::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:51:34,812::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:51:34,812::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 21:56:35,277::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:56:35,277::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 21:56:35,277::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 21:56:35,487::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 21:56:35,490::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 21:56:35,491::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-37334::WARNING::2012-12-04 21:56:35,517::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 21:56:35,529::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 21:56:35,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 21:56:35,533::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 21:56:35,536::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 21:56:35,536::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 21:56:35,536::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:01:36,006::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:01:36,006::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:01:36,007::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:01:36,228::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:01:36,230::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:01:36,230::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:01:36,266::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:01:36,266::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:01:36,375::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:01:36,378::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:01:36,378::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:01:36,378::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:06:36,799::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:06:36,800::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:06:36,803::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:06:37,014::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:06:37,016::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:06:37,016::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:06:37,054::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:06:37,055::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:06:37,057::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:06:37,060::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:06:37,060::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:06:37,060::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:11:37,484::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:11:37,484::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:11:37,485::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:11:37,697::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:11:37,699::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:11:37,699::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:11:37,735::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:11:37,736::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:11:37,739::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:11:37,742::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:11:37,742::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:11:37,742::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:16:38,159::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:16:38,159::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:16:38,160::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:16:38,365::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:16:38,367::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:16:38,367::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:16:38,403::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:16:38,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:16:38,407::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:16:38,409::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:16:38,409::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:16:38,410::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:21:38,840::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:21:38,841::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:21:38,841::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:21:39,056::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:21:39,058::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:21:39,059::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:21:39,096::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:21:39,097::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:21:39,100::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:21:39,103::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:21:39,103::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:21:39,103::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:26:39,569::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:26:39,569::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:26:39,570::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:26:39,779::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:26:39,780::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:26:39,781::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-37706::WARNING::2012-12-04 22:26:39,808::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 22:26:39,819::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:26:39,819::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:26:39,823::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:26:39,826::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:26:39,826::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:26:39,826::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:31:40,243::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:31:40,243::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:31:40,243::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:31:40,453::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:31:40,455::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:31:40,455::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:31:40,492::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:31:40,493::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:31:40,497::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:31:40,499::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:31:40,500::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:31:40,500::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:36:40,984::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:36:40,984::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:36:40,985::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:36:41,191::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:36:41,193::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:36:41,193::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:36:41,232::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:36:41,232::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:36:41,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:36:41,239::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:36:41,239::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:36:41,239::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:41:41,752::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:41:41,752::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:41:41,753::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:41:41,958::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:41:41,960::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:41:41,960::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:41:41,999::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:41:41,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:41:42,002::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:41:42,004::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:41:42,004::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:41:42,005::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:46:42,471::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:46:42,471::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:46:42,471::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:46:42,678::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:46:42,679::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:46:42,680::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:46:42,716::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:46:42,717::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:46:42,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:46:42,724::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:46:42,724::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:46:42,724::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:51:43,160::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:51:43,160::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:51:43,161::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:51:43,366::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:51:43,368::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:51:43,368::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:51:43,403::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:51:43,404::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:51:43,407::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:51:43,409::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:51:43,409::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:51:43,410::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 22:56:43,908::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:56:43,908::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 22:56:43,909::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 22:56:44,120::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 22:56:44,121::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 22:56:44,122::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 22:56:44,157::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 22:56:44,158::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 22:56:44,160::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 22:56:44,163::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 22:56:44,163::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 22:56:44,163::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:01:44,574::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:01:44,574::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:01:44,577::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:01:44,798::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:01:44,800::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:01:44,800::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:01:44,838::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:01:44,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:01:44,843::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:01:44,846::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:01:44,846::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:01:44,846::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:06:45,258::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:06:45,259::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:06:45,259::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:06:45,497::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:06:45,499::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:06:45,499::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:06:45,536::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:06:45,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:06:45,540::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:06:45,542::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:06:45,542::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:06:45,542::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:11:45,999::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:11:45,999::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:11:46,000::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:11:46,210::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:11:46,212::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:11:46,213::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:11:46,249::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:11:46,249::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:11:46,254::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:11:46,257::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:11:46,257::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:11:46,257::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:16:46,667::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:16:46,668::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:16:46,668::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:16:46,878::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:16:46,879::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:16:46,880::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38326::WARNING::2012-12-04 23:16:46,907::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38329::WARNING::2012-12-04 23:16:46,909::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 23:16:46,919::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:16:46,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:16:46,923::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:16:46,926::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:16:46,927::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:16:46,927::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:21:47,335::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:21:47,336::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:21:47,336::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:21:47,541::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:21:47,542::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:21:47,543::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:21:47,581::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:21:47,581::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:21:47,584::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:21:47,587::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:21:47,587::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:21:47,587::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:26:48,078::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:26:48,078::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:26:48,079::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:26:48,282::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:26:48,284::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:26:48,284::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:26:48,321::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:26:48,322::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:26:48,325::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:26:48,328::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:26:48,328::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:26:48,329::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:31:48,746::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:31:48,747::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:31:48,747::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:31:48,951::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:31:48,952::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:31:48,953::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:31:48,990::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:31:48,990::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:31:48,994::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:31:48,997::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:31:48,997::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:31:48,997::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:36:49,413::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:36:49,413::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:36:49,414::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:36:49,620::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:36:49,622::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:36:49,622::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38558::WARNING::2012-12-04 23:36:49,642::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol4 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38567::WARNING::2012-12-04 23:36:49,646::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38566::WARNING::2012-12-04 23:36:49,647::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1813 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38571::WARNING::2012-12-04 23:36:49,649::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38573::WARNING::2012-12-04 23:36:49,650::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38575::WARNING::2012-12-04 23:36:49,651::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 23:36:49,668::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:36:49,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:36:49,673::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:36:49,676::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:36:49,676::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:36:49,676::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:41:50,172::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:41:50,172::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:41:50,173::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:41:50,378::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:41:50,380::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:41:50,380::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-04 23:41:50,418::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:41:50,418::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:41:50,422::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:41:50,425::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:41:50,425::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:41:50,425::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:46:50,856::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:46:50,856::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:46:50,857::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:46:51,063::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:46:51,064::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:46:51,065::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38697::WARNING::2012-12-04 23:46:51,091::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_0919 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 23:46:51,101::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:46:51,102::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:46:51,104::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:46:51,106::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:46:51,106::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:46:51,106::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:51:51,550::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:51:51,550::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:51:51,551::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:51:51,755::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:51:51,756::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:51:51,756::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38760::WARNING::2012-12-04 23:51:51,784::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 23:51:51,796::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:51:51,797::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:51:51,801::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:51:51,804::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:51:51,804::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:51:51,804::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-04 23:56:52,233::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:56:52,234::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-04 23:56:52,234::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-04 23:56:52,441::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-04 23:56:52,443::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-04 23:56:52,443::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38815::WARNING::2012-12-04 23:56:52,466::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38820::WARNING::2012-12-04 23:56:52,469::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38823::WARNING::2012-12-04 23:56:52,470::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-04 23:56:52,487::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-04 23:56:52,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-04 23:56:52,497::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-04 23:56:52,500::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-04 23:56:52,500::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-04 23:56:52,500::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:01:52,956::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:01:52,957::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:01:52,960::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:01:53,188::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:01:53,191::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:01:53,192::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-38882::WARNING::2012-12-05 00:01:53,217::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-38885::WARNING::2012-12-05 00:01:53,218::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/10.35.97.44:_vol7 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-05 00:01:53,233::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:01:53,233::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:01:53,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:01:53,239::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:01:53,239::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:01:53,239::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:06:53,668::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:06:53,669::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:06:53,669::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:06:53,877::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:06:53,878::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:06:53,879::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:06:53,915::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:06:53,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:06:53,918::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:06:53,920::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:06:53,920::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:06:53,921::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:11:54,349::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:11:54,350::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:11:54,350::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:11:54,561::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:11:54,562::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:11:54,563::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:11:54,600::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:11:54,600::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:11:54,605::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:11:54,608::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:11:54,608::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:11:54,608::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:16:55,053::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:16:55,053::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:16:55,053::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:16:55,262::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:16:55,264::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:16:55,264::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:16:55,303::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:16:55,304::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:16:55,307::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:16:55,309::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:16:55,310::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:16:55,310::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:21:55,707::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:21:55,708::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:21:55,708::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:21:55,915::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:21:55,916::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:21:55,917::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:21:55,953::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:21:55,954::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:21:55,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:21:55,960::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:21:55,960::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:21:55,960::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:26:56,359::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:26:56,359::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:26:56,360::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:26:56,566::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:26:56,567::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:26:56,568::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:26:56,604::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:26:56,604::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:26:56,614::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:26:56,617::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:26:56,617::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:26:56,618::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:31:57,030::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:31:57,030::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:31:57,030::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:31:57,238::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:31:57,240::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:31:57,241::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:31:57,276::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:31:57,276::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:31:57,280::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:31:57,283::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:31:57,283::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:31:57,283::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:36:57,750::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:36:57,750::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:36:57,750::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:36:57,959::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:36:57,961::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:36:57,961::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39320::WARNING::2012-12-05 00:36:57,988::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-05 00:36:57,998::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:36:57,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:36:58,002::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:36:58,005::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:36:58,005::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:36:58,005::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:41:58,418::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:41:58,418::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:41:58,418::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:41:58,626::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:41:58,628::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:41:58,628::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:41:58,664::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:41:58,664::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:41:58,668::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:41:58,671::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:41:58,671::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:41:58,671::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:46:59,097::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:46:59,097::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:46:59,098::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:46:59,306::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:46:59,308::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:46:59,308::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39445::WARNING::2012-12-05 00:46:59,335::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-39444::WARNING::2012-12-05 00:46:59,335::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-05 00:46:59,347::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:46:59,347::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:46:59,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:46:59,353::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:46:59,353::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:46:59,353::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:51:59,799::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:51:59,799::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:51:59,800::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:52:00,004::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:52:00,005::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:52:00,006::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:52:00,041::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:52:00,041::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:52:00,043::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:52:00,046::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:52:00,046::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:52:00,046::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 00:57:00,468::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:57:00,468::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 00:57:00,469::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 00:57:00,674::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 00:57:00,675::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 00:57:00,675::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 00:57:00,713::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 00:57:00,714::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 00:57:00,718::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 00:57:00,721::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 00:57:00,721::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 00:57:00,721::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:02:01,119::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:02:01,120::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:02:01,123::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:02:01,348::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:02:01,350::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:02:01,351::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 01:02:01,387::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:02:01,388::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:02:01,391::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:02:01,394::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:02:01,394::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:02:01,394::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:07:01,833::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:07:01,833::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:07:01,834::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:07:02,041::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:07:02,043::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:07:02,043::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 01:07:02,080::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:07:02,080::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:07:02,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:07:02,086::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:07:02,086::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:07:02,086::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:12:02,498::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:12:02,498::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:12:02,499::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:12:02,712::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:12:02,713::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:12:02,714::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 01:12:02,748::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:12:02,749::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:12:02,752::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:12:02,754::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:12:02,754::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:12:02,754::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:17:03,159::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:17:03,160::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:17:03,160::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:17:03,367::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:17:03,369::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:17:03,369::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 01:17:03,406::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:17:03,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:17:03,409::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:17:03,412::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:17:03,412::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:17:03,412::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:22:03,830::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:22:03,830::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:22:03,830::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:22:04,038::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:22:04,040::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:22:04,040::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 01:22:04,078::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:22:04,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:22:04,081::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:22:04,083::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:22:04,084::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:22:04,084::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:27:04,486::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:27:04,487::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:27:04,487::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:27:04,695::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:27:04,697::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:27:04,697::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-39938::WARNING::2012-12-05 01:27:04,723::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-05 01:27:04,734::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:27:04,734::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:27:04,737::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:27:04,739::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:27:04,739::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:27:04,739::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:32:05,249::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:32:05,249::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:32:05,249::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:32:05,454::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:32:05,455::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:32:05,456::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-276::DEBUG::2012-12-05 01:32:05,492::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:32:05,492::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:32:05,495::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:32:05,497::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:32:05,497::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:32:05,498::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:37:05,953::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:37:05,954::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:37:05,957::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:37:06,170::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:37:06,172::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:37:06,172::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-40060::WARNING::2012-12-05 01:37:06,197::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-05 01:37:06,211::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:37:06,212::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-276::DEBUG::2012-12-05 01:37:06,214::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=as', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=4', 'POOL_DESCRIPTION=dc', 'POOL_DOMAINS=225bca28-f0a0-4899-b137-32acd04f57d4:Active,a00efcc1-5685-4dbe-ae70-440e9f403b76:Active,5971512c-ee51-475e-b956-3872c38ce193:Active,2ee486e4-abfd-4f11-9b29-42bfc74d7064:Active', 'POOL_SPM_ID=1', 'POOL_SPM_LVER=2', 'POOL_UUID=479cf195-bb99-45c5-b400-433d810eaa48', 'REMOTE_PATH=multipass.eng.lab.tlv.redhat.com:/export/images/rnd/laravot/2348', 'ROLE=Master', 'SDUUID=a00efcc1-5685-4dbe-ae70-440e9f403b76', 'TYPE=NFS', 'VERSION=3', '_SHA_CKSUM=dd2f0acf7afe2a595f944d0664efa339ccc86609'] >Thread-276::DEBUG::2012-12-05 01:37:06,217::fileSD::419::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-276::WARNING::2012-12-05 01:37:06,217::sd::348::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_imageNS already registered >Thread-276::WARNING::2012-12-05 01:37:06,217::sd::356::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace a00efcc1-5685-4dbe-ae70-440e9f403b76_volumeNS already registered >Thread-276::DEBUG::2012-12-05 01:42:06,695::domainMonitor::125::Storage.DomainMonitor::(_monitorDomain) Refreshing domain a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:42:06,695::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-276::DEBUG::2012-12-05 01:42:06,695::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600144f09dbd050000004ddbe8770016|3600144f09dbd050000004ddbe8830017|3600144f09dbd050000004ddbe8910018|3600144f09dbd050000004ddbe89f0019|3600144f09dbd050000004ddbe8ad001a%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free a00efcc1-5685-4dbe-ae70-440e9f403b76' (cwd None) >Thread-276::DEBUG::2012-12-05 01:42:06,906::__init__::1164::Storage.Misc.excCmd::(_log) FAILED: <err> = ' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found\n'; <rc> = 5 >Thread-276::WARNING::2012-12-05 01:42:06,908::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "a00efcc1-5685-4dbe-ae70-440e9f403b76" not found'] >Thread-276::DEBUG::2012-12-05 01:42:06,908::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-40117::WARNING::2012-12-05 01:42:06,932::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1447 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-40119::WARNING::2012-12-05 01:42:06,934::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_10 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-40121::WARNING::2012-12-05 01:42:06,934::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1113 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-40122::WARNING::2012-12-05 01:42:06,935::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_amureini_posix2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-40124::WARNING::2012-12-05 01:42:06,936::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_sd1353 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-40126::WARNING::2012-12-05 01:42:06,937::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_lastlast2 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-40127::WARNING::2012-12-05 01:42:06,937::fileSD::451::scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_1017 >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 440, in collectMetaFiles > constants.UUID_GLOB_PATTERN, sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 285, in callCrabRPCFunction > raise Exception("No free file handlers in pool") >Exception: No free file handlers in pool >Thread-276::DEBUG::2012-12-05 01:42:06,973::fileSD::107::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_laravot_2348/a00efcc1-5685-4dbe-ae70-440e9f403b76 >Thread-276::DEBUG::2012-12-05 01:42:06,974::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >